Chapter 6: Emotional Interaction in Design

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement not replaced the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

Welcome to the Deep Dive.

Today we're tackling one of the most fundamental shifts in technology design.

We're moving away from, you know, just simple function.

And diving right into pure feeling.

Exactly.

Our mission is to explore that deep connection between human emotion and interface design, and we're using the foundational concepts laid out in the study of human -computer interaction to do it.

Right.

We're no longer just asking if an app works, but really, how does the app make you feel?

This is such a bigger question.

It is.

We're talking about emotional interaction, that critical space where designers are actively trying to detect, influence, and respond to your mood.

And this involves everything from, say, the aesthetic curve of a product to the deep neural networks that are now trying to read the expression on your face.

That's right.

And this deep dive is, I think, crucial because it frames technology not as some neutral tool, but almost as an emotional partner.

It is.

And we're going to systematically explore the four main pillars of this field.

We'll look at how our emotions structure the user experience.

Why some interfaces are so expressive and others are just annoying.

Exactly.

Then the rise of emotional AI and effective computing.

And finally, the power of persuasive design and the attachment we form through anthropomorphism.

It's going to be a lot to unpack.

You should probably get ready to rethink every digital interaction you have.

Let's do it.

OK.

So let's jump right into the core of the user experience.

The whole interaction design process really has to start with understanding the emotional roller coaster we're all on every day.

Think about shopping online for something big, like a new piece of furniture.

You start with that feeling of desire, maybe some joy when you find something you like.

And then almost immediately, the frustration of navigating all those filters or trying to make sense of confusing reviews.

Yes.

That frustration followed by the shock at the price, the relief when you finally click buy.

And then that creeping doubt that hits you like five minutes later.

Oh, the buyer's remorse that I really choose the right one.

Exactly.

That whole complex, rapid sequence of states is what emotional interaction aims to map and well, to manage.

The whole discipline uses knowledge of what makes people happy or frustrated or motivated to inform every single design decision.

And that complexity is key, right?

Because we have to distinguish between two different kinds of emotional responses.

We do.

First, you have these fleeting, really rapid, automatic responses that researchers call effect.

Effect.

Yeah.

These are automatic physical or psychological changes, like a sudden feeling of fear or surprise.

It happens without any conscious thought.

OK, so that's the fast one.

What's the other?

The other is the slower, more conscious, contemplative emotions.

And that's referred to as reflection.

Reflection.

So that's you mulling over a purchase afterwards.

Right.

Or feeling that lasting disappointment after a system fails on you.

That's reflection.

And this distinction, it has enormous implications for usability.

I mean, the designer Don Norman detailed this.

You really did.

He found that if you're in a positive, relaxed state, you're more creative, you're better at making quick decisions.

And maybe most importantly, when you're happy, you are far more tolerant of minor interface problems.

A slow loading screen, a little glitch.

You just let it slide.

Yeah.

And if you're anxious or angry, that tolerance just vanishes completely.

You're less patient, more critical.

You demand perfection.

And that observation led to this highly influential three -level model of emotional design,

which

uses the structure of the human brain as its framework to explain how design actually works on us.

OK, let's break down those three layers.

They seem like the foundation for this entire approach.

They are.

So at the very bottom, we have the visceral level.

Visceral, like a gut reaction.

Exactly.

It's the pre -wired automatic response, that immediate subconscious judgment of good or bad.

Before you even think about it.

Right.

And so visceral design is purely about aesthetics.

The look, the feel, the smell, the sound of a product.

It's that instantaneous attraction or repulsion.

OK, what's next?

Next up is the behavioral level.

This controls our everyday, well -learned operations.

Things like talking, typing, navigating a menu, driving.

So behavioral design, really, it equates to traditional usability, function, and performance.

The main question is just, does it work quickly and easily?

And the final, highest level.

That is the reflective level.

This is where your conscious thought lives.

Contemplation, meaning personal value.

So this is how a product fits into my life, my culture, my identity.

That's it.

It's the story you tell yourself and others about the product.

Is there a good example of something that hits all three of those levels?

A perfect one is the swatch watch.

Especially those brightly colored dip and color designs.

Oh, yeah.

The brilliant colors and the bold style, they hit you instantly at the visceral level.

It's designed to be attention grabbing and fun.

OK, I see that.

Then the ease of setting the time, the clear hands, that's the behavioral level.

It has to work as a watch.

It has traditional utility, sure.

But the brand itself, that cultural image of fun and fashion, that's the reflective level, isn't it?

Precisely.

That's what makes the watch mean something more than just telling time.

I do wonder though, is the reflective level just brilliant marketing?

Or can design genuinely create lasting meaning?

That's the challenge.

Because if the aesthetics are awful, if that visceral level fails,

does the usability even matter?

Most people won't stick around to appreciate a great function if the look immediately turns them off.

That's the challenge of true emotional design.

All three have to align.

But if the visceral appeal is high, it buys you tolerance for minor behavioral failures.

Ah, so that's why aesthetics are so critical.

They're crucial, which leads us perfectly into how interfaces are designed to express themselves.

Exactly.

Moving on to how designers intentionally try to create an emotional connection using what are called expressive interfaces.

Right.

They use everything from emojis and sounds to colors and virtual agents to elicit responses, you know, like comfort or happiness.

Think about the history of competing feedback.

When the original Macintosh booted up, it showed you a classic happy Mac icon.

Just a simple smiling face.

A simple smiling face.

It was designed to convey friendliness and just give you that immediate reassurance that this expensive complex machine was working.

Now compare that to when modern software freezes.

You get the colorful spinning beach ball on a Mac or the little moving clock on Android.

And while they're aesthetically clean, they're so much more impersonal.

We also get expressivity through sonifications.

Like the whoosh sound when you drag a file to the trash.

Or the ding of a new email.

Or those custom vibro -tactile buzzes on your phone that tell you who's calling without you even having to look.

And the research really backs up this aesthetic bias.

Noam Traktinsky showed pretty conclusively that aesthetically pleasing interfaces positively affect how a user perceives the usability.

So if it looks good, you think it works better.

And you're more patient.

You're willing to wait a few more seconds for a download if the system looks good.

We see that principle in practice when you compare, say,

the Nest thermostat.

A great example.

Which is minimalist,

round, sleek, and clearly displays the temperature.

You compare that to the complicated, utilitarian, traditional thermostats of the past.

The Nest is a masterclass in appealing to that visceral level.

It makes you feel good about the device before you've even touched it.

But if expressive interfaces are the goal, let's talk about where it all goes wrong.

Annoying interfaces.

This is where technology inadvertently triggers the worst emotions, like sheer frustration or anger.

We've all been there.

Common causes include things like complex remote controls.

That's classic featureitis.

Too many buttons.

Just way too many buttons and operations.

Then you have faulty apps,

a lack of sufficient feedback, and of course, obtuse error messages that assume you're a programmer.

And developers have desperately tried to inject personality to be friendly with often hilarious.

Or infuriating results.

We all remember those.

Exactly.

We saw the rise and fall of these friendly agents that were designed to help novices.

The first big attempt was Microsoft Bob.

It used a cozy living room metaphor with these cartoon agents, like a pet bog or a bunny, to guide you.

And it failed.

It failed commercially because most people found the whole thing too cute, childish,

and patronizing.

And then the true villain of the 90s.

Clippy.

Clippy, the infamous animated paperclip agent in Windows 98.

I remember the sheer frustration.

It was meant to help, but it was incredibly intrusive and distracting.

It was always topping up with redundant advice when you least needed it, which just fundamentally ruined the experience.

Even IKEA's attempt with Anna, a virtual agent that blinked and nodded, it eventually just disappeared, replaced by static text.

So the lesson is clear.

Trying too hard to be human, especially when the help isn't wanted, can backfire spectacularly.

Which brings up this idea of courtesy.

Early research suggested that computers should actually adhere to human social norms.

People are far more forgiving when a computer says it's sorry after making a mistake.

Think about the dreaded 404 arrow page.

That number itself implies client error.

That you, the user, did something wrong.

You misspelled the URL or something.

It feels accusatory.

It does.

Which is why using a humorous or a self -deprecating or just an apologetic error message is always better.

And speaking of courtesy, this raises a huge question about modern voice assistants like Alexa and Siri.

Children often learn they don't have to say please or thank you to a machine.

And some people worry that this lack of learned etiquette could bleed into their human social interactions.

It's a huge ethical dilemma for designers and for parents.

I mean, while some systems can be configured to reward polite behavior, we have to ask.

Is teaching a child to be polite to a machine undermining real human interaction by suggesting these things are equivalent?

Or is it just useful, low -stakes social training?

It really forces us to consider the social contract we're building with technology.

That's a powerful thought.

Now if designers can make us feel happy or frustrated, the next question is, can technology actually read our emotional state?

And that takes us into Effective Computing and Emotional AI.

Okay, what's the difference?

Effective computing, a term from Rosalind Picard back in 1998,

is the design of computers specifically to recognize and express emotions like humans.

And emotional AI.

Emotional AI takes this a step further.

It automates the measurement of feelings using really sophisticated tech to analyze everything from your facial expressions to your voice quality.

This is the technology powering the next generation of personalized interaction.

So how do they actually do it?

The field relies on four main sensing techniques.

First you have cameras, which are used for analyzing facial expressions.

Second, biosensors, often placed on your singers or palms, that measure things like galvanic skin response.

That tracks sweat, and it's a good proxy for anxiety.

What else?

Third, they analyze effective expression in speech.

They're scrutinizing pitch, intonation, rhythm.

A rising pitch can indicate excitement, for example.

And the last one?

And fourth, they track body movement and gestures, inferring your mood based on posture or sudden rapid movements.

The real world applications for this must be vast.

They are.

One of the leading commercial examples is AFTX software from a company called Affectiva.

This software uses advanced computer vision, often just through a simple webcam, to catalog precise emotional reactions to digital content, like ads or shopping sites.

So what does it look for?

It classifies six fundamental emotions,

anger, contempt,

disgust, fear, joy, and sadness.

And it does that by tracking specific quantifiable facial markers.

Like smiling.

Like the degree of a smile, the furrowing of a brow, or even just the rate of your blinking.

And we're seeing this used for real -time intervention, right?

Like for driver safety.

Absolutely.

If AFTX software detects anger through a tight facial expression, it might trigger a calming intervention, like suggesting the driver take a deep breath or changing the music.

And it also tracks drowsiness.

Crucially.

It tracks markers for drowsiness, like excessive eye closure or blinking rate, and it can prompt the driver to pull over immediately.

So beyond cameras and biosensors, I imagine they also use indirect methods?

Yeah, things like analyzing language patterns on social media.

And a really interesting application is popping up in streaming video games.

There's an app on the Twitch platform called All the Feels.

This system overlays biometric data, like the streamer's heart rate, skin conductance, and their emotional output right onto the screen for the viewers.

So it enhances the viewer's emotional connection to what the streamer is actually experiencing in that moment.

Dramatically.

Okay, now we have to shift gears completely.

If we can detect emotion, the next logical step is to use design to manipulate it.

And this brings us to persuasive technologies.

Persuasive design, as defined by Fogg, describes techniques used to entice or cajole or just gently nudge people into doing certain things.

Like Amazon's one -click button.

Or personalized product recommendations.

They're all designed to reduce friction and encourage that immediate purchase behavior.

But this technology is often used for positive behavioral change, too.

It is.

Take Nintendo's Pokemon Pikachu device.

The little pedometer thing.

Right.

The owner had to physically walk or run or jump to earn watts.

Which was the virtual currency used to buy presents for the pet.

And what happened if he didn't exercise?

Crucially, the virtual pet would get angry and sulk.

And that's a really clever use of negative emotional feedback to motivate physical activity.

Or what about the hate bescork?

Another great one.

That utensil vibrates and lights up if you're eating too quickly.

It interrupts that automatic behavior and encourages you to consciously reflect on your eating habits.

Just a gentle, physical nudge to drive a healthier choice.

Exactly.

And this concept of positive persuasion is vital in what's called sustainable HCI.

Especially for energy reduction.

Right.

Because research consistently shows that getting continuous or daily feedback is way more effective than just getting a monthly bill.

Much more.

But in a classic social norm study, they found a problem.

Households that were using less energy than average sometimes increased their consumption once they realized they were below the norm.

The dreaded boomerang effect.

That's it.

But designers quickly counteracted that effect using simple emotional feedback.

Emoticons.

The smiley face.

If low usage households got a smiley face with their conservation data, they kept conserving.

If high usage households got a sad face, they were motivated to decrease consumption even faster.

And the power of just visualizing this stuff goes even further, right, to the whole community.

Yeah.

The Tidy Street Project in Brighton is a great real -world example.

What did they do?

They spray painted a massive visualization of the street's collective electricity usage right on the road surface using chalk.

Wow.

And this public, real -time feedback immediately sparked community conversation and peer pressure which led to a remarkable 15 % reduction in electricity consumption across the whole street.

The emotional impact of making data public was huge.

Massive.

But while persuasion can drive positive change, we have to acknowledge the darker side.

Right.

Deceptive technology.

Like phishing scams.

They use authentic -looking, emotionally manipulative emails, often playing on fear or urgency or greed,

pretending to be from a bank or a legal authority to lure personal details from vulnerable people.

So the better designers understand emotion, the more potent the tools of deception can become.

It's a double -edged sword.

That ethical note is so critical.

Let's conclude with a related topic, anthropomorphism.

Which is attributing human qualities, traits, or emotions to animals or inanimate objects.

Which naturally leads to emotional attachment.

We've seen this evolve over decades, starting with early educational toys like Actimus Barney.

Barney the dinosaur.

That's the one.

It would react to TV content with appropriate emotions, cheering at good news and expressing concern at bad news.

And now you have more complex devices like the Luvabella Interactive Doll, which has facial expressions and learns to speak through interaction.

And in robotics, designers are making very deliberate choices about how anthropomorphic a device should be.

They are.

Early robot pets like Sony's AIBO are hard plastic and explicitly mechanical.

But you can contrast that with the soft, furry haptic creature.

And that was designed differently, wasn't it?

The haptic creature was designed to avoid having a face entirely, to prevent users from trying to read human facial emotions onto it.

Instead, it uses internal mechanisms to simulate breathing, a purr, and warmth.

It communicates its emotional state solely through touch.

Attachment through tactile sensation, not visual expression.

Exactly.

And now we see these devices moving into social care.

The Zora robot, a two -foot -tall robot, is used in nursing homes for entertainment and motivation.

Right alongside human staff.

And while many patients develop a real emotional attachment to Zora, it serves as this powerful reminder that these robots, however advanced they get, can never replace the human touch necessary for full care.

OK, so to synthesize this entire deep dive.

Interaction design has fundamentally shifted.

It's moved from focusing only on function to understanding and actively influencing feeling.

And those three levels, visceral, behavioral, and reflective,

they provide the framework for designing that emotional resonance.

Right.

And emotional AI is a rapidly growing field that's seeking to detect our exact emotional states using sophisticated sensors.

And persuasive technologies, for better or worse, are manipulating those emotions to drive specific behaviors.

That's the whole picture.

So as we look toward the future, think back to that tidy street project.

Simple visual feedback, leveraging social and community emotions, changed behavior by a significant 15%.

It's an incredible result.

So how could designers use the advanced emotional data we just discussed, tracking collective frustration levels or happiness or community well -being, to encourage positive collective action and make their own communities healthier or more harmonious?

That's the true power of emotional interaction.

Thank you for joining this deep dive.