Chapter 8: Data Gathering for Interaction Design

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement not replaced the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

Welcome to the Deep Dive.

Today, we are taking a rapid shortcut straight into the foundational engine of interaction design research,

data gathering.

That's right.

Our mission is to really understand the essential methods designers and researchers use to figure out what users actually need and, maybe more critically, whether the solutions they come up with actually work.

Yeah, we're focusing entirely on requirements, discovery, and evaluation methods.

And, you know, the core challenge we have to address right away is that gap between what people say and what people do.

Ah, yes.

The entire complexity of this field, really, it just stems from that fundamental disconnect.

That's the perfect frame for this.

But before we get into the techniques, we have to define our ingredients.

I mean, when we say data in this context, what are we actually collecting?

It's much broader than just spreadsheets.

Data can be numbers, words, descriptions, photos, videos, really anything useful for building an understanding.

And we immediately break it down into two types.

First, you have quantitative data.

Think measurable statistics.

The time it takes a user to do a task, number of errors, click counts on a button.

So the stuff you can easily put on a graph.

Precisely.

And second, there's qualitative data.

This is more descriptive.

It could be a user's verbal comments, their emotional responses, you know, something they wrote in a diary, or even just observations of their body language.

I can give you both.

Oh, absolutely.

A great study uses both.

So how do we collect this mix of metrics

and feelings?

Our sources lay out three foundational techniques.

And they're flexible enough to grab both those quantitative and qualitative nuggets.

Yep.

First, we have interviews.

These are synchronous.

They happen live either face to face or more and more remotely on something like Zoom.

They can range from, you know, a very rigid structured script to a totally free flowing conversation.

Then you've got questionnaires, which are asynchronous.

So the investigator isn't actually there when the participant is answering.

It relies completely on how clear the written questions are.

Right.

Usually distributed online now.

And finally, observation.

This can be direct so, watching the activity as it unfolds in real time, or it can be indirect.

Indirect.

How does that work?

That's where we record the activity, say through logs or video, and then study that record later.

But the key thing to remember is that you have to combine these.

Combining them is how you counter the biases that are, you know, just built into any single approach.

That makes a lot of sense.

But before we even pick up a camera or write a question, the sources insist there are these five non -negotiable issues that have to be addressed for any kind of success.

Yes.

These set the whole foundation professionally and logistically.

So let's dive into those.

Okay.

The first one is absolutely non -negotiable.

Goal setting.

Your data gathering must have clear, specific, and concise goals.

So what are you actually trying to find out?

Exactly.

Are you trying to see if a new interface is usable for beginners?

Are you comparing two different icons?

If the goal is muddy, honestly, the data you get back will be useless.

Right.

And the second is identifying participants.

This is where I guess things get tricky with logistics and rigor.

They do.

The entire group you're interested in is called the population.

And the act of choosing a smaller, manageable segment from that group, that's sampling.

And you pretty much always have to sample, right?

Almost always.

Accessing the entire population, what we call saturation sampling, is incredibly rare.

And we have two main approaches here.

There's probability sampling, which is very rigorous.

What does that give you?

It allows you to use statistical tests and robustly generalize your findings to that whole population.

Think simple random sampling.

Okay.

And the other kind?

The other is non -probability sampling.

This includes things like convenience sampling, you know, just using people who are readily available, like your colleagues, or snowball sampling, where your first participants go out and recruit others.

So they're more practical, faster maybe, but you're giving something up.

You're sacrificing statistical confidence.

You just can't generalize those findings with the same kind of reliability.

This brings up a critical, really practical question our audience always has.

How many people do you actually need?

Because if probability sampling is best, shouldn't you need hundreds?

Ideally, yes, statistically speaking.

But data from interaction design conferences, like CHI 2014, it shows a sort of pragmatic truth.

The local standard or rule of thumb in industry is often around 12 participants.

12.

That feels incredibly low, especially if you're aiming for statistical rigor.

What's the thinking there?

It's a trade -off.

It really reflects the industrial reality where things like cost and just feasibility, they often trump absolute statistical certainty.

I see.

These are often formative studies.

You know, they're not meant to be published in some high -stakes journal.

They're meant to generate quick, actionable information to improve a

Which brings us to the next issue,

the relationship with participants.

And this feels heavily focused on the ethical contract.

In a lot of places, this relationship is formalized with an informed consent form.

That form is vital.

It's the written guarantee that the participant understands the purpose of the study, how their data is going to be used securely and privately, and it confirms their right to stop at any time.

And this is a big deal globally, right?

It's a very serious issue.

It's reinforced by regulations like the EU's GDPR the General Data Protection Regulation, which sets a really high bar for how personal data has to be handled.

I wonder, though, what about incentives?

You know, giving someone 20 bucks to participate doesn't that risk introducing a different kind of bias?

Are you just sampling people who need the money?

That is a valid, critical point.

And researchers have to weigh the need for representation against that potential for skewing the sample.

It's often a logistical necessity, especially studies with kids or people with rare expertise, but it's something you have to report, honestly.

Okay, so moving past recruitment and ethics.

The fourth key issue is what helps us deal with that say versus do problem you mentioned earlier.

Triangulation.

Yes.

Triangulation just means investigating something from at least two different perspectives.

If one method tells you one thing and another tells you something else, well, that's where the real insight is.

The sources list four types, but the most common one is methodological triangulation.

Using different techniques like an interview and log data.

Exactly.

It doesn't guarantee validation because different methods give you different kinds of data, but using multiple approaches strengthens your insights immensely.

It's like checking a map with both satellite images and on -the -ground surveys.

And finally, before you launch the whole thing, you have to run a pilot study.

This is a small, absolutely essential trial run.

The whole point is to check viability.

Does the equipment work?

Are the instructions clear?

Are your questions maybe a bit ambiguous?

But if you're pressed for time and resources,

why bother with a mini study?

Why not just jump straight in?

Because catching a flaw in your method, a confusing question, a piece of failing equipment during a pilot study is cheap.

Catching that same flaw after you've run 100 participants is, well, it's catastrophic and expensive.

And there's a crucial rule here, right?

A crucial rule.

Pilot testers cannot participate in the main study.

Their prior knowledge will just fundamentally skew your findings.

Okay.

Before we move on to the techniques themselves, let's quickly nail down this conceptual chain the source has mentioned.

There are three stages, data, information, and conclusions.

Right.

Data is the raw material.

So if you run an observation study, the raw data is the user's keystrokes, the time logs, the video files, the raw stuff.

Information is what you get after you analyze and interpret all that.

For example, analyzing the logs shows that the new feature is used quickly by long -term users, but it causes a lot of errors for newcomers.

That's information.

And the conclusion is the action you take based on that information.

Exactly.

So the conclusion drawn from that error rate information might be, we need to design a much better help system specifically for new users.

Data leads to information, which leads to conclusions.

Excellent.

Okay.

Let's talk logistics, how we actually capture all this raw data.

Questionnaires and logs are often self -documenting, but interviews and observations, they require recording.

Let's start with the least technical, notes plus photographs.

Right.

Notes are flexible and they're generally less intrusive than, you know, typing frantically on a laptop or sticking a camera in someone's face.

A big advantage of handwriting is it forces you to filter and focus on what's important.

Kind of starts the analysis process right there.

The disadvantage though is limited speed and your own researcher bias.

So supplementing with photos or short smartphone videos is essential.

Okay.

Next level up, audio plus photographs.

Audio is fantastic for interviews.

It lets the interviewer maintain eye contact and really focus on the interviewee instead of being distracted by scribbling.

The recordings themselves are rarely fully transcribed, but they're excellent reminders or sources for direct quotes and anecdotes later on.

And then we have the big one, video.

This can be anything from simple smartphone clips to a professional multi -camera setup.

Right.

For detailed sessions in a makerspace or a usability lab.

And our sources point out three key things to consider when you use video based on work by Heath and colleagues.

What are they?

First, you have to decide between a fixed versus roving camera, depending on the activity.

Second, you have to figure out where to point the camera.

And you can't know that until you do some field work first.

And third, this is a big one.

You have to understand the recording's impact on participant behavior.

The Hawthorne effect.

Pretty much.

Do they act differently?

Do they keep looking at the camera?

You have to check your data for evidence that just the act of recording itself changed what they did.

Okay.

Let's jump into that first major pillar.

Interviews.

The conversation with a purpose.

There are four main types.

We start with the unstructured interview.

This is exploratory.

It's very conversation -like.

It's designed to generate really rich, complex, qualitative data.

You use open questions and you have to constantly be probing.

What do you mean by probing?

Just asking things like, can you tell me a bit more about how you felt when that happened?

But crucially, even though it's unstructured, you still need a planned agenda of topics to make sure you cover what you need.

Got it.

And at the other end of the spectrum.

We have structured interviews.

These are highly formal.

They're basically a questionnaire read aloud.

You use predetermined, standardized, closed questions with fixed answers.

Which makes the data easy to compare, I assume.

Very easy to compare.

But it's only useful if your goals are equally specific, which is why the most common is the semi -structured interview.

The hybrid.

The hybrid.

You use a basic script for guidance, but you give the interviewer freedom to probe and follow up on unexpected issues.

This balances replicability with depth.

But the interviewer has to maintain strict neutrality, no leading questions, no biased reactions.

And finally, focus groups.

Right.

This is when you bring three to ten people together with a facilitator.

They're incredibly useful for looking at shared issues, or maybe even more importantly, for finding areas of conflict or consensus among different groups, like getting faculty, students, and administrators all in one room to talk about a new university website.

It's important to remember that not all users are the same, either.

The sources talk about adapting techniques for specialized users.

Yes, this is key.

For very young children, for instance, you might not use complex text.

You'd use image -based rating tools like Smiley's to see how they feel.

Or in animal -computer interaction, you need specialized tracking, like a system that captures a dog's head turns, because the standard methods just won't work.

This all brings us back to the heart of the matter.

What they say and what they do.

That critical dilemma that just plagues interviews.

It's the biggest danger of relying purely on self -report.

People give socially desirable answers or they just, they simply forget their actual behavior.

There's a perfect example of this in the sources.

The twinkling lights study, yes.

An office floor was embedded with these lights to try and encourage people to use the stairs.

When interviewed, people swore they hadn't changed their habits at all.

But the objective interaction logs, they proved their behavior did change significantly.

So we have to recognize that the interview, no matter how well you plan it, is only ever one side of the story.

Exactly.

When you're planning, just stick to the basics.

Split long questions, get rid of jargon, keep questions neutral, and always follow that sequence.

Introduction, warm -up, main session, cooldown, and closing.

And if you can't meet in person, remote interviews are an option.

Upside is comfort for the participant, wider reach.

Downside is, you know, you can't read body language and you're at the mercy of a bad internet connection.

A great way to fight the artificiality of an interview space is to enrich the experience with props, prototypes, or artifacts the user actually works with every day.

It grounds the discussion in a real crucial context.

Let's move to the second pillar, questionnaires.

The biggest benefit here is you can distribute them to a huge number of people really easily.

But they're so much harder to design well.

Because you, the investigator, you aren't there to clarify if something is ambiguous.

Every single question has to be crystal clear and usually closed -ended.

And the structure matters.

Immensely.

Always start with essential context, like the respondents' experience level, and organize a long survey into logical topics.

Provide clear instructions.

And if you use number ranges, like for age, make sure they don't overlap.

It's a really common mistake.

We rely heavily on rating scales for these.

Let's look at the two main types, like curt scales and semantic differential scales.

Right, like curt scales use statements like the speed of the application is excellent and a discrete scale, usually from strongly agree to strongly disagree.

Semantic differential scales, on the other hand, explore bipolar attitudes.

They use pairs of opposite adjectives, like ugly attractive or confusing clear.

There's this constant debate about scale lengths, three points, five, seven, nine.

Why does that matter so much?

Well, five points are often good enough for simple like -dislike judgments.

But you might use seven or nine points when you need more subtle judgments, like gauging the precise appeal of a movie trailer.

And the odd versus even number of points is a big deal.

It is.

Odd numbers give you a neutral middle option.

Even numbers force the respondent to lean one way or the other.

But whatever you choose, consistency in your scale design is absolutely paramount.

Okay, so online distribution is obviously huge.

It's interactive.

You get automatic data validation.

But there's a critical challenge, right?

Getting a truly random sample.

That's the major trade -off.

It's very difficult to get a random sample online, which often forces researchers back into convenience sampling.

And that really limits your ability to generalize.

Got it.

Okay, onto our third pillar, observation.

This seems essential for understanding the context, the tasks, the goals of users in their own environment.

It is.

And we can split it into four quadrants, direct or indirect, and in the field or in a controlled environment.

Direct observation in the field is just invaluable for catching those nuances that self -report methods always miss.

But field work must be incredibly complex.

How do you even organize it?

It's very fluid.

So researchers use structuring frameworks to organize events.

These can be simple, like person, place, thing, or they can be super detailed, covering everything from actors and activities to objects, events, time, and feelings.

And the researcher's role can vary a lot here.

From a passive observer, just an outsider, which is better for a lab setting, to a participant observer.

Right, where the observer actively tries to become an insider in the group they're studying, all while trying to maintain professional objectivity in their notes.

And that participant observer role is central to ethnography.

It is.

Ethnography is a major type of field study.

Its distinguishing feature is that you observe without imposing a structure beforehand.

You consciously view everything that participants do as strange, so you can capture their true perspective.

The case study of the Merboard tool for the Mars Rovers team is a classic example of this.

Okay, so if we move from the field to a controlled environment, like a usability lab,

observation gets more formal.

It does.

It usually requires a strict script for standardization.

And here, the focus is on capturing the really granular detail of individual actions, often using multiple cameras.

And in this setting,

we use the think -aloud technique, asking users to just say what they're thinking as they work.

Yes.

But it has this major frustrating problem.

Users often fall completely silent the moment the task gets difficult or they hit a roadblock.

Which is the worst possible time.

So what's the fix?

The fix is constructive interaction.

You have two participants work together on the task.

It's far more natural and it's more revealing because they have to talk to each other to solve the problem rather than just talking awkwardly to themselves.

That makes sense.

Finally, indirect observation.

This is for when direct viewing is intrusive or just impossible.

Right.

This includes things like diaries, where participants record their own activities, and the experience sampling method, or ESM.

That's similar.

But it uses prompts like a text message at random times to capture immediate context and feelings.

And the big one.

Interaction logs, web analytics, and data scraping.

Software just automatically records these huge volumes of data.

The scale of the data is massive, but you need robust tools to analyze it.

And we have to always remember the ethics.

Even if it's unobtrusive, users still need to be aware that their activities are being logged.

So to bring it all together, the final decision on which method to use or combine, it comes down to four factors.

The study's focus, the participants, the required expertise and equipment, and of course, the available time and money.

That's the reality.

Interviews are great for rich exploratory data, but they can feel artificial.

Questionnaires are efficient for specific questions across large groups, but you have to watch out for low response rates.

Direct observation gives you that crucial context, but it is incredibly time consuming.

You have to balance rigor against reality.

Always.

You balance the rigor against the industrial reality of cost and time.

Which brings us back, full circle, to the challenge you laid out at the very beginning.

The difficulty of getting truthful data.

That persistent dilemma of what people say versus what they do.

And this challenge is universal.

The methods we've talked about, interviews, observation, logs, they're all designed to combat this very uncertainty.

They force us to look at the world from multiple angles.

So we'll leave you with this provocative thought.

Now that you understand the need for methodological triangulation and research, how can you apply this thinking in your own life?

Right.

When you encounter conflicting information, whether it's in news reports, social media, even personal advice, challenge yourself to find the ground truth.

Don't just rely on self -report, what someone tells you.

Try to seek out observational data, what records or actions actually show.

That's how you become a critical, informed deep diver.

Exactly.

Applying research principles to the information overload of the modern world.

Excellent point.

Thank you for joining us for this deep dive into the fundamentals of interaction design research.

We hope this provided a clear, fast path to mastering these crucial methods.

We'll catch you next time.