Survey and Interview Approaches

1:06actually produced them. Well, welcome to this deep dive. Today our mission is to peel back the curtain on that exact invisible machinery. We're diving into chapter seven, survey and interview approaches from the textbook research methods, from theory to practice. It's a fantastic chapter. Really foundational stuff. Yeah. And we're treating this like a one on one tutoring session for you, the listener. We're going to trace the whole journey of how human thoughts are quantified from the initial design choices all the way to the math that proves the data isn't just, well, random noise.

1:40By the end of this, you will never look at a data driven headline the same way again. You really won't. Because to understand how data collection works, we first have to look at what happens when it goes terribly wrong out in the wild. Let's look at that tooth fairy example you mentioned. Oh, please. I need to know the going rate for a bicuspid. Right. So in 2013, Visa actually conducted an online poll and proudly claimed that the tooth fairy was leaving an average of three dollars and seventy cents per tooth.

2:07Wow. Good for the kids. Yeah. They said it was a 42 percent jump from their 2011 data. And even more shockingly, they claimed about six percent of kids were finding a 20 dollar bill under their pillow. Wait, 20 bucks for a single tooth. That's insane. It is. And the wildest part of this whole saga is that major, reputable media outlets. I mean, we're talking the Chicago Tribune, ABC News.

2:31They just reported this as absolute fact. They brought on financial analysts to speculate about the causes. Like blaming the economy or something. Exactly. And improving economy or parents trying to keep up with the Joneses or just parental guilt over working long hours. They spent all this airtime analyzing the societal causes of the data. But nobody stopped to analyze the methodology. OK, so what was wrong with the methodology?

2:54If you look at Figure 7 .1 in the text, it shows the actual survey questions. The entire premise just falls apart. They didn't allow people to type in a specific dollar amount. They forced respondents into these really confusing preset brackets. Wait, so you couldn't just say I gave my kid a five dollar bill? Nope. You had to choose from four broad categories. Less than one dollar. One dollar exactly.

3:15Two to three dollars or more than four dollars. Hang on. I'm looking at those brackets in my head. And how on earth do you calculate an exact average of three dollars and 70 cents from that? You can't. It's mathematically impossible. Because the top category is just more than four dollars. Exactly. What number did Visa actually plug into their calculators to get that exact average? They had to make massive unscientific assumptions.

3:41If someone clicks less than one dollar, do you count that as 50 cents or zero? Right. And if someone clicks more than four dollars, maybe they left five dollars or maybe they left 50. Right. The survey's foundational structure was literally incapable of producing the precise number they published. It's like trying to calculate the exact average speed of traffic on a highway when your speedometer only has settings for slow, kind of fast, and whoa, it just doesn't work.

4:07That is a great analogy. The tool itself is broken. And this failure highlights two foundational concepts in research methods. Validity and reliability. Okay, break those down for us. So, validity asks whether a tool actually measures the specific thing it was designed to measure. Reliability asks whether that tool provides consistent, stable results. The Tooth Fairy survey failed on both fronts. The open -ended response categories destroyed its reliability and the broad framing threatened its validity.

4:38So, if the foundation is that flawed, how do real researchers know where to even begin? While they don't just wing it, they follow a strict decision diagram. Long before any data is collected, researchers complete ESSEC's training. They formulate a highly specific research question, generate testable hypotheses. So, the grammar comes first. Always. And then they choose the method. The first major fork in that road is choosing between surveys and interviews.

5:05Okay, let's go down the survey path first. Questionnaires that you just complete on your own. They seem super common, probably because they're efficient, right? Efficiency is huge. They're very economical. But the text points out that the real allure of a survey, especially for sensitive topics, is anonymity. Oh, that makes sense. Yeah, take John Schulenberg's longitudinal study on adolescent drug use. Researchers tracking high -risk behaviors need teenagers to be completely honest about highly sensitive illegal behaviors.

5:31Like heroin use, for example. I mean, a teenager's never going to look an adult researcher in the eye and admit they do heroin. Exactly. Not if they fear judgment or getting arrested. The anonymous survey removes that physical authority figure. It creates a psychological safety net, so you get much more valid data on stigmatized behaviors. But doesn't that distance also create problems? Like, you have no control over who is actually sitting there taking the survey.

5:56It creates massive problems. The dark side of surveys is bias, specifically selection bias. Going back to the Visa poll, they posted that on a website dedicated to financial literacy. Oh, well, there you go. Right. The demographic visiting a corporate financial education site is heavily skewed toward higher income, highly educated people. So the sample was fundamentally unrepresentative from day one. And even if you don't mess up the location, you still have participation biases, right?

6:24Like non -response bias. Yes, that's a huge one. Because if you send out a 40 -minute survey to parents, the parents working three jobs with a bunch of kids simply do not have the time to fill it out. Exactly. So your data is only coming from people with SOPLESS free time, which heavily skews your results. And then there's self -selection bias and motivated respondent bias. Motivated respondent bias.

6:47What's that? It's when people take a survey specifically because they want to influence the outcome. There's a 2012 study mentioned in the chapter about attitudes toward carrying concealed handguns on college campuses. Wow. OK, that's a heavily debated topic. Very. And strictly looking at the methodology here, impartially, the researchers compared students who took the survey as a mandatory part of a class against students who actively opted to take the same survey via a web link.

7:14I'm guessing the people who went out of their way to take the web survey had some pretty strong opinions. They did. The students who actively chose the web survey had systematically more extreme views than the in -class group. And it didn't matter which side of the polar vile they were on. The mere fact that they were highly motivated meant they were polarized. So the web survey acted like a filter.

7:36It only caught the loudest voices. Precisely. Now, what about the opposite problem? Careless responding, or CR. People who don't care at all, they're just rushing through to get the compensation. Just clicking randomly? Yeah, outright fraud. So researchers build traps. They use lie scales like embedding the statement, I never get angry. Well, everyone gets angry. So if you say true to that, you're either lying or just blindly clicking.

8:01They also use instructed response items like, please select, slightly disagree for this item. If you miss it, your data gets tossed. They even track IP addresses to make sure one person isn't taking it 50 times. Honestly, if people are just clicking randomly or lying, shouldn't we just look them in the eye and ask them? And that pivot leads us down the second path. Interviews. Oral questioning. If surveys are distant, interviews are intimate.

8:25Right. What are the pros there? You get incredibly rich data. You can confirm understanding. For example, if you're interviewing older adults and suspect cognitive impairment, you could administer the mini mental state examination right there. Oh, so you can test if they actually understand what's happening. Yes. And you can catch careless responding in real time. If a child is just laughing and saying yes to every question, you know the data is garbage.

8:49But interviews have to be horribly inefficient. They are massive drains on time and resources. And even with Skype or FaceTime, you lose nuance. But the biggest danger is interviewer effects. Interviewer effects? Yes. Have you ever heard of Clever Hans? Wait, the horse, the math horse? Yes. The horse that appeared to do math. His handler would ask him what three times three is, and the horse would tap his hoof nine times.

9:17Right. But he wasn't really doing math, was he? No, he was just reading the subtle unconscious physical cues of his handler. The handler would tense up and then relax exactly when the horse reached the right number. And the horse just knew to stop tapping when the guy relaxed. Exactly. Now apply that to a human interview. An interviewer might raise an eyebrow or shift their tone when a respondent gives a good answer.

9:42The respondent unconsciously picks up on that and changes their answers to please the interviewer. It's like a suspect in an interrogation room changing their story just because the detective leaned forward in their chair. Exactly. It leads to response bias, specifically acquiescence bias, where people just say yes to everything to be agreeable. OK, so surveys have bias, interviews have bias. Once you pick your poison, how do you actually come up with the questions?

10:06Well, the textbook strongly suggests borrowing before building. Do not build your own tools if existing validated ones are available. Because why reinvent the wheel, right? A great example is socioeconomic status or SES. It's a crucial control variable. Remind us what a control variable does. It's a variable you statistically remove so you can see the true effect of what you're studying. Like if you're studying the effects of divorce on children, you have to statistically control for SES because income and education also massively impact a child's outcomes.

10:40But people hate talking about their income. They lie or they hide it. Which is why researchers borrow tools like the MacArthur scale of subjective social status. Figure 7 .3 shows it. It's a visual ladder. You don't ask for a dollar amount. You show them a ladder and say the people at the top have the most money, best education, most respected jobs. Where does your family stand? Oh, so they just put an X on a room.

11:02Yes. And the insight here is that a person's perception of their social standing is often a more valid predictor of outcomes than their objective income. That is fascinating. But what if there isn't a preexisting tool for your specific nooch? What if you have to build your own? Then you have to navigate the strict rules of question wording. First, keep the reading level around eighth or ninth grade.

11:25Makes sense for accessibility. Right. Next, avoid double barreled questions like was the class interesting in the instructor engaging? Because if they say no, you have no idea which part they're rejecting. Was the class boring or was the teacher awful? Exactly. You also have to avoid loaded questions. The classic textbook example is have you stopped beating your spouse? Oh, wow. Yeah, that forces guilt either way. If you say yes, you admit you used to.

11:53If you say no, you admit you still do. Right. There's actually a comic strip example in the book where a survey implies only a big time poop head would support the mayor. You can't lead the witness like that. OK, so you get the wording right. How do you format the answers? You usually use closed ended formats like the Likert scale developed by Rinus Likert in 1932. That's the strongly agree to strongly disagree thing, right?

12:15Yes. But there's a huge debate over midpoints. If you use odd numbers like a five point scale, you give people a neutral middle ground. Some researchers prefer even numbers to force a choice. OK, but what about the ends of the stale, the extremes? The endpoints or anchors. That is where the real danger is. Let's use a critical medical example. The 10 point pain scale. Right, 10 being the worst pain imaginable.

12:39But worse imaginable is subjective. Women who have experienced the intense trauma of childbirth often use that as their internal anchor for a 10. OK, I see where this is going. So a woman comes in having a heart attack and she rates her pain as a seven because she's comparing it to childbirth. Meanwhile, a man who has never experienced that level of trauma might rate that exact same chest pain as a 10.

13:03Wow. So because the anchors are different, the doctors might literally undertreat the woman for chest pain just because of how the scale is designed. Exactly. The measurement flaw has real world consequences. That is terrifying. So say you've designed your survey, you avoided all the wording traps. How do you actually prove your tool works before you publish your results? You assess its reliability. First, you get expert feedback and run a pilot study, which is basically a pre -study.

13:29Then you run mathematical tests like test retest reliability. Which is just giving the same survey twice. But the flaw is participants might just remember their first answers and repeat them. Oh, so they just look consistent even if they aren't. Exactly. So researchers prefer parallel forms reliability. You use two different versions of the survey. Like the SAT or ACT, Form A and Form B. Yes. The items are different, but they measure the same thing.

13:57Then we have to look at internal consistency. Do the multiple questions measuring the same concept actually agree with each other? How do you test that? Split half reliability. You split the survey in half and see if the scores correlate. And there's a statistical measure for this called Cronbach's alpha. Okay, Cronbach's alpha sounds intimidating. It's just a score where the upper bound is 1 .0. Would be perfect internal consistency.

14:20You want a high score to prove your questions align. Got it. And finally, there's advanced analysis like factor analysis. There's a statistical technique that looks at relationships between items to see if they group into subscales. Like if you have a general spirituality survey. Factor analysis might mathematically split it into a self -awareness bucket and a spiritual needs bucket based on how people answer. So Cronbach's alpha and factor analysis are basically the quality control inspectors at the end of the assembly line, making sure the parts we build actually work together.

14:52That is a perfect way to put it. Yes. Wow. Okay, so summarizing all of this, we traced the whole logical chain today. From spotting flawed data like that ridiculous tooth fairy pole to weighing the biases of surveys against the clever Hans interviewer effects of face -to -face talks. And we looked at meticulously crafting and statistically validating the questions themselves. Right. Which leaves you, the listener, with a really provocative thought to mull over.

15:20If our tools for measuring the human experience are so vulnerable to a misplaced anchor on a pain scale, or a double -barreled sentence, or an interviewer's raised eyebrow, how much of what we accept as objective data -driven truth in our society is actually just a reflection of how the questions were designed. It really makes you reconsider everything you read. It really does. Well, a very warm thank you directly to you, the listener, from the Last Minute Lecture team.

15:44Keep questioning the questions and we'll catch you next time.