Analyzing Your Data II: Specific Approaches

1:10the rigid cutoff for statistical significance is 0 .05. So by the standard rules, the drug basically failed to show efficacy. But instead of just, you know, accepting that result, Harkonnen started slicing the data. He carved out a very specific subgroup of patients. I think it was those with mild to moderate cases. And he just ran the math a second time. Yeah. And in that smaller, highly specific slice of data, the p -value suddenly dropped to 0 .04.

1:37Wow. Yeah. He then put out a press release claiming the drug reduced mortality by 70 % in that specific group. But wait, wasn't the math for that specific subgroup technically calculated correctly? Technically, yes. But the presentation was deeply misleading. I mean, if you slice any large data set enough ways, you'll inevitably stumble onto like a random cluster of data points that looks statistically significant purely by chance.

2:04Right. The text actually quotes a fantastic article from the journal Nature that compares this to throwing weathermen in jail if they predicted a 40 % chance and it happened to rain. Exactly. It's a stark reminder that running data analysis isn't just about plugging numbers into a software program until you get the result you want. Yeah, it requires an underlying logic. Right. You have to know the precise mechanics of the test you're running and you have to apply it ethically.

2:29The chapter uses these great flow charts, figures 14 .1 and 14 .2 as decision trees to prevent exactly this kind of data manipulation. So before you even touch the calculator, you have to ask structural questions like what is the study design? What scale of measurement is being used? Are we comparing averages or counting votes or looking for associations? Let's start with comparing averages since that's probably the most common scenario in behavioral studies.

2:56Okay. So we're talking about continuous data here, interval or ratio scales, things like time, weight or test scores. When you want to compare the means of your foundational tool is the yet test. Right. The yet test. So like the school one data from the text figure 14 .3. Yes. Imagine a scenario where students at a specific school watch an average of 19 .5 hours of TV a week.

3:18We want to know if that 19 .5 hours is meaningfully different from a known national standard, which is 22 hours. Okay. So we're comparing one sample against a known standard. Exactly. That's a one sample t -test. We're fundamentally asking, you know, is this school's 19 .5 statistically distinct from the national 22 or did we just happen to sample a few kids who watch a ton of streaming video?

3:41Right. And to figure that out, the math requires calculating a t value, which means you need the degrees of freedom. Yes. The text specifies that the degrees of freedom here is n minus one, but I've always found that concept like a bit abstract to visualize. Why is it n minus one? Well, think of degrees of freedom as the number of independent values that can vary without breaking your calculation.

4:02Okay. So if I tell you I have 30 numbers and their final mean is exactly 19 .5, the first 29 of those numbers can be absolutely anything. They can vary wildly. Oh, I see. But the last one can't. Exactly. The moment those 29 numbers are locked in, that 30th number has no freedom whatsoever. It has to be a very specific mathematical value to force the overall average to land at exactly 19 .5.

4:26So out of 30 items, 29 are free to vary. That makes so much sense. So n minus one. In this TV example, calculating the t value with 29 degrees of freedom yields a p value of 0 .13. And because 0 .13 is way above our 0 .05 threshold, we fail to reject the null hypothesis. We just don't have enough mathematical evidence to claim this school is statistically different from the rest of the country.

4:51Correct. But the chapter also mentions we should calculate an effect size here, like Cohen's d. Right. Because p values just give you yes or no. Right. Exactly. Effect size tells us the actual magnitude of the difference, which gives you real world context. Now, what if we don't have a national standard to compare against? Like if we just have two completely separate groups? Yep. That requires an independent sample stat test.

5:12The text uses a Stroop interference task as an example. So table 14 .1 and 14 .2. 10 people are shown color words printed in conflicting ink, like the word red, printed in blue ink, and their reaction times are measured. And then 10 completely different people are given no interference. Right. And figure 14 .4 shows these box and whisker plots for the two groups, and they're pretty striking.

5:35Yeah. If you map the reaction times on a timeline, the bulk of the times for the interference group is entirely further down the timeline than the no interference group. They barely even overlap. And the math backs up that visual separation. Figure 14 .5 shows the 95 % confidence intervals. The interval for the difference between the two groups ranges from 0 .28 to 2 .0 seconds. And because that range completely steps over zero, we can be confident the groups are statistically different.

6:03Exactly. If zero was in that interval, we'd have to admit the interference might have had zero actual effect. Okay. But wait, I see a structural flaw in testing two separate groups like that. If you use 10 random people in group A and 10 completely different random people in group B, aren't you introducing immense random noise? What do you mean? Well, what if group A just happens to be populated by speed readers?

6:25Shouldn't we test the same individuals twice to control for natural ability? Oh, absolutely. That is a highly effective way to eliminate that noise. That's called a repeated measures or within subjects design. And for this, you use a matched pair set test. Table 14 .3 shows 10 students taking both a math test and a verbal test. Instead of comparing the overall average of the math scores to the overall average of the verbal scores, the math isolates the differences within each individual student.

6:55Oh, so it just asks, did student one score higher on math than virtual? Did student two? Exactly. It elegantly controls for the fact that some students are just generally stronger test takers across all subjects. That's super smart. Now, before we scale up, the chapter emphasizes some strict assumptions for all these test tests, right? The data must be interval or ratio. It has to be normally distributed. Observations must be independent.

7:18And for independent samples, the variances between groups need to be roughly equal. And if your data fails these checks, the underlying math breaks and you have to find non -parametric alternatives. Yep. Like the Wilcoxon or Mann -Whitney U tests. But assuming our data is solid, how do we handle more complex experiments? Right. Like if I want to test memory recall across people who drank herbal tea, green tea, and like three shots of espresso.

7:43Right. Three groups. I'd probably just be tempted to run three separate tests. You know, herbal versus green, green versus espresso, and herbal versus espresso. Well, doing that unleashes a major statistical hazard called the family -wise error rate. Oh, the compounding error thing? Yeah. Every time you run a statistical test, you accept a standard 5 % risk of committing a type I error, which is a false positive.

8:06If you run three separate overlapping tests on the same pool of data, those risks compound. So your chance of finding a difference that isn't actually there just skyrockets. Exactly. So to evaluate multiple groups safely, you use an analysis of variance, the ANOVA. Okay. So ANOVA is the tool for juggling three or more means. Right. The chapter breaks down a one -way between subjects in ANOVA using that exact caffeine study.

8:31Table 14 .4. So 30 participants split into no caffeine, moderate caffeine, and high caffeine, then given a memory task. And looking at Table 14 .5, the engine driving in ANOVA is this analysis of sums of squares, right? It literally compares different types of variability against each other. Exactly. It takes the variance between the three different tea groups and compares it to the baseline random variance within the groups themselves.

8:55Oh, so if the jump in memory scores between the herbal tea drinkers and the espresso drinkers is massive compared to the natural everyday memory differences among the espresso drinkers themselves, then the resulting F test gives you a significant P value. In this case, 0 .046. Okay. But this output is an omnibus null hypothesis test. I like to picture the omnibus test as like a metal detector sweeping a beach.

9:21A metal detector. Yeah. Like it beeps, alerting you that something significant is buried beneath the sand. It tells you at least one of these groups is different from the others, but it doesn't tell you where the difference is. Oh, I see what you mean. Right. It doesn't confirm if the moderate green tea group is specifically different from the high espresso group. Exactly. So how do you find out?

9:39To find the exact location of the difference, you have to do some digging, meaning you apply post -Hakkaim multiple comparison tests. Methods like Tukey's HSD or the Bonferroni correction allow you to safely compare specific pairs within your ANOVA without triggering that family -wise error rate inflation we were worried about. Makes sense. And I assume just like with TUT tests, we can run this with a repeated measures design too.

10:03Yeah. Table 14 .6 shows that. Testing the exact same participants across all three caffeine conditions to control for individual memory ability. But there's a new mathematical assumption introduced here called sphericity. Sphericity. That sounds like we're, I don't know, measuring the curvature of planets. It is a dense term, but you can conceptualize sphericity like checking the wear and tear on your car's tires. Sphericity requires that the variances of the differences between all possible pairs of conditions are roughly equal.

10:33Imagine your four tires wearing down at completely different rates. One is totally bald. Another is brand new. Right. The car begins pulling to one side and the handling gets incredibly wobbly. Exactly. If your experimental data lacks sphericity, the mathematical handling of the ANOVA gets wobbly. So you have to apply specific statistical adjustments like the greenhouse geyser correction to straighten the math back out. Okay. That makes sense.

10:57But what if the study has layers? Like say we factor in how much sleep the participants got four to five hours versus seven to eight hours layered on top of the three caffeine levels. Adding a second variable requires a two way ANOVA. Table 14 .8 introduces this. It allows us to look for two distinct outcomes. First, main effects. Like regardless of caffeine intake, does more sleep generally result in better memory?

11:24Right. And second and far more complex, we look for interaction effects. Oh, figure 14 .7. The interaction chart for this is fascinating. Yeah. What do you see there? Well, you map the data out and the trajectory lines actually cross over each other rather than running parallel. It shows that if you're in the high caffeine group, the difference in memory recall between getting a little sleep and a lot of sleep is massive.

11:45Right. But in the no caffeine group, the sleep difference barely impacts memory at all. So the one factor caffeine heavily alters the effect of the second factor sleep. Spot on. Now let's shift categories entirely. Everything we've analyzed so far acts on a sliding scale, right? Time, test scores, continuous variables. Yeah. But human behavior often doesn't fit on a scale. Sometimes it falls into distinct buckets. What if you're analyzing categorical data, like counting votes for political candidates?

12:14Well, you definitely cannot calculate the of candidate A and candidate B. Exactly. For categorical counts and frequencies, we employ the chi -square test. The text walks through a two by two contingency table, table 1 .1, comparing sex, so men and women, with preferred candidate A or B. Okay. So we aren't comparing means here at all. No, we're comparing proportions. The raw data shows 63 % of men prefer candidate A, while only 48 % of women do.

12:41The chi -square test determines if that 15 % gap is statistically significant or just random sampling variance. And the effect size for that would be the odds ratio or Kramer's V, right? Yes, exactly. And for larger tables, like table 14 .15 with four candidates, we look at absolute and relative differences via bar plots, like figure 14 .9. Okay. So we've covered categories and averages. What about mapping out associations?

13:06The text presents correlation and regression. Yeah. Correlation feels pretty intuitive. It does. We look at a scatter plot, like figure 14 .10, mapping math scores against verbal scores, and we can visually see they drift upward together. Calculating Pearson's R gives us a value of .60, which confirms a solid positive linear relationship. Right. But a correlation only identifies that a relationship exists. How do we take that relationship and use it as a predictive engine?

13:34Transforming a relationship into a prediction requires linear regression, as shown in figure 14 .11. Okay. You treat the math score as the independent variable and the verbal score as the dependent variable. Mechanically, the regression formula attempts to draw a perfectly straight, best fitting line through that scattered cloud of data points. How does it do that? It calculates the distance from every single dot to a proposed line and continuously adjusts the angle of that line until the total distance from all those dots is minimized as much as mathematically possible.

14:06Wow. So it minimizes the errors between the raw data and the prediction line. Exactly. Once that line is set, you get a regression equation with a regression coefficient. This tells you exactly how much the dependent variable, the verbal score, is expected to change for every single one -unit increase in the independent math score. And multiple regression takes this even further, right? It allows researchers to predict an outcome, like a college GPA, using several distinct variables simultaneously, balancing high school grades and standardized test scores together.

14:37It's incredibly powerful. But this all functions beautifully when the data is pristine. Textbooks rely on perfect bell -shaped curves. Right. But when a college student actually collects data for their research project, it's invariably messy. What is the protocol when those foundational assumptions, normality, equal variance, independence, just totally collapse? Well, spotting the collapse is the first step. The chapter highlights QQ Plot's figure 14 .13 for checking normality.

15:05Right. If your data set is nicely normal, the data dots will tightly hug a perfect 45 -degree diagonal line on the plot. But if your data is severely skewed, those dots bend dramatically away from the line, forming an S -shape. And you can follow that up with formal mathematical evaluations, like the Shapiro -Wilk test to verify normality, or Levine's test to check the variances, or Moshley's test for that sphericity we talked about.

15:29Okay, so once you confirm the data violates the rules of parametric tests, you adapt. Right. Take ordinal data, for example. We have table 14 .18 showing a 7 -point funniness rating for different Three Stooges comedy routines. Oh, yeah. You cannot legitimately calculate the mathematical average of slightly funny and extremely funny. No. Standard parametric tests fail completely. Instead, you deploy the Mann -Whitney U test. This method strips away the raw numbers entirely and analyzes the rank order of the data, evaluating who placed first, second, and third, rather than trying to average subjective opinions.

16:03That's a great workaround. But what if the data is basically normal, but you have massive outliers warping the mean? The notes detail robust statistics for that. Yeah, the trimmed mean. I thought this was a brilliant, pragmatic solution. You literally just chump off the top 10 % and bottom 10 % or 20 % of your data values before running the calculations. It prevents your entire analysis from being hijacked by one anomalous participant who drank like 14 espressos before the memory test.

16:34Right. But examining all these patches and corrections brings us to a much larger debate within the research community. Everything we've discussed today operates under the traditional framework of null hypothesis significance testing or NHST. But as highlighted by researchers like in the Inside Research Spotlight, NHST is facing severe criticism. I mean, some journals have gone so far as to ban the use of p -values completely. It is a massive paradigm shift.

17:01If the standard p -value is being phased out, the alternative gaining traction is Bayesian data analysis. The text uses this troop interference task again to explain this through a Bayesian framework. It involves three distinct components. Okay, what's the first one? It starts with the prior distribution. This is establishing what the scientific community already knows before the new experiment even begins. So by examining decades of previous literature, researchers know the troop interference effect typically slows reaction times by between 0 .5 and 2 .0 seconds.

17:34That prior knowledge sets a baseline. Exactly. And if we know nothing, we use an objective or minimally informative prior. The second component is the data, the actual fresh response times collected in the current experiment. Okay, and the third? The third and most critical component is calculating the posterior distribution. This is the sophisticated mathematical merging of our prior knowledge with our new data. Figure 14 .14 shows this visually.

18:00And the result of this merge is entirely different from a p -value, right? Instead of a binary reject or fail to reject conclusion, Bayesian analysis outputs credible intervals. So it explicitly states, based on everything we knew before, combined with this exact new data, there is a 95 % probability that the true difference in reaction times falls between 0 .136 and 2 .13 seconds. It directly answers the question researchers actually want to ask.

18:29Which really begs the question, if Bayesian analysis provides such a richer, more intuitive understanding of the data, why hasn't it universally replaced standard NHST? Well, two primary barriers prevent universal adoption. First, selecting the prior distribution requires a level of human judgment. Critics worry that researchers could intentionally select a prior that biases the final results in their favor. Oh yeah, that makes sense. Second, the mechanics and notorious are difficult.

18:55You can't simply plug numbers into simple standard formulas. Bayesian analysis requires complex, computer -intensive sampling techniques, generating thousands of simulated data pathways to arrive at that posterior distribution. Wow. Okay, zooming out, we have mapped out a massive landscape of statistical methodologies today. The overarching lesson is that data analysis relies on navigating a decision tree. You choose UTTest to evaluate two means, ANOVA to juggle multiple groups or variables, Chase Square for categorical counts, and Regression to build predictive models.

19:27And you rigorously test your assumptions with Q -Q plots, you adapt with robust statistics when the data is flawed, and you recognize that the entire field is slowly tilting toward a Bayesian future. And as we conclude this deep dive, I'd like to leave you with a final, provocative thought regarding that shift. If robust statistical methods and Bayesian analysis clearly offer a far more nuanced, accurate, and intuitive understanding of reality than a binary p -value ever could, why does the broader scientific establishment still fiercely demand the traditional NHSD approach?

20:03Are we, as a scientific culture, stubbornly clinging to an outdated mathematical framework simply because it is what our legacy software packages have been programmed to do best? Wow. That is a question that forces us to look right back at W. Scott Harkonnen and the illusion of certainty. Data analysis is not a machine that dispenses unquestionable truths. It is an evolving language used to navigate uncertainty with honesty and precision.

20:28Well said. Good luck on your exam. From all of us here on the Last Minute Lecture Team, you've got this.