Analyzing Your Data I: An

1:11the black box of data analysis. We really are. We're doing a deep dive into Chapter 13 of Research Methods. From theory to practice, it's titled Analyzing Your Data Eye, An Overview. It's such a crucial chapter. It totally is. The mission here is to walk you through the exact journey that raw data takes to become a valid, scientifically sound conclusion following the precise logic of the text. Right, because as those disasters show, you know, you can have the smartest people in rooms.

1:41Exactly. But if the underlying numbers are flawed, the whole thing built on top of them just collapses. Okay, let's unpack this. Because before we get into any of the grand scientific theories or complex math, there is this incredibly unglamorous manual process that has to happen first. Oh, unglamorous is the perfect word for it. But it's the foundation of literally everything that follows. I'm talking about checking and cleaning your data.

2:05Right. This is simply the process of verifying that your measurements are complete and accurate before you run a single analysis. Like you don't even touch the calculator yet. Exactly. The chapter highlights those real -world disasters, the orbiter, the London whale, to prove that isn't just a polite suggestion for a term paper. Yeah, it's a rigid requirement. It is. The text also brings up the famous case of Harvard economists Carmen Reinhardt and Kenneth Rogoff.

2:32Oh, right. The spreadsheet thing. A graduate student uncovered a relatively simple Excel coding mistake in their work. Just a highlighting error, basically. Pretty much. And again, just looking at the mechanics of the error, that unchecked spreadsheet heavily global economic policy regarding government indebtedness. We aren't taking a stance on the politics of that policy, but the text uses it to show how a simple objective math error can ripple out into the real world.

3:02It's the ultimate garbage in, garbage out scenario. I think of it like prepping ingredients in a professional kitchen. Oh, I like that. Like you can have a Michelin star recipe, you can have the best oven in the world. Right. But if the mechanism of cooking starts with spoiled chicken, the heat isn't going to magically fix it. The cooking process just amplifies the rot, you know? Yeah, just makes hot spoiled chicken.

3:25That analogy gets at the mechanics perfectly. You are inspecting the raw material before you apply the heat as statistical analysis. You have to. But once you have verified the data is clean, you know, once the units are correct and the formulas are sound, you encounter a totally new problem. Which is? You're staring at a massive spreadsheet of thousands of individual numbers. Yeah. It's just a wall of digits.

3:47Right. And you just can't comprehend it. So you have to summarize the story the data is telling. This is where the chapter transitions into computing descriptive statistics. Which simply means summarizing what is physically there in the sample. Exactly. The chapter breaks this down into univariate and bivariate statistics. Let's tackle looking at one variable at a time, first univariate. I know I'm looking for central tendency, dispersion, and shape.

4:14Yep, those are the big three. But mechanically, what are those three things actually doing for me? Think of them as three distinct lenses. Central tendency gives you the typical score. So the mean, median, and mode. Right. It finds the middle. But the middle doesn't tell you how spread out the group is. Okay. Why does that matter? Well, if the average temperature of a city is 60 degrees, it matters a lot whether it's 60 degrees every single day, or if it wildly fluctuates between 0 and 120.

4:43Oh, wow. Yeah, that's a very different place to live. Exactly. That spread is called dispersion. To measure it, we calculate the range, the variance, and the standard deviation. Okay, got it. And then the third lens is the shape, which tells us how those numbers actually stack up visually. Spot on. Like, are they skewed heavily to one side? Or do they form a perfectly symmetrical bell curve? The chapter also mentions kurtosis here.

5:10Yes, kurtosis. It measures how sharp or flat the peak of that distribution is. Right. So if all the scores are tightly packed around the mean, you get a very tall, sharp mountain. And if they are widely dispersed, you get a flat plateau. Okay, that makes sense. So that's one variable. But when we want to understand how two variables relate to each other, we move to bivariate statistics.

5:30Right. We're looking for measures of association. And this is where the text introduces, like, a whole alphabet soup of tests. You've got Pearson's R, Spearman's R, Phi, Kramer's V. Honestly, this is where my eyes usually glaze over. I get that a lot. If I'm looking at two variables, how do I possibly know which of those specific tests to pull out of the toolbox? It's probably the most common hurdle in data analysis.

5:54The fundamental rule from the chapter is that your choice entirely depends on the underlying type of data you collected. So the data type dictates the test. Exactly. The math has to match the measurement scale. For instance, if you're comparing two continuous variables with equal intervals, like high school GPA and college GPA, you use Pearson's R's. Okay, so Pearson's R is specifically for that. Yeah, it's designed specifically to measure the strength of a linear relationship between interval or ratio data.

6:22So it's looking for a straight line correlation between numbers on a continuous scale. But what if my data isn't a continuous scale? Like what if I'm looking at class rank? You know, first place, second place, third place. The distance between first and second might be a fraction of a point, but the distance between second and third could be 10 points. The intervals aren't equal. Right. That is ordinal data or rank order data.

6:47Pearson's Rackham would totally misinterpret that because it automatically assumes equal intervals. So what do I use instead? For rank data, you have to use Spearman's Row. Okay, Spearman's Rack for ranks. And it's just, again, if you're dealing with nominal or categorical data, say you are looking at whether people prefer chocolate or vanilla and seeing if that relates to whether they prefer cats or dogs. Right. There are no numbers there.

7:11Like you can't calculate a mean average of vanilla. Exactly. They are just discrete buckets. So continuous math won't work at all. So what's the test for buckets? For two categorical variables, you count the frequencies in each bucket. If it's a simple two by two grid, you use the phi coefficient. And if it's a bigger? If you have more than two categories, like a three by three grid, you use Cramer's V.

7:33Okay, so the logic is really straightforward once you see it. Define the data type first and the data type dictates the test. Interval gets Pearson, rank order gets Spearman, categories get phi or Cramer's V. You mail it. But you know, even when we calculate these summaries, a table full of standard deviations and correlation coefficients can actually lie to you. Oh, absolutely. It can obscure massive patterns. Right.

7:58To prove it, we need to stop calculating and start looking, which is why visualizing data is the mandatory next phase in the text. Visual displays are crucial because human brains are exceptionally good at pattern recognition. Far better than looking at raw numbers. Yeah, the chapter walks through a brilliant example of this, comparing the TV watching habits of two different schools. It's a great example if you just compute the descriptive statistics like the mean, the range, the standard deviation.

8:25The numbers for school one and school two are completely identical. Which is wild. On paper, they look like the exact same population. Right. But when you plot that same data out visually using a stem and leaf plot or a histogram, the illusion just shatters. Completely. The visual reveals that school one's data is unimodal. It has a single massive peak right in the middle, with most students watching around 20 hours a week.

8:50Yep. But school two is bimodal. It has two distinct peaks. Meaning there's a cluster of kids watching very little TV and another cluster watching a massive amount. Exactly. The average is the same, but the actual behaviors of the two schools are totally different. You would be blind to that without the visual. The text also digs into box and whisker plots, which are fantastic for spotting outliers. And rather than just like guessing what an outlier is, the chapter gives us the rigid mathematical formula for it.

9:20Yeah. It's not a guessing game. A box and whisker plot shows you the median and a box that represents the middle 50 % of your data. And that's known as the interquartile range, right? The IQR. Exactly. You find the IQR by subtracting the 25th percentile from the 75th percentile. Let's walk through the response time example the text uses to make this concrete for everyone listening. The 75th percentile of response times is 1059 milliseconds.

9:45And the 25th percentile is 770 milliseconds. So 1059 minus 770 gives us an IQR of 289. That 289 is the literal width of the box. So how do we find the outliers from there? From there, the formula to find the whiskers, which are the limits of what is considered normal variance, is to multiply that IQR by 1 .5. Okay, so 289 times 1 .5 is 433 .5.

10:10Exactly. You add 433 .5 to the top of the box and subtract it from the bottom. Any data point that falls outside those specific bounds isn't just a casually high or low number. It's an official outlier. Mathematically, it is defined as an outlier, and it gets plotted as a separate distinct circle on the graph. I love how objective that is. It just removes the human bias of eyeballing a number and saying, eh, that looks a little high, let's just delete it.

10:34Yeah, you can't just delete data because it's inconvenient. Right. Now, we talked about Pearson's R earlier for bivariate data. The visual equivalent of that is a scatter plot, and the text provides a mind -bending example of why you must graph your scatter plots. It really is a striking example. You have to remember that Pearson's R only measures linear relationships, the straight line. Right. The chapter presents a scatter plot where the data points form a perfectly clear symmetrical inverted U shape.

11:00Like a hill. Yeah, as one variable goes up, the other goes up, hits a peak, and then comes back down. It is a massive, undeniably strong relationship. But then this is the crazy part. If you run Pearson's R on that data, the correlation coefficient comes back as exactly 0 .0. Yeah, zero. The math is telling you there is absolutely no relationship here, but your own eyes can see a perfect curve.

11:23The math is completely blind to the curve. Which is why relying solely on summary statistics is so dangerous. Finally, for categorical data, the text emphasizes contingency tables or cross tabulations. Oh, right, the voter study. Yeah, using a hypothetical voter study comparing two cities, the critical lesson is that you must convert raw counts into percentages. Because otherwise it's apples and oranges. Exactly. Comparing raw numbers between a city of a million people and a city of 10 ,000 is useless.

11:54Percentages provide the necessary context. Okay, so we've cleaned the data, we've summarized it, we've visualized it. We know what is physically there, but does it actually matter? That's the million dollar question. Right. A difference might exist, but is it meaningful? This leads us to affect sizes and confidence intervals. Descriptive stats don't tell us about the magnitude of a finding. That is the effect size. And the text distinguishes between raw effects and standardized effects, right?

12:21It does. A raw effect is just the simple subtraction. If group A's mean IQ is 101 and group B's is 113, the raw effect is 12 points. But I'm assuming a raw effect is dangerous because it doesn't account for the scale or the spread of the data. How so? Well, 12 points on an IQ test is huge. 12 points on the SAT is literally nothing. How do we compare them?

12:42We use a standardized effect size like Cohen's D. Cohen's D. Okay. Cohen's D adjusts the raw effect by dividing it by the standard deviation. It's like trying to compare the heights of a tall dog and a short horse. Oh, I like that. Yeah. Looking at raw inches doesn't help because they have completely different biological baselines. Standardizing them allows us to compare apples to apples. So with the IQ example, the raw difference is 12.

13:04The standard deviation of IQ is 15. You divide 12 by 15 and you get a Cohen's D of 0 .8. Exactly. We now know the groups differ by 8 tenths of a standard deviation, regardless of what the original point scale was. The text also notes the point by serial correlation, which measures the effect size between a continuous variable and a binary yes or no variable. Right. But I want to pivot to confidence intervals or CIs because I think this is the most misunderstood concept in the entire chapter.

13:33Oh, without a doubt. What's fascinating here is how almost everyone, even seasoned researchers, intuitively gets the 95 percent confidence interval wrong. The text makes a massive point of correcting this. People naturally look at an interval, say between 40 and 60, and think there is a 95 percent chance that the true population mean is inside that specific box. Right. It sounds like a probability statement about that specific result.

13:59Like, I am 95 percent sure the true number is 50, give or take 10. It's intuitive, but philosophically and statistically, it's completely false. Wait, really? Yes. The true population mean is a fixed, absolute reality. It's not moving around. So it is either inside your interval or it isn't. The probability is either 100 percent or 0 percent. Oh, wow. I never thought about it like that. Yeah. The 95 percent doesn't apply to the specific numbers you generated.

14:27It applies to the process. Wait, let me make sure I'm tracking this. The 95 percent refers to the reliability of the method over the long haul. Exactly. It means if we repeated your exact study design thousands of times, drawing new samples, calculating new means, and constructing new intervals every single time, 95 percent of the intervals we would construct will successfully capture the true population mean. That is a staggering distinction.

14:51It is. It's a statement about the track record of the tool, not the contents of the specific box you just drew. That makes so much more sense, and it perfectly sets up the next hurdle. We know what happened in our sample, but how do we prove it wasn't just a random fluke? We have to make a leap. Right. A leap from our sample to the wider population.

15:08We enter the realm of inferential statistics, specifically null hypothesis, significance testing, or NHST. NHST. This is the historical bedrock of statistical research, and it operates on a very specific, almost counterintuitive philosophy. Okay, what's step one? Step one of NHST is specifying a null hypothesis. In science, the burden of proof is on the researcher. So we start by assuming that our experiment did absolutely nothing. We assume there is zero effect, zero difference in the population.

15:40Precisely. Step two is choosing an alpha level, which is our threshold for statistical significance, almost universally set at .05 in behavioral research. Okay, so .05 is the magic number. Step three is computing the test statistic and its p -value. And step four is using that p -value to either reject or fail to reject the null hypothesis. Let's ground this philosophy with the text's real -world example, assessing the fairness of a coin.

16:05Good idea. If I flip a normal coin 20 times, I expect 10 heads and 10 tails. But what if I get 15 heads? The question is, is the coin physically rigged, or did I just witness a weird but totally natural run of luck? To answer that, we run the four steps. Step one. The null hypothesis is that the coin is perfectly fair. Step two. Alpha is .05.

16:28Step three. We calculate the p -value. Now, the logic of the p -value is critical here. How so? It calculates the probability of getting a result at least as extreme as ours, assuming the coin is totally fair. So we don't just calculate the odds of exactly 15 heads. We have to add the probability of 15 heads plus 16, 17, 18, 19, and 20 heads. Exactly. And because a rigged coin could also favor tails, we have to add the probability of 15 to 20 tails as well.

16:52So we add up the probabilities of all the extreme outcomes on both sides. And the chapter shows that when you do that math, the total probability of getting a result that extreme with a fair coin is 0 .041. Which leads directly to step four. Our calculated p -value is .041. Our predetermined alpha cutoff is .05. And because .041 is less than .05, we cross the threshold. We do.

17:18We reject the null hypothesis. The data is too extreme to comfortably believe the coin is fair. But because we are playing a game of probabilities, we can be wrong. The chapter calls these Type I and Type II errors. Right. A Type I error is a false positive. We yell, the coin is rigged! But actually it was a perfectly fair coin and we just happened to hit a genuinely rare 4 % fluke.

17:40It happens. And that error rate is tied directly to our alpha of .05. We are accepting a 5 % risk of crying wolf. Conversely, a Type II error is a false negative. The coin truly is rigged, but it only lands on heads 12 times, so we fail to detect the rigging. Which is frustrating. Yeah. And this relates to statistical power, denoted by beta, the ability of a test to actually detect an effect when it's really there.

18:03But what allows us to make these probabilistic calculations in the first place? Why can we predict the behavior of random flukes? That is the magic of the central limit theorem. The text uses the example of measuring the heights of 20 men. Can you explain why this theorem is so foundational? Imagine you randomly select 20 men, calculate their average height, and put a dot on a graph. Okay.

18:27And you grab another 20 men, find their average, and add another dot. If you do this thousands of times, the distribution of those averages will magically form a perfectly symmetrical bell curve. A normal distribution. A normal distribution. And this happens even if the underlying population's heights are wildly skewed and chaotic. So chaos at the individual level creates perfect mathematical order at the level of the averages. Exactly.

18:51Because this sampling distribution forms a predictable bell curve, we know its exact mathematical properties. We know that 95 % of all sample means will fall within 1 .96 standard deviations of the true center. Because we know the shape of the curve, we can calculate the exact probability of our specific sample occurring by chance. It is the mathematical engine of the p -value. It really is. It's an elegant system.

19:17But, and this is where the chapter takes a hard pivot, NHST has a dark side. It really does. It has been the gold standard for decades, but researchers are waking up to some deeply concerning systemic flaws in how it's used. The most dangerous flaw is treating the p -value as a measure of importance. Right. A tiny p -value does not mean you have discovered a massive world -changing effect.

19:39It literally only means you have enough data to confidently say the effect is not exactly zero. The text illustrates this beautifully with table 13 .8. It contrasts two hypothetical studies comparing teaching methods. In one study, method 2 gives students a massive 8 -point advantage over method 1. Which is a huge behavioral effect. Huge. But because the sample size is only 10 students, the p -value comes out to .09.

20:03It misses the .05 cutoff. So scientifically, it is labeled not statistically significant. We fail to reject the null. Now look at the fourth study in that exact same table. Method 2 provides a practically invisible 2 -point advantage. Barely anything. Barely anything. But because they tested 200 students, the massive sample size shrinks the margin of error. The p -value drops to .046. Oh wow. It crosses the threshold.

20:31It is officially statistically significant. We reject the null. So what does this all mean? We have a massive 8 -point improvement that science ignores as a failure, and a measly 2 -point improvement that gets published as a significant breakthrough all dictated by the sample size. It's wild. It proves the p -value is blind to the actual size and practical importance of the discovery. Not to mention the arbitrary nature of the cutoff.

20:53Why is a p -value of .049 considered a publishable scientific breakthrough, but .051 is a failure relegated to a desk drawer? Nature doesn't operate in an all -or -nothing dichotomy. No, but our statistical rules do. The chapter also outlines the dirty dozen, twelve specific fallacies researchers fall into when interpreting p -values. Three of them really highlight the confusion. Yeah, they do. First, the inverse probability fallacy. People mistakenly think the p -value is the probability that the null hypothesis is true.

21:25Which fundamentally misunderstands the math. The formula assumes the null hypothesis is 100 % true from the very beginning and calculates the odds of the data. You can't use the result to prove the premise. Right. Then there is the replication fallacy. The myth that a p -value of .05 means you have a 95 % chance of getting the exact same significant result if you run the study again.

21:47Completely false. And the odds against chance fallacy. People think a p -value of .01 means there's a 1 % probability the result was just random sampling error. Also false. But again, the calculation assumes the results were entirely caused by sampling error. It's wild how easily the bedrock metric of science is misunderstood. So if NHST is this flawed, how do we fix it? Well this brings us to the chapter's final roadgap out of a current replication crisis in science.

22:14Okay, what's the plan? The text argues for a major shift in focus. We need to stop obsessing over key values and start prioritizing effect sizes to see if findings actually matter. That makes sense. We need to report confidence intervals to show the precision of our estimates. And crucially, we must rely on replication and meta -analysis. No single study, no matter the p -value, is definitive. The chapter's Media Matters section makes it clear why this reform is so urgent.

22:42It dives into publication bias. Academic journals want sexy, innovative, statistically significant results. They do. They rarely publish studies that conclude, we tested this and nothing happened. This pressure creates two systemic toxins in the literature. First is p -hacking. Researchers desperate to publish will endlessly torture their data. It's a real problem. They drop an outlier here, change a variable there, running the numbers over and over until they magically cross that .05 threshold.

23:10The second toxin is the file drawer problem. Imagine a researcher spends a year running a study, they crunch the numbers, and the p -value is 0 .15. No significance. So it's basically a failed study in the eyes of a journal. Right. Because journals won't publish null results, the researcher literally prints out the study, stuffs it in a file drawer, and moves on. If 20 researchers do this, but the 21st researcher gets a .04 fluke and gets published, the public only sees the fluke.

23:39The literature is heavily biased because the failures are hidden away. To dismantle this system, the text highlights the growing adoption of registered reports. This is a revolutionary publishing model. How does it work? A researcher submits their introduction, their methods, and their exact data analysis plan to a journal before they collect a single piece of data. Oh, so the journal evaluates the rigor of the recipe, not the taste of the cake.

24:03Exactly. If the methodology is rock solid, the journal guarantees publication regardless of the outcome. Whether the final p -value is .01 or .80, it gets published. That's amazing. This completely eliminates the incentive to PSAC, and it solves the file drawer problem by publishing rigorous null results. It's a true maturation of the scientific process. It forces science to care about the truth of the method rather than the sexiness of the result.

24:30Well, we have covered an immense amount of ground today. We've gone from the vital necessity of inspecting your raw data for spoiled ingredients to untangling how data types dictate your descriptive stats. They've covered a lot. We've seen how visualizations can expose hidden truths, explore the philosophy of the null hypothesis, and navigated the deep flaws of the p -value and the replication crisis. It is a dense journey, for sure, but understanding this underlying statistical logic is the only reliable way to evaluate evidence and separate the signal from the noise.

25:05Here is a final provocative thought for you to ponder based on today's deep dive. If the scientific community entirely banned the use of the word significant in research papers, how would it fundamentally change the way you and I read the headlines about the latest scientific breakthroughs? That's a great question. Would we finally start asking how big is the effect instead of just, did it work? It's definitely a vital question to keep in mind the next time a flashy headline crosses your feed.

25:30Thank you for joining us for this deep dive. A warm thank you from the Last Minute Lecture Team, and we will catch you next time.