Chapter 15: Statistical Evaluation of Data

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement not replaced the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

Welcome to the Deep Dive.

We take complex information, zero in on the core ideas, and unpack what really matters.

Think of us as your guides, helping you cut through the noise and get straight to the insights.

Absolutely.

Today, we're tackling something really fundamental, the statistical evaluation of data.

We're doing a deep dive into a specific chapter from Research Methods for the Behavioral Sciences, six that's the source you've shared with us.

Right.

Our mission really is to walk you through this material, pull out the key concepts, the main techniques, and focus on why they're so essential for anyone trying to make sense of research findings.

Yeah, because you get this pile of numbers from a study, right?

And this chapter is basically the instruction manual for turning those numbers into actual understanding.

It's that crucial step eight in the research process the book talks about.

Exactly.

You've done the hard work, collected the data, maybe pages and pages of scores, measurements, whatever.

Now what?

Statistics are the tools you absolutely have to have at this point.

Okay, let's unpack this then.

What's the fundamental role statistics play here?

How do they actually help?

Well, statistics really have two big jobs.

First, they help you organize and summarize all that raw data.

This is essential so you, or anyone else looking at the results, can actually see what happened.

Just making sense of the initial, maybe, mess of numbers.

Precisely.

And communicate those findings clearly.

The second job, just as critical, is helping you answer your original research questions, determining what conclusions are truly justified based on the data you've got.

And this often involves looking beyond just the specific group you studied, yes.

That's key.

And these two big jobs line up perfectly with the two main branches of statistical techniques the chapter introduces.

Ah, okay, which are?

Descriptive statistics and inferential statistics.

Descriptive statistics are all about the methods designed simply to describe a set of data you have in hand.

Like putting scores into a clear graph, making a simple table,

or calculating the average score for your group.

The goal is just to organize, summarize, simplify, make it graspable.

Exactly.

And then there are inferential statistics.

These are the methods that use the information you have from your sample, that smaller group you studied, to try and answer bigger questions about the entire population that sample came from.

Because usually researchers work with a relatively small sample, but the questions they're asking apply to a much larger group.

That's the heart of it.

And here's a core idea right from the start.

A sample is never a perfect mirror of the population.

There's always some natural difference.

There's always some sampling error, as it's called.

Sampling error.

Okay, that sounds important.

It is.

Inferential statistics are basically our toolkit for navigating that uncertainty, for figuring out when it's reasonable to generalize from our sample findings to the population.

So sampling error is kind of the reason we need inferential stats.

Got it.

Before we go deeper, the chapter clarifies some terms about samples versus populations.

Yes.

A really crucial distinction.

A statistic that's a value summarizing data for your sample, like the average score of the students you actually tested.

Okay, statistic for the sample.

And a parameter that's the corresponding value for the entire population.

The true average score for all students, like the ones you sampled, which you usually don't know directly.

Parameter for the population, okay.

So inferential statistics use those sample statistics we calculate to make educated guesses or inferences about those unknown population parameters.

It's that jump.

And the chapter stresses something important about planning.

Don't wait until you have the data.

Absolutely not.

You need to plan your statistics before you even start collecting data.

It's part of the research design.

You have to think about what conclusions you want to draw.

Which means you need to make sure you're collecting the right kind of data in the first place, yeah?

Data you can actually use for the stats you have in mind.

Exactly.

If you want to compare means, you need numerical scores you can average.

You need to anticipate the inferential tests you'll likely use.

So your whole study is set up to answer your questions statistically.

It's baked in from the start.

Okay, so the big picture is describe the sample, then infer carefully to the population.

Let's drill down into that first part.

Descriptive statistics.

How do we start making sense of a pile of raw scores?

Right.

The goal is organized and summarized.

The chapter outlines two main techniques.

First up, just organizing the whole set of scores to see the overall picture.

Like a frequency distribution.

Exactly.

A frequency distribution just shows how many times each score or category showed up in your data.

It lists the categories or scores and they count the frequency for each one.

You can do that in a table, right?

Like the quiz score example in the book.

Two columns score and how many people got it.

Simple.

Or, often more helpful visually, you use graphs.

Scores or categories usually go on the horizontal axis, frequency on the vertical.

The type of graph depends on your data.

For numerical data interval or ratio scales, you've got what?

Histograms and polygons?

Right.

A histogram uses bars.

One for each score where the height shows the frequency and the bars touch each other indicating it's a continuous scale.

Okay.

And a polygon.

That uses dots above each score at the right frequency height and then connects the dots with lines.

It forms a shape and the lines usually get drawn down to the axis at the ends.

And if the data isn't numerical,

like categories nominal or ordinal?

Then you use a bar graph.

It looks like a histogram, but you leave spaces between the bars.

That space is a visual cue that these are distinct categories, not points on a continuous number line.

Like majors in college or something.

Exactly.

Frequency distributions, especially the graphs, are fantastic for just getting an initial feel for your data.

The chapter says, you know, even if you don't publish the graph, draw one.

It helps you think.

See the shape of your data.

Okay.

Beyond just visualizing, we usually want some summary numbers, right?

Definitely.

And for numerical data, the key summaries focus on where the center is and how spread out the scores are.

Okay.

So first, central tendency.

Finding that middle or typical score.

The most common one is the mean m, the arithmetic average.

Sum up the scores, divide by how many there are.

That's the workhorse for numerical data.

Then there's the median.

What's that again?

The median is the score that literally cuts the distribution in half.

50 % of scores fall below it, 50 % above it.

It's really useful when the mean might be misleading.

Like with extreme scores, income data, or house prices.

Perfect examples.

A few billionaires can pull the average income way up, but the median income gives you a better sense of the typical person.

Makes sense.

And the third one is the mode.

The simplest one.

It's just the score or category that occurs most frequently.

The peak in your graph.

And that's the only one you can really use for nominal data, like most common major.

Right.

But it can be useful for numerical data too.

Sometimes you might have two peaks.

A bimodal distribution.

Okay.

So mean median mode.

Tell us about the center.

What about the spread?

That's variability.

How scattered are the scores around that center point?

And the main measure here is standard deviation, SD.

That's the one.

Conceptually, it's like the average distance of the scores from the mean.

So a small SD means scores are all bunched up close to the average.

And a large SD means they're spread out more widely.

There's that rule of thumb the chapter mentions.

Often around 70 % of scores are within one SD of the mean, and about 95 % within two SDs.

It gives you a feel for the distribution.

How is it related to variance, S2?

There are two sides of the same coin.

Variance is the average squared distance from the mean.

You actually calculate variance first.

Standard deviation is just the square root of the variance.

It gets you back into the original units of measurement, which is often easier to interpret than squared units.

Ah, okay.

And there's that little detail about dividing by N1 the degrees of freedom, DF, when calculating sample variance.

Yes, that's a correction factor.

Using N1 instead of just N makes the sample variance a better unbiased estimate of the population variance.

It counts for the fact that samples tend to slightly underestimate population variability.

Got it.

So for numerical data,

mean median mode for center, SD variance for spread, what if the data isn't numerical nominal or ordinal?

Well, you can't really compute a meaningful mean or standard deviation because there's no consistent numerical distance between categories.

So you use?

Proportions or percentages.

You describe the data by saying what percentage fell into each category, like 60 % preferred option A, 40 % preferred option B.

Very useful for describing a single sample or comparing groups.

The mode still applies, of course, the most frequent category.

And variability doesn't really make sense there either, right?

Not in the same way.

The concept of distance from a central point is meaningless for nominal categories.

The chapter also mentions using graphs to compare groups, like plotting the means for different conditions.

Yeah, very common way to present results, especially from experiments.

You put the different groups or treatment conditions on the horizontal axis and the statistic, usually the mean on the vertical axis.

And again, line graph versus bar graph depends on what's on that horizontal axis, right?

If the groups represent points along a numerical scale, like different drug dosages, you typically use a line graph to connect the means.

If the groups are distinct categories, like different therapy types, you use a bar graph with spaces between the bars.

Figures 15 .3 and 15 .4 show those.

And what about those more complex graphs for factorial studies with two independent variables?

Ah, yes.

For those, you typically plot the means for one factor on the horizontal axis and then use separate lines or separate sets of bars on the graph to represent the different levels of the second factor.

So you might have temperature on the bottom axis and then one line showing performance in low humidity, another line showing performance in high humidity.

Exactly.

It lets you visually inspect the means for all combinations of the factors and see if there might be an interaction effect where the effect of temperature depends on the humidity level.

Okay, shifting focus slightly.

The chapter moves to describing relationships between two variables measured on the same people.

Correlational stuff.

Right.

Big difference from comparing group means.

Here, it's one group, but you have two scores, X and Y, for each person, like their score on a stress test and their number of sick days.

And we use correlation to describe that relationship.

Yes.

A scatter plot is the first step visually.

Plot each person as a dot based on their X and Y scores.

You can often see the pattern immediately.

Does it slope up, down, is it tight or scattered?

Figure 15 .6 is a good example.

Looks pretty clear there.

And the correlation coefficient itself gives you a number summarizing that relationship.

It tells you three key things.

One,

direction.

Okay.

Plus or minus sign.

Right.

Positive means X and Y tend to move in the same direction.

Negative means they tend to move in opposite directions.

Two, form.

Is it a straight line?

Usually we're looking at linear relationships with Pearson correlation R.

If it's not linear but still consistently one directional, or if you have ranked data, you might use the Spearman correlation, RS.

And three,

strength.

How strong is the connection?

That's the numerical value.

From 0 to 1 .00, ignoring the sign.

1 .00 is a perfect relationship.

All dots on a line.

0 .00 means no consistent linear relationship.

The closer to one, or a Naga Horde, the stronger, more consistent the relationship.

His strength is separate from direction, right?

Plus 0 .80 is just as strong as Meco 8.

Exactly.

Just opposite directions.

Okay.

Correlation describes the relationship.

Then comes regression.

What's that about?

Regression is the process of finding the equation for the best fitting straight line through those dots on the scatter plot, assuming a linear relationship.

The line that minimizes the errors in prediction.

So you get an equation like y equals bx plus a?

Yep.

Where b is the slope, how much y changes for one unit change in x and a is the y -intercept, the predicted value of y when x is zero.

Regression finds the specific a and b that make that line the best fit.

And the point is prediction.

If I know x, I can predict y.

That's a major use.

Plug an x value into the equation and it gives you the predicted y value on that line.

Figure 15 .7 illustrates that.

The chapter mentions a standardized form too, beta.

Yeah, zy equals bx.

For simple linear regression, beta actually equals the Pearson correlation coefficient error.

Standardized coefficients like beta are useful because they allow for some comparison across studies using different scales.

But the prediction isn't usually perfect, right?

Unless r is 1 or net is 1.

Exactly.

There's usually some prediction error.

The dots don't all fall perfectly on the line.

How much error there is, on average, is directly related to the correlation.

Which brings us back to r -squared r2.

Right.

In the context of regression, r2 tells you the proportion of the variance in the y -scores that is accounted for or predictable by the x -scores using that regression equation.

It's a measure of the overall accuracy of the predictions.

Higher r2 means x explains more of the variation in y.

And if you use more than one predictor, like predicting GPA from both SAT scores and high school rank.

That's multiple regression.

You find an equation that uses multiple predictors, x1, x2, etc., to get the best possible prediction of y.

y equals b1x1 plus b2x2 plus a.

And the overall accuracy measure there is r2, capital R.

Correct.

r2 tells you the proportion of y variance accounted for by the combination of all the predictors in the model.

You might also see r2 delta r -squared, which tells you how much additional variance is explained when you add a new predictor to the model.

Okay, that covers descriptive stats pretty thoroughly.

Now for the other big branch in differential statistics.

The heavy hitters.

This is where we try to use our limited sample data to draw conclusions about the wider population.

And the fundamental challenge, as we mentioned, is sampling error.

That unavoidable difference between our sample statistic and the true population parameter.

Figure 15 .8 shows that we'll take two different samples from the same population, you'll get slightly different sample means, and both will probably be a bit off from the true population mean.

It's just random chance and who ends up in your sample.

So the huge question for inferential stats is, is the pattern I see in my sample, like a difference between my treatment and control groups, is that a real pattern that exists in the population, or is it just the sampling error, this random noise?

That's the core dilemma.

Figure 15 .9 poses it perfectly.

Your treatment group scores four points higher on average.

Did the treatment work, or did you just happen to get slightly higher scoring people in that group by chance?

And hypothesis tests are the formal procedures designed to help us make that call.

Exactly.

A hypothesis test uses your sample data to evaluate a claim, a hypothesis, about the population.

It tries to distinguish between two possibilities.

One, there's a real systematic effect or relationship.

Or two, what you're seeing is just random variation or sampling error.

The goal is often to rule out chance.

The chapter breaks down a hypothesis test into five key parts.

What's first?

You start with the null hypothesis.

This is a statement about the population that assumes nothing happened.

No effect, no difference, no relationship.

It posits that any pattern you see in your sample is purely due to sampling error.

So it's kind of the default assumption of no effect, and the researcher often hopes to find evidence against it.

Precisely.

You set up the null so you can try to reject it.

Second, you have the sample statistic, the actual value you calculated from your data, like the difference between your sample means or your sample correlation.

Third, the standard error.

This is crucial.

It measures the average amount of discrepancy you'd expect between a sample statistic and the population parameter just by chance, assuming the null hypothesis is true.

It quantifies the expected random wobble.

So it gives you a baseline for how much difference is normal due to chance alone.

Exactly.

Fourth, you calculate the test statistic.

This is usually a ratio that compares your actual sample result, like the observed mean difference, to the difference you'd expect by chance, based on the standard error.

So, observed difference, difference expected by chance.

Difference expected by chance.

Something like that.

That's the conceptual idea behind many test statistics, yes.

A value near zero, or one, suggests your result is consistent with chance, the null.

A large value suggests your result is much bigger than what chance would typically produce.

And fifth, the final piece, the alpha level, level of significance.

This is the decision criterion.

It's a probability value you set in advance, usually .05 or 5%, that defines what counts as extremely unlikely to occur if the null hypothesis were true.

So the logic is, if the probability of getting my sample result, or something even more extreme, just by chance, is less than my alpha level, I conclude it's too unlikely to be chance, and I reject the null hypothesis.

That's the logic, exactly.

Yeah.

If the probability, the p -value, is less than alpha, reject a null.

If p is greater than alpha, don't reject a null.

Chance remains a plausible explanation.

The coin toss example helps clarify getting way more heads than expected might have a p .05, leading you to reject the idea the coin is fair, the null.

And when we reject the null, we call the result statistically significant.

It means that the result is unlikely to be due to chance alone, based on our chosen alpha level.

We typically report this with the alpha level, the example significant at p .05, or the exact p -value if the stats software provides it, for your GP .028.

Absolutely.

Because we're dealing with probabilities and sample data, we can make two types of errors.

First is a type I error.

Rejecting the null hypothesis when it's actually true.

You conclude there is an effect in the population, but really there isn't.

You just got unlucky with an unusual sample.

It's a false positive.

And the probability of making a type I error is actually equal to the alpha level we set.

Yes.

If you set alpha .05, you have a 5 % chance of making a type I error if the null is true.

That's the trade -off.

Okay.

And the other error, type II error?

That's failing to reject the null hypothesis when it's actually false.

There is a real effect in the population, but your study failed to detect it.

It's a miss, a false negative.

This might happen if the real effect is small.

Or maybe the study didn't have enough power.

Exactly.

Small effects are harder to detect.

These errors aren't mistakes in calculation, they're inherent risks of inferential statistics.

So what influences whether we actually find a significant result?

The chapter mentions a couple of key factors.

Two big ones, especially for comparing means, are sample size and variance.

Bigger sample size makes it easier to find significance.

Generally, yes.

Larger samples give more stable and precise estimates of population parameters.

The same size effect is more likely to be statistically significant with a larger N, because the larger sample provides more convincing evidence against the null hypothesis.

It increases statistical power.

And variance, how does that play in?

This refers to the variability within your groups.

Lower variance, scores are tightly clustered around their group mean, makes it easier to see a difference between the groups.

High variance,

scores are very spread out, creates more noise and can obscure a real difference.

Figure 15 .10 shows that, well, same mean difference, but significance depends on how spread out the scores are.

High variance makes the groups overlap more, harder to distinguish.

Exactly.

High variance reduces the power of your test.

Okay, this leads to a really important point.

If significance depends so much on sample size and variance, then just finding a significant result, a low p -value, doesn't automatically mean the effect is practically important, or large, does it?

Critically important point, no it doesn't.

With a large enough sample, even a tiny trivial effect can become statistically significant.

This is why relying only on p -values can be misleading.

Which is why the chapter pushes for measures of effect size.

Absolutely.

Effect size measures tell you about the magnitude of the effect or relationship, independent of sample size.

They quantify how big the difference is, or how strong the relationship is.

One common one mentioned is Cohen's D for comparing two means.

Yes, it measures the difference between the two means in terms of standard deviation units.

Difference, standard deviation.

A D of .5 means the means are half a standard deviation apart.

Figure 15 .11 visualizes that separation, and there are benchmarks,

like .2 small, .5 medium, .8 large.

Those are the conventional guidelines, yes.

Gives you a standardized way to interpret the size.

What's the other type of effect size?

Percentage of variance?

Right.

Measures like R2 for t -tests and correlations, or .2 at a squared for N of O.

These tell you what proportion of the variability in the scores is accounted for by the group

or the relationship between variables.

So an R2 of BNU0 means that 10 % of the variance in the outcome is associated with the predictor or group difference.

Exactly.

And again, there are benchmarks, .01 small, .09 medium, .25 large for these variance accounted for measures.

They give a different perspective on the effect's practical significance.

The chapter also mentions confidence intervals.

How do they fit in?

Confidence intervals offer another way to think about the results, often complimenting hypothesis tests.

Instead of just a yes -no decision about the null, a confidence interval estimates a range of plausible values for the population parameter, like the population mean difference or correlation.

So it's centered around your sample statistic, and you calculate an interval, say, a 95 % confidence interval.

Right.

A 95 % confidence interval means we're 95 % confident that the calculated range contains the true population parameter.

It gives you a sense of the precision of your estimate.

Wider intervals mean less precision.

Yes.

Wider interval implies more uncertainty about the true value.

The width depends on the standard error.

So larger samples give narrower, more precise intervals.

And the confidence level chosen, higher confidence, like 99%, requires a wider interval.

So it gives you a range for the effect size, but it's still influenced by sample size.

It is.

While informative about magnitude, it's not a pure measure of effect size, like Cohen's D or R2, because sample size affects its width directly through the standard error.

Okay, so hypothesis tests for significance affect size for magnitude, confidence intervals for plausible range.

Using them together gives a much better picture.

Definitely a more complete understanding.

Now, with all these different statistics, means, correlations, t -tests, ANOVAR, chi -square, how on earth do researchers know which one to use?

Ah, the million dollar question.

Section 15 .4 tackles this directly.

The key is that your research strategy leads to a certain data structure.

And that structure, along with your scales of measurement, dictates the appropriate statistics.

So you need to consider the scales of measurement again, nominal, ordinal, interval, ratio.

Absolutely.

And whether your data is numerical,

interval ratio,

or categorical, nominal, ordinal, is a major decision point.

The chapter lays out three common data structures.

What's the first one?

Structure one.

One group, one score per participant.

This often comes from descriptive research just looking at a single variable,

like measuring the number of hours of sleep students get.

For this, you'd use descriptives like meanest D if numerical, median if ordinal, or mode percentages for any.

Exactly.

Inferential tests are less common, but possible.

Like a single sample test comparing your sample mean to a known value, or a chi -square goodness of fit test comparing observed category frequencies to expected ones.

Figure 15 .12 is the flowchart for this.

Okay, structure two.

One group, but two or more variables measured for each participant.

This is the setup for correlational research.

You're looking for relationships between variables within that single group, like is stress related to performance?

So here you'd use correlation coefficients, Pearson or Spearman, maybe regression?

Right for descriptives.

For inferential, you'd test if the correlation is significantly different from zero, or test the significance of the regression equation.

Or if you have categorical variables, you'd use a chi -square test for independence to see if the variables are related.

Figure 15 .43 maps this out.

And structure three.

Two or more groups being compared on the same variable.

This covers experiments, quasi -experiments, non -experiments comparing groups.

The groups could be different treatment conditions, different pre -existing groups, like experts versus novices, or the same people measured at different times within subjects.

And the statistics depend on whether it's between subjects or within subjects, and the scale of measurement.

Exactly.

If you're comparing two groups on a numerical score, you use t -tests, independent measures for between groups, repeated measures for within groups.

If you have more than two groups, or multiple factors, you move to ANOVA, analysis of variance.

Like single factor ANOVA for one grouping variable, factorial ANOVA for two or more.

Precisely.

And if you're comparing groups on a categorical outcome, like success failure,

you'd typically use a chi -square test for independence again.

Figure 15 .14 provides the detailed flowchart for navigating all these group comparison scenarios.

Those flowcharts seem incredibly useful for students trying to figure this out.

They really are.

Once you can identify your data structure and measurement scale, the flowchart points you to the appropriate statistical path.

Okay, the chapter ends with a section on some special statistics for research.

Often related to measurement quality.

Yes, particularly focusing on evaluating reliability, the consistency of your measurements.

Like split half reliability for tests, where you correlate scores on two halves of the same test.

Right, but there's a catch.

The correlation you get is based on only half the items, which usually underestimates the reliability of the full test, because longer tests tend to be more reliable.

So you need to adjust it.

Yes, using the Spearman -Brown formula, it mathematically estimates what the reliability would be for the full -length test based on the correlation between the two halves.

It almost always boosts the reliability estimate.

Well, what if there are lots of ways to split the test?

Doesn't that give different results?

Good point.

That's where other measures come in.

For items with only two answer choices, like true -false, there's the Cooder -Richardson Formula 20, KR20.

It basically estimates the average of all possible split half correlations.

And for items with more options, like Likert scales.

Then you use Cronbach's Alpha.

It's a generalization of KR20 for items with multiple response options.

Both KR20 and Cronbach's Alpha give you a single value, indicating the overall internal consistency of the items on your scale.

Higher values, closer to 1 .00, mean better internal consistency.

Okay, internal consistency.

What about reliability between observers?

Inter -rater reliability.

Yeah, how much do two independent judges agree when scoring behaviors or observations?

A simple measure is just a percentage of agreement.

Count agreements divide by total observations.

Seems straightforward.

It is, but it can be misleadingly high because some agreement will happen just by chance, especially if there are a few categories observers can choose from.

Ah, the chance factor again.

Exactly.

So a better measure is Cohen's Kappa.

It calculates agreement, but specifically corrects for the amount of agreement you'd expect purely by chance.

How does it do that?

It compares the observed agreement to the chance agreement, which you calculate based on the rates each observer uses each category and scales the result.

Kappa, observed agreement, chance agreement, one -chance agreement.

And the example in the book shows Kappa can be much lower than the simple percentage agreement.

Often significantly lower, yes, because it removes that inflation due to chance.

It gives a more conservative, realistic estimate of the true non -random agreement between the raters.

Wow, okay.

We have covered a lot of ground here.

A really comprehensive deep dive into this entire chapter on statistical evaluation.

We really have.

From the basic role of stats through descriptive techniques like means, medians, standard deviations, graphs, correlations, regression.

To the whole world of inferential statistics, sampling error, hypothesis testing, p -values, type I and type II errors.

And the absolute necessity of looking at effect sizes, like Cohen's d and r squared, plus confidence intervals.

And then navigating how to choose the right test using those flow charts based on data structure and measurement scales.

And finishing up with those specialized stats for assessing reliability, like Spearman -Brown, Crombex -Alpha, and Cohen's Kappa.

It really covers the statistical journey from raw data to drawing meaningful conclusions and evaluating your tools.

Absolutely.

And understanding these concepts, it's not just for people doing the research, right?

It's crucial for anyone trying to understand research findings they read about.

It's like having the decoder ring.

Couldn't agree more.

We've touched on all the main sections outlined in the chapter.

The role, descriptive, inferential, finding the right tests, and the special reliability stats.

Full coverage.

So here's something for you, our listener, to think about.

Next time you encounter a research finding reported with that phrase, p -value less than and maybe an effect size mention,

how might understanding sampling error, the risk of type I errors, and the meaning of measures like Cohen's, de -change how you interpret that result?

Does significant mean the same thing it did before?

Something to mull over.

Keep digging.

Keep questioning.

And join us for the next deep dive.