Why Running Three t-Tests Is a Bug (and How One-Way ANOVA Fixes It)

You have three groups and one question: do their averages differ?

So you do the obvious thing. You run a t-test on A vs B, another on A vs C, and a third on B vs C. One of them comes back at p = .03. You write it up, feel good, and ship it.

Congratulations, you may have just shipped a bug.

This post is about why that bug exists, how one-way ANOVA (analysis of variance) patches it, and how to run, check, and report the whole thing in practice. We'll build the test by hand, verify it in Python and R, break its assumptions on purpose to see what happens, and finish with a calculator you can use to check your own work.

TL;DR

• Many t-tests across many groups inflate your false-positive rate. ANOVA tests all groups with one test at your chosen α.
• ANOVA compares variation between group means to variation within groups. That ratio is the F-statistic.
• A significant F says "something differs". It doesn't say what. That's the job of a post-hoc test like Tukey HSD.
• Unequal variances plus unequal group sizes can quietly wreck classic ANOVA. Welch's ANOVA is the fix.
• Always report an effect size, not just p.

Part 1: The bug (a.k.a. the multiple comparisons problem)

Each t-test at α = .05 accepts a 5% chance of a false alarm when nothing is going on. One test is fine. But comparisons pile up fast:

The formula behind that table is 1 − 0.95^m, where m is the number of comparisons. But pairwise tests share data (A appears in both A–B and A–C), so they aren't independent and the real inflation is a bit smaller. Let's measure it instead of arguing. Here are four groups drawn from the same population, so any "significant" result is by definition a false alarm:

import itertools import numpy as np from scipy import stats rng = np.random.default_rng(7) trials, pairwise_fp, anova_fp = 5000, 0, 0 for _ in range(trials): groups = [rng.normal(0, 1, 15) for _ in range(4)] # no true differences  if any(stats.ttest_ind(a, b).pvalue < .05 for a, b in itertools.combinations(groups, 2)): pairwise_fp += 1 if stats.f_oneway(*groups).pvalue < .05: anova_fp += 1 print(f"any pairwise t-test 'significant': {pairwise_fp / trials:.1%}") print(f"one-way ANOVA 'significant': {anova_fp / trials:.1%}") 

With that seed I got 20.3% for the six pairwise t-tests and 4.9% for one ANOVA. A "5% error rate" test procedure that errs one time in five is not a 5% procedure.

ANOVA's pitch is simple: ask one question, "is there any difference among these means?", and answer it with one test that keeps its false-positive rate at α.

Part 2: The intuition (no formulas yet)

Picture three study groups. Each person gets an exam score. Two things make scores vary:

1. People in different groups tend to differ. Maybe practice tests really work better. That's the signal.
2. People in the same group differ from each other. Some are tired, some are lucky, some just test badly. That's the noise.

ANOVA asks whether the signal is loud compared with the noise.

Think of three tables at a party. If the tables sit at very different volumes (one is a library, one is a rock concert) while within each table everybody talks at about the same volume, the tables clearly differ. If volume varies wildly within every table, and the table averages are only slightly apart, you can't tell the tables apart from random chatter.

ANOVA turns that into a number:

F = (variation between group means) / (variation within groups) 

• F ≈ 1: the group averages differ about as much as you'd expect from noise alone. No evidence of a real difference.
• F ≫ 1: the group averages are far apart relative to the noise. Something is going on.

Under the null hypothesis, F follows an F-distribution. It's right-skewed, can't be negative, and only a large F counts as evidence, so the test is effectively one-tailed. (Under the null the average F isn't exactly 1 either. It's df_within / (df_within − 2), which is about 1.29 for our example below.)

Fun fact: the F is named after Ronald Fisher, who developed ANOVA while analyzing agricultural experiments in the 1920s. The name was later given to the ratio by George Snedecor.

Part 3: Our dataset

Twelve students, three study techniques, four students each:

Grand mean (all 12 scores): 1001 / 12 = 83.4167.

⏸️ Pause and predict: will ANOVA be significant? (click to reveal)

Almost certainly. The group means are spread over 15 points (76 to 91), while the typical within-group SD is only 3 to 4 points. The signal is several times louder than the noise. Four students per group is tiny, but the effect is huge. Keep reading to see how huge.

Part 4: The hypotheses

H₀: μ₁ = μ₂ = μ₃ (all population means are equal) Hₐ: at least one μᵢ is different (not all equal) 

ANOVA is an omnibus test. A significant result means "not all means are equal", full stop. It doesn't say which groups differ or by how much. Hold that thought for Part 7.

Part 5: Building the F-statistic by hand

ANOVA works by partitioning the total variation into two pieces. All three pieces are "sums of squares" (SS), meaning sums of squared distances.

SS_total = Σ (every score − grand mean)² SS_between = Σ nⱼ × (group mean − grand mean)² ← the signal SS_within = Σ (each score − its own group mean)² ← the noise SS_total = SS_between + SS_within 

Step 1: SS between (the signal)

SS_between = 4(83.25 − 83.4167)² + 4(76.00 − 83.4167)² + 4(91.00 − 83.4167)² = 4(0.0278) + 4(55.0069) + 4(57.5069) = 450.167 

Step 2: SS within (the noise)

Squared distances from each person to their own group's mean:

Flashcards: (82−83.25)² + (85−83.25)² + (78−83.25)² + (88−83.25)² = 54.75 Summarization: (75−76)² + (72−76)² + (80−76)² + (77−76)² = 34.00 Practice tests: (90−91)² + (92−91)² + (88−91)² + (94−91)² = 20.00 SS_within = 54.75 + 34.00 + 20.00 = 108.75 

Step 3: degrees of freedom

df_between = k − 1 = 3 − 1 = 2 df_within = N − k = 12 − 3 = 9 df_total = N − 1 = 11 

Step 4: mean squares (average squared deviation per df)

MS_between = 450.167 / 2 = 225.083 MS_within = 108.75 / 9 = 12.083 

MS_within is also just the average of the three group variances (18.25, 11.33, 6.67), a pooled estimate of the noise.

Step 5: the F-ratio

F = MS_between / MS_within = 225.083 / 12.083 = 18.63 

The ANOVA table

Sanity check: 450.167 + 108.750 = 558.917 ✓.

Where does the p-value come from?

The critical value for F with (2, 9) degrees of freedom at α = .05 is 4.256. Our F of 18.63 is more than four times that, so we reject H₀. The exact p-value is the probability of getting an F of 18.63 or larger if H₀ were true: p ≈ .0006.

Conclusion: at least one study technique produces different mean exam scores.

Now you can see why the test is called analysis of variance even though it tests means: it compares two variances.

Fun fact: ANOVA is regression in disguise

A one-way ANOVA is exactly a linear regression with a categorical predictor (dummy-coded groups), and the F-test above is the regression's overall model test. If you already know regression, you already know most of ANOVA.

Part 6: "Significant" is not the same as "big"

The p-value says whether an effect is detectable. It says nothing about size. Report an effect size alongside it.

η² (eta squared) = SS_between / SS_total = 450.167 / 558.917 = 0.805 ω² (omega squared) = (SS_between − df_between × MS_within) / (SS_total + MS_within) = (450.167 − 2 × 12.083) / (558.917 + 12.083) = 0.746 Cohen's f = √(η² / (1 − η²)) = 2.03 

• η² is the proportion of total variance explained by group membership. Here, about 80%.
• ω² is the same idea, but corrected for the upward bias that η² has in small samples. Prefer it when n is small.
• Cohen's f is a standardized effect for ANOVA. Cohen's rough conventions are 0.10 (small), 0.25 (medium) and 0.40 (large).

Notice that our f of 2.03 is absurdly large. Real-world effects are almost never this big, and that's another reminder this is a toy dataset built to make the arithmetic readable.

Part 7: Okay, but which groups differ?

The F-test said "something's different". To find out what, we run a post-hoc test that compares pairs while controlling the family-wise error rate. The most common choice with equal-ish group sizes is Tukey's HSD (Honestly Significant Difference).

For each pair, Tukey computes a studentized range statistic:

q = (mean_i − mean_j) / √(MS_within / n) 

and compares it to a critical value from the studentized range distribution, which depends on the number of groups and df_within. For k = 3 and df = 9 at α = .05, q_crit = 3.948.

A convenient shortcut: any two means further apart than the HSD threshold are significantly different.

HSD = q_crit × √(MS_within / n) = 3.948 × √(12.083 / 4) = 6.863 

Every difference exceeds 6.863, so all three pairs differ. The 95% simultaneous confidence intervals for the differences: Practice − Flashcards [0.89, 14.61], Flashcards − Summarization [0.39, 14.11], Practice − Summarization [8.14, 21.86]. None of them includes 0.

Why you can't just fall back to plain t-tests

Here's something fun. Run ordinary two-sample t-tests on the same three pairs and apply a Bonferroni correction (multiply each p by 3):

With separate t-tests plus Bonferroni, two of the three comparisons stop being significant. Tukey keeps them. Why? A separate t-test only uses the 8 observations in its two groups (6 degrees of freedom) to estimate the noise. Tukey uses the pooled MS_within from all 12 observations (9 degrees of freedom), a more stable noise estimate, which buys power. (Bonferroni-adjusted t-tests that also use the pooled MS_within land at p ≈ .049, .035 and .0005, close to Tukey.)

Moral: the post-hoc method is part of your analysis, not an afterthought. Choose it before you look at the results, and say which one you used.

Part 8: The assumptions (and what happens when they break)

Classic one-way ANOVA assumes:

1. Independence. Each observation is unrelated to the others. This is by far the most important assumption, and no test can rescue you if it's violated. Measuring the same students repeatedly? Students nested in classrooms? You need a different model (repeated-measures ANOVA, mixed models).
2. Normal residuals. The scores within each group are roughly normal. In practice, check the residuals (each score minus its group mean), pooled together.
3. Equal variances (homogeneity or homoscedasticity). The groups have similar spread.

For our data:

import numpy as np from scipy import stats A = np.array([82, 85, 78, 88]) B = np.array([75, 72, 80, 77]) C = np.array([90, 92, 88, 94]) data = [A, B, C] residuals = np.concatenate([g - g.mean() for g in data]) print(stats.shapiro(residuals)) # W ≈ 0.969, p ≈ .90 → no sign of non-normality print(stats.levene(*data)) # W ≈ 0.553, p ≈ .59 → no sign of unequal variances 

Two honest caveats. First, with n = 4 per group, both tests have almost no power to catch anything, so "not significant" here isn't reassurance. Look at plots (histograms, Q-Q plots, box plots by group). Second, a significant Levene test on a huge sample can flag harmless differences. Judge the size of the variance differences, not just the p-value.

Let's break it on purpose

ANOVA is often described as "robust", which is true under certain conditions. The condition that matters most is balanced group sizes. Watch what unequal variances do when group sizes are also unequal. Here's a Welch ANOVA implementation (it doesn't assume equal variances) followed by a simulation where every population mean is exactly 0, so any rejection is a false alarm:

import numpy as np from scipy import stats def welch_anova(*groups): k = len(groups) n = np.array([len(g) for g in groups], float) m = np.array([np.mean(g) for g in groups]) v = np.array([np.var(g, ddof=1) for g in groups]) w = n / v weighted_mean = (w * m).sum() / w.sum() a = (w * (m - weighted_mean) ** 2).sum() / (k - 1) lam = ((1 - w / w.sum()) ** 2 / (n - 1)).sum() b = 1 + 2 * (k - 2) / (k ** 2 - 1) * lam F = a / b df2 = (k ** 2 - 1) / (3 * lam) return F, k - 1, df2, stats.f.sf(F, k - 1, df2) rng = np.random.default_rng(11) sizes, sds = [5, 15, 40], [6, 3, 1] # smallest group is the noisiest trials = 10_000 classic = welch = 0 for _ in range(trials): groups = [rng.normal(0, s, n) for n, s in zip(sizes, sds)] classic += stats.f_oneway(*groups).pvalue < .05 welch += welch_anova(*groups)[3] < .05 print(f"classic ANOVA false-positive rate: {classic / trials:.1%}") print(f"Welch ANOVA false-positive rate: {welch / trials:.1%}") 

I got roughly 39.5% for classic ANOVA and 5.9% for Welch. A nominal 5% test was wrong about 40% of the time.

I also flipped the pairing so the smallest group had the smallest variance (SDs of 1, 3, 6 with the same sizes). Then classic ANOVA became badly conservative (about 0.35% false positives), which means it loses power to detect real effects, while Welch stayed near 4.8%. And with equal group sizes of 20 and SDs of 1, 2, 3, classic ANOVA drifted only to 6.3% (Welch: 5.2%).

The takeaways:

• Balanced designs are forgiving. Equal n protects you against moderate variance differences.
• Unequal n plus unequal variance is the danger zone, and the direction of the mismatch decides whether you get false alarms or lost power.
• Some methodologists recommend using Welch's ANOVA by default since it costs little when variances are equal. At minimum, use it whenever Levene's test or a box plot shows clearly unequal spread.

(Two-group sanity check for the code above: with two groups, welch_anova returns exactly the square of Welch's t-statistic from scipy.stats.ttest_ind(..., equal_var=False), with the same p-value.)

On our toy dataset, Welch's ANOVA gives F(2, 5.77) = 22.79, p = .0018. Same conclusion, slightly different numbers, because it doesn't pool the variances. If you go the Welch route, pair it with Games-Howell post-hoc comparisons instead of Tukey.

And if normality is the problem?

When the data are clearly non-normal (heavy skew, ordinal scales, small samples with outliers), the rank-based alternative is the Kruskal-Wallis test:

print(stats.kruskal(A, B, C)) # H ≈ 8.98, p ≈ .011 

Conclusion unchanged. Note that Kruskal-Wallis tests whether the groups' distributions are shifted relative to each other, which is only a statement about means under extra assumptions. Its p-value here (.011) is larger than ANOVA's (.0006) because it throws away magnitude information and keeps only ranks. A good illustration of what that costs you when the parametric assumptions are fine.

Part 9: How many people do you actually need?

Our example had 4 students per group, a demonstration, not a design. Real studies need to ask about power, the probability of detecting an effect that really exists.

from scipy.stats import f, ncf def anova_power(effect_f, k, n_per_group, alpha=0.05): N = k * n_per_group df1, df2 = k - 1, N - k crit = f.ppf(1 - alpha, df1, df2) return 1 - ncf.cdf(crit, df1, df2, effect_f ** 2 * N) print(anova_power(0.25, 3, 15)) # ≈ 0.28 print(anova_power(0.25, 3, 53)) # ≈ 0.81 

For three groups at α = .05 and 80% power:

A medium effect with 15 people per group gives you only about a 28% chance of finding it. Underpowered studies don't just miss effects. When they do hit significance, they tend to overestimate the effect size. Decide the sample size before collecting data.

Part 10: Code, from scratch and from libraries

Python: from scratch (matches the tables above)

import numpy as np from scipy import stats data = [np.array([82, 85, 78, 88]), np.array([75, 72, 80, 77]), np.array([90, 92, 88, 94])] k = len(data) N = sum(len(g) for g in data) all_scores = np.concatenate(data) grand_mean = all_scores.mean() ss_between = sum(len(g) * (g.mean() - grand_mean) ** 2 for g in data) ss_within = sum(((g - g.mean()) ** 2).sum() for g in data) ss_total = ((all_scores - grand_mean) ** 2).sum() df_between, df_within = k - 1, N - k ms_between = ss_between / df_between ms_within = ss_within / df_within F = ms_between / ms_within p = stats.f.sf(F, df_between, df_within) print(f"SS_between={ss_between:.3f} SS_within={ss_within:.3f} SS_total={ss_total:.3f}") print(f"F({df_between}, {df_within}) = {F:.3f}, p = {p:.5f}") # SS_between=450.167 SS_within=108.750 SS_total=558.917 # F(2, 9) = 18.628, p = 0.00063 

Python: library one-liners

print(stats.f_oneway(*data)) # F = 18.628, p = 0.000632 print(stats.tukey_hsd(*data)) # pairwise comparisons with p-values and 95% CIs (SciPy 1.8+) 

(If you prefer a DataFrame workflow, statsmodels offers ols(...) with anova_lm, plus pairwise_tukeyhsd, and pingouin has anova, welch_anova and pairwise_gameshowell.)

R

scores <- c(82,85,78,88, 75,72,80,77, 90,92,88,94) group <- factor(rep(c("Flashcards", "Summarization", "Practice"), each = 4)) fit <- aov(scores ~ group) summary(fit) # classic ANOVA table TukeyHSD(fit) # post-hoc oneway.test(scores ~ group, var.equal = FALSE) # Welch's ANOVA kruskal.test(scores ~ group) # rank-based alternative # car::leveneTest(scores ~ group) # equal-variance check 

Excel

1. Enable the Analysis ToolPak (File > Options > Add-ins).
2. Put each group in its own column.
3. Data > Data Analysis > Anova: Single Factor.

Excel gives you the F-test and p-value, but no post-hoc test and no Welch option, so for follow-ups you'll need other software.

SPSS

Analyze > Compare Means > One-Way ANOVA. Put the outcome in Dependent List and the grouping variable in Factor, click Post Hoc and tick Tukey (or Games-Howell if variances differ), and under Options tick Homogeneity of variance test and Welch.

Part 11: Choose your path

1. Are the observations independent? No → stop. You need repeated-measures or mixed models.
2. Are group variances reasonably similar, and are group sizes close? Yes → classic ANOVA is fine. No → Welch's ANOVA.
3. Is the outcome clearly non-normal, ordinal, or full of outliers? Yes, and the sample is small → Kruskal-Wallis, or transform the data.
4. Is F significant? No → don't go fishing with pairwise tests. Report the effect size and a confidence interval, and note that the study may be underpowered. Yes → continue.
5. Which post-hoc? Tukey (classic ANOVA, similar group sizes), Games-Howell (Welch), Bonferroni or Holm (a small set of planned comparisons).
6. Report F, df, p, an effect size, descriptives, and the post-hoc results.