Hypothesis testing

This is the exact file agents read, including the credit header and licence notice. Save it as hypothesis-testing/SKILL.md.

---
name: hypothesis-testing
description: >
  Decide whether a difference between two or more groups is real or noise: pick the right significance test, interpret the p-value correctly, and report an effect size with a confidence interval, not a bare significant-or-not. Covers stating the null and alternative before seeing the data, choosing the test by data type and design (Welch and paired t-tests, two-proportion z-test, chi-square, Mann-Whitney, Wilcoxon, ANOVA, Kruskal-Wallis), checking assumptions, one versus two-sided tests, statistical versus practical significance, equivalence testing for no-difference claims, and multiple-comparison corrections. Use it for "is this difference significant", "run a t-test", "what test should I use", "is p less than 0.05", or "are these proportions different". Do NOT use it to size or power a test not yet run (see sample-size-calculation), to analyse a live experiment with traffic allocation and guardrails (see ab-test-analysis), or to measure association between continuous variables (see correlation-analysis).
license: Free to use and adapt under CC BY 4.0, with credit to Afeez Adebayo. Link afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and this skill's page at https://afeezadebayo.com/skills-md/data/statistics-experimentation/hypothesis-testing/.
---

# Hypothesis testing

## When to use this, and when not to

Use this skill when you have measured something in two or more groups and need to decide whether the difference is more than random variation. The triggers are a request to run a significance test, a question like "is this difference real", "are these two conversion rates different", "what test should I use here", or a result quoted as a p-value that needs interpreting. The work is to state the hypothesis before looking at the data, pick the test that matches the data type and design, compute the p-value together with an effect size and a confidence interval, and turn that into an honest decision.

Do not use it to work out how many observations a planned test needs; that is `sample-size-calculation`, and it comes before data collection, not after. Do not use it to analyse a running online experiment, where traffic allocation, sample ratio mismatch, guardrail metrics and peeking all matter; that is `ab-test-analysis`. Do not use it to measure how strongly two continuous variables move together (`correlation-analysis`) or to look around a fresh dataset (`exploratory-data-analysis`). Mixing these jobs is how a quick "is it significant" check turns into a decision the data cannot support.

The core idea: a p-value answers one narrow question, and only if the test was set up before the data was seen. It is the probability of a result at least this extreme if the null hypothesis were true. It is not the probability that the null is true, and it says nothing about how large or important the effect is (American Statistical Association, 2016). A result is worth acting on when the effect is big enough to matter and the data is strong enough to trust, and those are two separate checks.

## Inputs the agent must have before starting

1. The raw data or summary counts for each group, and what one row represents. A test on 1,000 sessions from 200 users is not a test on 1,000 independent users; the unit of analysis must match the unit of randomisation or sampling.
2. The metric type. Continuous (order value, time on page), binary or a proportion (converted or not), a count (events per user), or ordinal (a 1 to 5 rating) each need a different test.
3. The design: are the groups independent, or paired (the same unit measured twice, before and after)? Paired data uses a paired test and loses its power if treated as independent.
4. The hypothesis and direction, stated before the data is examined. What is the null, what is the alternative, and is the interest one-sided (only a rise counts) or two-sided (any change counts)? One-sided tests are valid only when the direction was committed in advance.
5. The significance level and the practical threshold. Default alpha is 0.05, two-sided. Separately, ask what size of effect would actually change a decision, because a difference can be statistically significant and far too small to act on.
6. Whether this is one test or many. If several metrics, segments or variants are being tested at once, the false positive risk compounds and needs correcting.

If the direction or the practical threshold is unknown, ask before running the test. Choosing a one-sided test or a "meaningful" cut-off after seeing the data invalidates the result.

## The workflow, step by step

### 1. State the hypotheses and the decision rule first

Write the null and alternative in plain terms before touching the data. The null is almost always "no difference" (the two means are equal, the two proportions are equal). Fix alpha (0.05 unless there is a reason to change it), fix whether the test is one or two-sided, and write down the effect size that would matter in practice. Committing to these in advance is what keeps the p-value meaningful; choosing them afterwards is how noise gets dressed up as a finding.

### 2. Pick the test by data type and design

Use this table as the decision rule, not a menu to browse.

| Data and design | Test | Notes |
|---|---|---|
| Two independent groups, continuous | Welch's t-test (`equal_var=False`) | Default. Do not assume equal variance |
| Two paired measurements, continuous | Paired t-test | Same unit measured twice |
| Non-normal, small samples, continuous | Mann-Whitney U (independent) or Wilcoxon signed-rank (paired) | Rank-based, no normality assumption |
| Two proportions | Two-proportion z-test or chi-square | Mathematically equivalent for a 2x2 table |
| More than two groups, categorical counts | Chi-square test of independence | Expected count at least 5 per cell |
| More than two groups, continuous | One-way ANOVA, or Kruskal-Wallis if non-normal | Follow a significant result with a corrected post-hoc test |
| Counts or rates per unit | Poisson or negative binomial test | Negative binomial when variance exceeds the mean |

Decision rules worth stating: prefer Welch's t-test over Student's by default, because assuming equal variance underestimates the p-value when the variances differ and costs almost nothing when they do not (scipy documentation). For two proportions, a z-test and a chi-square on the same 2x2 table give the same p-value, so either is fine.

### 3. Check the assumptions before trusting the output

- **Independence.** Observations must be independent within and between groups. Repeated measures on the same user break this; aggregate to the user first, or use a paired or mixed model.
- **Sample size for the approximation.** For means, the t-test is robust once each group has roughly 30 or more observations (the central limit theorem carries it); below that, normality matters and a rank test is safer. For a chi-square or two-proportion test, every expected cell count should be at least 5 (scipy documentation); below that use Fisher's exact test.
- **Variance and outliers.** A handful of extreme values can create or hide a difference in means. Inspect the distribution first, and prefer a rank-based test or a trimmed mean when the tails are heavy.

### 4. Compute the p-value, the effect size and the confidence interval together

Never report the p-value alone. Report, every time:

- the **effect size**: the raw difference (in percentage points, or in the metric's units), the relative difference where it helps, and a standardised measure (Cohen's d for means, Cohen's h for proportions) when comparing across metrics;
- the **95% confidence interval** for the difference, which shows the range of effects the data is consistent with;
- the **p-value**, interpreted against the pre-set alpha.

The confidence interval does most of the honest work. A result can be significant with a CI that runs from "trivial" to "large", which means the data has not pinned down whether the effect is worth anything.

### 5. Interpret the result properly

- **Significant (p below alpha).** The data is inconsistent with the null. Act on it only if the effect size, and the lower bound of the confidence interval, clear the practical threshold from step 1. A significant but tiny effect is a real effect that does not matter.
- **Not significant (p at or above alpha).** The data did not provide enough evidence against the null. This is not proof that the groups are equal. Report the confidence interval; if it is wide, the test was underpowered and the honest answer is "inconclusive", not "no difference".
- **When the claim is really "no difference"**, prove it with an equivalence test (two one-sided tests, TOST) against a pre-set margin, not by failing to reject the null. Absence of evidence is not evidence of absence.

### 6. Guard against the common invalidators

- **Peeking.** Repeatedly testing as data arrives and stopping at the first significant result inflates the false positive rate far above alpha (see `ab-test-analysis` and `sample-size-calculation`). Run to a pre-set sample size, or use a sequential method built for early stopping.
- **Multiple comparisons.** Testing many metrics, segments or variants multiplies the false positive risk. With m independent tests, the chance of at least one false positive is 1 minus (1 minus alpha) to the power m: 20 tests at alpha 0.05 give about a 64% chance of a bogus "winner" (Statsig, 2024). Apply a Bonferroni correction (alpha divided by m) when any false positive is costly, or Benjamini-Hochberg to control the false discovery rate when you care more about finding real effects.
- **Hypothesis chosen after the data (HARKing).** Deciding what you were testing, or which subgroup matters, after seeing the result turns exploration into a foregone conclusion. Pre-register the hypothesis, or label the finding as exploratory and confirm it on fresh data.

### 7. Report the full picture

State the test used, the sample size per group, the effect size with its 95% confidence interval, the p-value, the decision tied to the practical threshold, and which assumptions were checked. A reader should be able to see not just whether the result was significant, but how large the effect is and how much uncertainty remains.

## Worked example, end to end

A team compares checkout conversion for two page designs. Control: 1,000 users, 80 conversions (8.0%). Variant: 1,000 users, 104 conversions (10.4%). The metric is a proportion, the groups are independent, and the team wants to know whether the variant is different, so the test is a two-proportion z-test, two-sided, alpha 0.05. The practical threshold agreed in advance was a one percentage point absolute lift.

The wrong approach quotes the p-value alone and calls it a win:

```python
# p = 0.047, "significant, ship it", but this ignores effect size, CI and context
```

The honest approach reports the effect and its interval:

```python
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
count = [104, 80]        # variant first
nobs  = [1000, 1000]
stat, p = proportions_ztest(count, nobs)          # z-test on two proportions
# absolute difference and its 95% CI
low, high = confint_proportions_2indep(104, 1000, 80, 1000, method="wald")
print(round(p, 4), round(low*100, 2), round(high*100, 2))
# p = 0.0569, CI for the difference is about -0.07 to +4.87 percentage points
```

The result is not significant at 0.05 (p = 0.057), and the 95% confidence interval for the absolute lift runs from about -0.1 to +4.9 percentage points. So the data is consistent with anything from a tiny loss to a solid gain. The absolute lift seen is 2.4 points (a 30% relative rise), which clears the practical threshold, but the interval includes zero, so the honest reading is "promising, not yet proven": collect more data to the pre-set sample size rather than shipping on a borderline p-value. The report states the test, both counts, the 2.4 point observed lift, the confidence interval, the p-value and the decision to continue, so no reader mistakes a near-miss for a win or for proof of no effect.

## Templates you can use verbatim

```python
# Two independent groups, continuous metric: Welch's t-test with effect size and CI
import numpy as np
from scipy import stats
a, b = control_values, variant_values          # arrays of per-unit measurements
t, p = stats.ttest_ind(b, a, equal_var=False)   # Welch, two-sided
diff = np.mean(b) - np.mean(a)
# Cohen's d (pooled sd) for a standardised effect size
sd = np.sqrt((np.var(a, ddof=1) + np.var(b, ddof=1)) / 2)
d = diff / sd
ci = stats.ttest_ind(b, a, equal_var=False).confidence_interval()  # SciPy >= 1.11
print(round(p, 4), round(diff, 3), round(d, 3), ci)
```

```python
# Two proportions: z-test, with the absolute difference and its CI
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
stat, p = proportions_ztest([c_var, c_ctrl], [n_var, n_ctrl])
low, high = confint_proportions_2indep(c_var, n_var, c_ctrl, n_ctrl, method="wald")
```

```python
# More than two categories: chi-square, with the expected-count guard
from scipy.stats import chi2_contingency, fisher_exact
chi2, p, dof, expected = chi2_contingency(table)   # table = rows x columns of counts
if (expected < 5).any():                           # assumption broken
    _, p = fisher_exact(table)                     # use Fisher's exact for a 2x2
```

```text
Reporting template
Test: <name>, <one/two>-sided, alpha = <value>
n: control <n1>, variant <n2> (unit = <what one row is>)
Effect: <absolute difference> (<relative %>), standardised <d or h>
95% CI for the difference: <low> to <high>
p = <value>  ->  <reject / do not reject H0>
Decision: <ship / do not ship / inconclusive, collect more> because <effect vs practical threshold and CI>
Assumptions checked: <independence, sample size, variance, expected counts>
```

## Failure modes and anti-patterns

1. **Reading the p-value as the probability the null is true, or as the effect size.** A p of 0.01 does not mean a 1% chance the null holds, and a smaller p does not mean a bigger effect (ASA, 2016). Spot it when a report says "99% sure it works" from p = 0.01, or ranks results by p-value. Fix: report the effect size and confidence interval, and interpret p only as evidence against the null.
2. **Treating "not significant" as "no difference".** A wide confidence interval that includes zero means the test was underpowered, not that the groups are equal. Spot it when a flat result is declared "no effect" with a small sample. Fix: report the CI, and use an equivalence test if the claim is really no difference.
3. **The wrong test for the design.** Student's t-test when variances differ, a t-test on binary data, or an independent test on paired measurements. Spot it by matching the data type and design to the table in step 2. Fix: Welch by default, a proportion test for proportions, a paired test for paired data.
4. **Peeking and optional stopping.** Testing repeatedly and stopping at the first significant look drives the real false positive rate well above alpha. Spot it when a test was "checked daily and stopped when it hit significance". Fix: fix the sample size in advance (`sample-size-calculation`) or use a sequential method (`ab-test-analysis`).
5. **Uncorrected multiple comparisons.** Twenty metrics tested at 0.05 give roughly a two-in-three chance of a false winner. Spot it when many segments or metrics are scanned and only the significant ones are reported. Fix: apply Bonferroni or Benjamini-Hochberg, and pre-register the primary metric.

## Quality bar

- The null, alternative, alpha, direction and practical threshold were stated before the data was analysed.
- The test matches the data type and design, and its assumptions (independence, sample size, variance, expected counts) were checked.
- Every result carries an effect size and a 95% confidence interval, not a bare p-value.
- The interpretation separates statistical significance from practical importance, and never reads non-significance as proof of equality.
- Multiple comparisons are corrected, and one-sided tests are used only when the direction was pre-committed.
- The report states the test, sample sizes, effect, confidence interval, p-value and the decision rule that links them.

## Data handling and safety

Test data often sits in records with personal fields (user IDs, revenue, demographics). Do not log raw rows, paste them into prompts or send them to a third-party service; run the test with the tools you already have and work on the data in place. A significance test informs a decision, it does not make it: an irreversible action such as shipping a change, pausing a campaign or reallocating budget needs explicit human confirmation, and should never be triggered automatically on a p-value. Never fabricate a p-value, an effect size or a confidence interval; every number comes from the actual data and the stated test. When a result affects health, money, legal standing or employment, a significance test is not a substitute for domain and professional judgement: present it as evidence, add the confidence interval and the assumptions, and route the decision through a human review.

## Appendix: tool-specific notes (verify before relying on these)

- **scipy.stats:** `ttest_ind(a, b, equal_var=False)` for Welch's test (default is `equal_var=True`, which is Student's; prefer Welch), `ttest_rel` for paired, `mannwhitneyu` and `wilcoxon` for rank tests, `chi2_contingency(table, correction=True)` (Yates' correction is applied automatically when degrees of freedom equal 1), `fisher_exact` for small 2x2 tables, `f_oneway` for ANOVA, `kruskal` for its non-parametric counterpart. `ttest_ind(...).confidence_interval()` returns a CI (SciPy 1.11 and later).
- **statsmodels:** `proportions_ztest` and `confint_proportions_2indep` for proportions, `proportion_effectsize` for Cohen's h, `anova_lm` for ANOVA tables, and `multipletests` for Bonferroni and Benjamini-Hochberg corrections. For power and sample size, see `sample-size-calculation`.
- **pingouin:** a tidy wrapper that returns the test statistic, p-value, effect size and confidence interval in one call (`ttest`, `mwu`, `chi2_independence`), which keeps the full picture together by default.
- **R:** `t.test` (Welch by default, unlike Python), `wilcox.test`, `prop.test` and `chisq.test`, `fisher.test`, `aov` and `kruskal.test`, and `p.adjust` for corrections. The `effectsize` and `TOSTER` packages give effect sizes and equivalence tests.

## Sources

- American Statistical Association, "ASA Statement on Statistical Significance and P-Values" (2016): that a p-value does not measure the probability the hypothesis is true, does not measure the size or importance of an effect, and that conclusions should not rest on whether p crosses a threshold. Verified 7 November 2026.
- SciPy documentation, `scipy.stats.ttest_ind`: the `equal_var` parameter and Welch's test, the default two-sided alternative, and the `confidence_interval` method; and `scipy.stats.chi2_contingency`: the `correction` (Yates) parameter, the returned chi2, p, dof and expected table, and the expected-count-at-least-5 guideline. Verified 7 November 2026.
- statsmodels documentation, `statsmodels.stats`: `proportions_ztest`, `confint_proportions_2indep`, `proportion_effectsize` and `multipletests`. Verified 7 November 2026.
- Statsig, "The multiple comparisons problem" (2024): the family-wise error rate formula 1 minus (1 minus alpha) to the power m, the worked figure of about 64% for 20 tests at alpha 0.05, and when to use Bonferroni versus Benjamini-Hochberg. Verified 7 November 2026.
- Evan Miller, "How Not To Run an A/B Test" (2010): that significance calculations assume a sample size fixed in advance, and that peeking inflates the false positive rate. Used here for the peeking warning; the full treatment is in `ab-test-analysis` and `sample-size-calculation`. Verified 7 November 2026.
- Existing skills reviewed and improved on: `wdavidturner/product-skills` trustworthy-experiments, `openclaudia` ab-test-setup and `phuryn` ab-test-analysis each run a significance test but stop at a p-value or a yes/no flag, with no effect size, confidence interval or equivalence test; this skill adds those and the correct-test decision table. Verified 7 November 2026.