Sample size calculation

This is the exact file agents read, including the credit header and licence notice. Save it as sample-size-calculation/SKILL.md.

---
name: sample-size-calculation
description: >
  Work out how many observations per group an experiment needs, and how long to run it, before launch, from the baseline rate or mean, the minimum detectable effect, the significance level and the power. Covers the correct two-proportion and two-means formulas, absolute versus relative MDE, alpha and power defaults, one versus two-sided tests, adjusting for several variants, converting a sample size into a run time over full business cycles, and a validator that recomputes power so a wrong number is caught. Use it for "how many users do I need", "sample size for my A/B test", "how long should I run this test", "power analysis", "minimum detectable effect", "is my test big enough", or "what lift can I detect with this traffic". Do NOT use it to analyse a finished test (see ab-test-analysis), to run the significance test itself (see hypothesis-testing), or to compute observed power after the fact, which is not meaningful.
license: Free to use and adapt under CC BY 4.0, with credit to Afeez Adebayo. Link afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and this skill's page at https://afeezadebayo.com/skills-md/data/statistics-experimentation/sample-size-calculation/.
---

# Sample size calculation

## When to use this, and when not to

Use this skill before an experiment runs, to decide how many observations each group needs and how long the test must stay live. The triggers are "how many users do I need", "what sample size for this A/B test", "how long to run it", "power analysis", "minimum detectable effect", or "can this traffic detect the lift I care about". The work is to turn a baseline, a target effect, a significance level and a power target into a per-group sample size and a run time, and to check that number is right.

Do not use it to read a test that has already finished; that is `ab-test-analysis`. Do not use it to run the significance test itself; that is `hypothesis-testing`. And do not use it to compute "observed power" from the result of a completed test: post-hoc power is a function of the p-value and adds nothing, so it should not be reported (Hoenig and Heisey, 2001). Power analysis belongs before the data, not after.

The core idea: a test that is too small cannot find the effect you care about, so a flat result tells you nothing. The sample size is set by four numbers fixed in advance, the baseline, the minimum detectable effect, the significance level and the power, and the single most common way the figure comes out wrong is confusing an absolute effect with a relative one. Many published calculators and skills ship sample-size tables that are two to ten times off, so this skill computes the number from the formula and then validates it.

## Inputs the agent must have before starting

1. The metric type and its baseline. For a proportion, the baseline conversion rate. For a continuous metric, the baseline mean and its standard deviation (sample size for means cannot be computed without the standard deviation).
2. The minimum detectable effect (MDE), and whether it is absolute or relative. A 10% relative lift on a 5% baseline is 0.5 percentage points (to 5.5%); a 10 percentage point absolute lift is to 15%. These give wildly different sample sizes, so make the agent confirm which one is meant.
3. The significance level, alpha. Default 0.05, two-sided. Use one-sided only when the direction is committed in advance.
4. The power, one minus the Type II error rate. Default 0.80; use 0.90 for a high-stakes or hard-to-rerun test. Higher power needs more data.
5. The number of variants, including control. More than two means the alpha should be split across comparisons (see `hypothesis-testing`), which raises the sample size.
6. The daily eligible traffic and the share exposed to the test, to convert the sample size into a run time.

If the MDE basis (absolute or relative), the standard deviation for a continuous metric, or the eligible traffic is missing, ask. A sample size computed on a guessed MDE basis is not an estimate, it is a wrong answer.

## The workflow, step by step

### 1. Pin the metric and the MDE in the metric's own units

Write the baseline and the target value explicitly. For a proportion, state both as rates: baseline 5%, target 5.5% (a 10% relative lift). For a mean, state the baseline, the standard deviation and the absolute change you want to detect. Converting the MDE to the metric's own units here is what stops the absolute-versus-relative error later.

### 2. Set alpha, power and sidedness, and adjust for variants

Default to alpha 0.05 two-sided and power 0.80. Raise power to 0.90 when a wrong "no effect" is expensive. For k variants against one control, there are k comparisons, so divide alpha by k (Bonferroni) before computing the size; this raises the per-group number, which is the honest cost of testing more variants at once.

### 3. Compute the per-group sample size with the correct formula

For two proportions, per group:

n = (z<sub>1-alpha/2</sub> + z<sub>1-beta</sub>)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2

For two means, per group:

n = 2 * (z<sub>1-alpha/2</sub> + z<sub>1-beta</sub>)^2 * sigma^2 / delta^2

Compute it with a power library rather than by hand, because the library handles the effect-size transform and the exact quantiles (see the templates). As a sanity check, Evan Miller's rule of thumb is n is about 16 * sigma^2 / delta^2 per group for 80% power and 95% confidence, which for a standardised effect size d is about 16 / d^2 (Evan Miller, 2010). If the library answer and the rule of thumb disagree by more than roughly 20%, something in the inputs is wrong.

### 4. Validate the number before you trust it

Recompute the power at the sample size you got; it should come back at the power you asked for. Then compare against a second source (a reputable calculator or the reference table below). If a calculator or an existing skill gives a number more than about 10% away, do not average them: find out why, because popular sample-size tables are frequently two to ten times wrong, usually from an absolute-versus-relative MDE mix-up or a dropped power term. The validator in the templates does this check in code.

Correct per-group sizes for a two-sided test at alpha 0.05 and 80% power (computed, not quoted):

| Baseline | Relative MDE | Target | Per group |
|---|---|---|---|
| 5% | +10% | 5.5% | about 31,000 |
| 5% | +20% | 6.0% | about 8,100 |
| 10% | +10% | 11.0% | about 14,700 |
| 10% | +15% | 11.5% | about 6,700 |
| 20% | +10% | 22.0% | about 6,500 |

### 5. Turn the sample size into a run time, with a floor and a ceiling

Days needed is the total sample (per-group size times the number of groups) divided by the daily eligible users times the share exposed:

days = (n_per_group * groups) / (daily_eligible * share_exposed)

Then apply two rules. Floor: run for at least one full business cycle, and preferably two, even if the sample is reached sooner, so weekday and weekend behaviour are both captured (Optimizely; Kohavi). Ceiling: beyond about four to six weeks, novelty effects and population drift start to distort the result, so a test that needs longer than that is a signal to change the design, not to wait.

### 6. If the traffic cannot support the test, change the design

If the honest run time is longer than about eight weeks, the test as specified is not feasible. The levers, in order of preference: accept a larger MDE (detect a bigger effect), reduce the number of variants, choose a higher-signal metric (one closer to the change, with more events per user), or fall back to a qualitative method. Report the trade-off rather than quietly running an underpowered test.

## Worked example, end to end

A team wants to test a new pricing page. The baseline checkout conversion is 5%, and the smallest lift worth shipping is a 10% relative rise, to 5.5%. They will use alpha 0.05 two-sided and 80% power, with one control and one variant, split 50/50.

```python
import math
from statsmodels.stats.proportion import proportion_effectsize
from statsmodels.stats.power import NormalIndPower
p1, p2 = 0.05, 0.055                       # baseline and target (relative +10%)
es = proportion_effectsize(p2, p1)          # Cohen's h
n = NormalIndPower().solve_power(es, alpha=0.05, power=0.80,
                                 ratio=1, alternative="two-sided")
print(math.ceil(n))                         # 31218 per group
```

The answer is about 31,218 per group, so about 62,436 users in total. A popular A/B planning skill lists roughly 19,000 per variant for a 5% baseline at a 20% MDE; the correct figure for 5% at a 20% relative MDE is about 8,100, so that table is more than twice too high, which would waste weeks of traffic. With 5,000 eligible users a day at full exposure, the run time is 62,436 / 5,000, about 13 days, which the two-week business-cycle floor rounds to 14 days. The output is: 31,218 per group, run for 14 days, powered to detect a lift from 5% to 5.5% at 80% power and 95% confidence, with the assumptions stated so no one later mistakes the floor for the earliest honest stop.

## Templates you can use verbatim

```python
# Proportions: sample size per group, then validate the power
import math
from statsmodels.stats.proportion import proportion_effectsize
from statsmodels.stats.power import NormalIndPower
def n_proportion(p1, mde_rel, alpha=0.05, power=0.80, k_variants=1):
    p2 = p1 * (1 + mde_rel)                              # relative MDE
    alpha_adj = alpha / max(1, k_variants)              # Bonferroni for >1 variant
    es = proportion_effectsize(p2, p1)
    an = NormalIndPower()
    n = an.solve_power(es, alpha=alpha_adj, power=power, ratio=1,
                       alternative="two-sided")
    check = an.power(es, nobs1=math.ceil(n), alpha=alpha_adj,
                     ratio=1, alternative="two-sided")  # should ~= power
    return {"n_per_group": math.ceil(n), "achieved_power": round(check, 3)}
```

```python
# Means: per group from baseline sd and an absolute delta
import math
from statsmodels.stats.power import TTestIndPower
def n_means(sigma, delta, alpha=0.05, power=0.80):
    d = delta / sigma                                   # standardised effect
    n = TTestIndPower().solve_power(effect_size=d, alpha=alpha, power=power,
                                    ratio=1, alternative="two-sided")
    return math.ceil(n)
```

```text
Run-time and feasibility check
total = n_per_group * number_of_groups
days  = total / (daily_eligible * share_exposed)
floor = max(days, 7)            # one business cycle, prefer 14
if days > 56:  change the design (bigger MDE, fewer variants, higher-signal metric)
report: n_per_group, total, days (>= floor), alpha, power, MDE basis, assumptions
```

## Failure modes and anti-patterns

1. **Absolute versus relative MDE mix-up.** A 10% lift on a 5% baseline is 0.5 points, not 10 points; getting this wrong moves the sample size by orders of magnitude. Spot it by stating the baseline and target as explicit values. Fix: confirm the basis before computing.
2. **Trusting a calculator or table without checking.** Many published tables are two to ten times off. Spot it by recomputing the power at the suggested n, or comparing to the reference table here. Fix: use the validator; investigate any gap over about 10%.
3. **Dropping the power term.** A formula with only the alpha quantile (missing z for beta) understates the sample size and leaves the test underpowered. Spot it when the number looks suspiciously small. Fix: use a power library, which includes both terms.
4. **Ignoring the business-cycle floor.** Hitting the sample in three days and stopping misses weekend behaviour. Spot it from a run time under a week. Fix: run at least one full cycle, preferably two, whatever the maths says.
5. **Post-hoc power.** Computing power from the observed effect after the test is circular and uninformative. Spot it when "observed power" is quoted to defend a null result. Fix: report the confidence interval instead (see `ab-test-analysis`), and never compute power after the fact.

## Quality bar

- The baseline, the MDE and its basis (absolute or relative), alpha, power and sidedness are all stated before the number is computed.
- The sample size is computed with a power library, cross-checked against the rule of thumb, and validated by recomputing the achieved power.
- Any disagreement over about 10% with another calculator is explained, not averaged away.
- The run time applies a business-cycle floor of at least one week (prefer two) and flags a test that would need more than about eight weeks.
- Multiple variants split alpha, and the higher sample size that follows is reported, not hidden.
- Post-hoc power is never reported; a finished test is read with a confidence interval instead.

## Data handling and safety

Sample-size work uses summary numbers (a baseline rate, a standard deviation, a traffic figure), so it rarely touches personal data; if a baseline is estimated from raw records, do not log or send those records, compute the summary in place. The output sizes a test, it does not launch one: deciding to run an experiment, allocate traffic or delay a release is a human call, and this skill produces a plan, not an action. Never fabricate a baseline, a standard deviation or a traffic figure to make a test look feasible; if an input is missing, ask for it or label the result as conditional on an assumption. When the test informs a decision about money, health, legal standing or employment, an underpowered test can do real harm by missing a true effect, so state the power and the smallest effect the test can detect, and let a human weigh whether that is enough.

## Appendix: tool-specific notes (verify before relying on these)

- **statsmodels:** `NormalIndPower().solve_power(effect_size, alpha, power, ratio, alternative)` for proportions with `proportion_effectsize(p2, p1)` giving Cohen's h; `TTestIndPower().solve_power(effect_size=d, ...)` for means with d = delta / sigma; `GofChisquarePower` for chi-square designs. `solve_power` leaves exactly one of `effect_size`, `nobs1`, `alpha` or `power` as None and solves for it, so you can also solve for the detectable effect given a fixed sample. Recompute `power(...)` at the rounded n to validate.
- **Evan Miller's calculator and rule of thumb:** n is about 16 * sigma^2 / delta^2 per group for 80% power and 95% confidence, a quick sanity check; his online tool is a reputable second source. Verify its MDE basis matches yours.
- **Optimizely and VWO:** their sequential Stats Engine does not require a fixed sample size in advance, but a planning estimate still helps; their calculators default to a relative MDE and a baseline, so confirm the basis when comparing. Treat their duration estimates as specific to their sequential method.
- **R:** `power.prop.test(p1, p2, power, sig.level)` and `power.t.test(delta, sd, power, sig.level)` in base R, and the `pwr` package (`pwr.2p.test`, `pwr.t.test`) using Cohen's effect sizes. R's functions take the effect directly, so state p1 and p2 in their own units.
- **G*Power:** a trusted desktop tool for cross-checking; match the test family, the tails and the effect-size input exactly, or its answer will differ from the library's.

## Sources

- statsmodels documentation, `statsmodels.stats.power` (`NormalIndPower`, `TTestIndPower`, `solve_power`, `power`) and `proportion_effectsize`: the solve-for-one-parameter design, the effect-size inputs, and recomputing power to validate. Verified 7 November 2026.
- Evan Miller, "How Not To Run an A/B Test" and "Sample Size Calculator" (2010): the rule of thumb n is about 16 * sigma^2 / delta^2 per group for 80% power and 95% confidence, and that committing to a sample size in advance is what keeps the significance level honest. Verified 7 November 2026.
- Optimizely documentation, "How long to run an experiment" and the sample size calculator: the default relative MDE and baseline inputs, that higher baselines need smaller samples, that high-variance metrics need more data, and the at-least-one-business-cycle rule. Verified 7 November 2026.
- Kohavi, Tang and Xu, *Trustworthy Online Controlled Experiments* (2020): that effects are usually small (so MDEs must be realistic), that tests should cover full business cycles, and the cost of underpowered experiments. Verified 7 November 2026.
- Hoenig and Heisey, "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis" (The American Statistician, 2001): that observed or post-hoc power is a function of the p-value and should not be used to interpret a completed test. Verified 7 November 2026.
- Existing skills reviewed and improved on: the `mohitagw15856` ab-test-planner and `openclaudia` ab-test-setup skills both publish sample-size lookup tables that do not match their own stated formulas, off by factors of roughly two to ten; this skill computes the figure from a power library, ships a validator that recomputes the achieved power, and gives a corrected reference table. Verified 7 November 2026.