A/B test analysis

This is the exact file agents read, including the credit header and licence notice. Save it as ab-test-analysis/SKILL.md.

---
name: ab-test-analysis
description: >
  Read a completed online A/B test and decide ship or not, by checking the test is trustworthy before looking at the winner. Covers the sample ratio mismatch (SRM) check with a chi-square test and a p < 0.0005 flag, whether the test ran to its planned sample size and full business cycles, the primary metric with an effect size and confidence interval, guardrail metrics, segment and novelty checks, multiple-comparison corrections, and a complete ship/iterate/stop decision table. Use it for "analyse my A/B test", "did the variant win", "read these experiment results", "is my test valid", "sample ratio mismatch", "SRM", "the control and variant have uneven traffic", "which version should we ship", or "my test is significant, should I roll it out". Do NOT use it to design or size a test before launch (see sample-size-calculation), to run a single significance test in isolation (see hypothesis-testing), or to build the tracking and event plan the test depends on (see event-tracking-plan).
license: Free to use and adapt under CC BY 4.0, with credit to Afeez Adebayo. Link afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and this skill's page at https://afeezadebayo.com/skills-md/data/statistics-experimentation/ab-test-analysis/.
---

# A/B test analysis

## When to use this, and when not to

Use this skill when an online experiment has finished and someone needs to decide whether to ship the variant. The triggers are a request to analyse an A/B test, read experiment results, check whether a test is valid, or turn a "significant" result into a roll-out decision. The work is to confirm the test can be trusted at all, then read the primary metric and the guardrails, and only then make a call.

Do not use it to design or size a test that has not run; that is `sample-size-calculation`, and it belongs before launch. Do not use it to run a single significance test in isolation with no experiment context; that is `hypothesis-testing`, which this skill calls for the maths. Do not use it to build the event and tracking plan the numbers come from; that is `event-tracking-plan`. An analysis is only as good as the instrumentation underneath it, so if the tracking is in doubt, fix that first.

The core idea: the first question is not "did it win" but "can I trust this result enough to act on it" (Kohavi, Tang and Xu, *Trustworthy Online Controlled Experiments*). A broken test can show a clean, significant, entirely false lift. So validity comes before the verdict, every time, and a result that fails a validity check is set aside until the cause is found, not shipped because the number looked good.

## Inputs the agent must have before starting

1. The primary metric and how it is defined, plus the guardrail metrics that must not get worse (revenue, latency, error rate, core engagement).
2. Per-variant results: users or sessions assigned, and the metric numerators and denominators for each variant. The denominator is the randomisation unit, usually users, not events.
3. The expected traffic split (for example 50/50) and the planned sample size, so the test can be checked against its own design.
4. How long the test ran, and whether it covered full business cycles (weekdays and weekends, at least one to two full weeks).
5. Whether the result was looked at and stopped early, or run to the planned sample size.
6. Any pre-registered segments, and whether segment cuts were planned in advance or are being picked after the fact.

If the traffic split, the planned sample size or the metric definitions are missing, ask. Without them the validity checks cannot run, and an analysis without validity checks is a guess with a confidence interval on it.

## The workflow, step by step

### 1. Check the sample ratio before anything else

Compare the actual split of users against the expected split with a chi-square goodness-of-fit test. If the p-value is below 0.0005, the test has a sample ratio mismatch (SRM) and the results cannot be trusted until the cause is found (Microsoft ExP, "Diagnosing sample ratio mismatch"). Microsoft reports that about 6% of their experiments have an SRM, and the missing users are usually not random, so the bias can point either way. A 50/50 test that came out 51/49 on a million users is almost certainly broken, even though 51/49 "looks" close.

Decision rule: SRM present (p < 0.0005) means stop and diagnose (assignment, logging, redirects, a variant that changes who stays). Do not report a winner from a test with an unexplained SRM.

### 2. Confirm the test actually completed

Check it ran to the planned sample size from `sample-size-calculation`, and covered at least one to two full business cycles so weekday and weekend behaviour are both represented (Optimizely; Kohavi). Check it was not stopped the moment it looked significant. A test stopped early on a good-looking peek has a false positive rate well above its stated level (Evan Miller, 2010), so an early stop is itself a reason to distrust the result.

### 3. Read the primary metric with an effect size and a confidence interval

Run the significance test for the metric type (see `hypothesis-testing` for the choice and the maths), and report the absolute difference, the relative lift, the 95% confidence interval and the p-value together. The confidence interval is the honest summary: it shows the range of true effects the data is consistent with. A significant lift whose lower confidence bound is below the smallest worthwhile effect is not yet a business win.

### 4. Check the guardrail metrics

A primary win bought with a worse guardrail is often a loss. Check the metrics that must not degrade, each with its own confidence interval: revenue per user, page latency, error rate, and the core engagement the product depends on. Decision rule: if the primary improved but a guardrail degraded beyond its agreed tolerance, do not ship on the primary alone; investigate the trade-off and escalate the decision.

### 5. Look at segments carefully, not opportunistically

Segment analysis (new versus returning, device, country) can find where an effect lives, but scanning many segments and reporting the significant ones is data dredging. Treat segment cuts as pre-registered or as exploratory, correct for the number of comparisons (see `hypothesis-testing`), and confirm any surprising segment result on fresh data before acting on it.

### 6. Check the effect is stable over time

Plot the metric by day. A lift that is large in the first days and fades is a novelty effect; a lift that only appears later may be a primacy effect as users adjust. Ship on the stable, settled effect, not the launch spike.

### 7. Make the call with a complete decision table

| Primary result | Guardrails | Decision |
|---|---|---|
| Significant positive, effect clears the worthwhile threshold | All stable | Ship, roll out in stages while monitoring |
| Significant positive | A guardrail degraded | Do not ship on the primary; investigate the trade-off |
| Significant positive, but effect below the worthwhile threshold | Stable | Usually not worth the complexity; decide on cost |
| Not significant, positive trend, test underpowered | Stable | Inconclusive; extend to the planned size or rerun, do not ship |
| Not significant, tight CI around zero | Stable | No meaningful effect; do not ship, and record the learning |
| Significant negative | Any | Do not ship; revert and find out why |
| Significant negative on primary, but a guardrail or secondary improved | Mixed | Do not ship the variant as is; isolate what helped |

### 8. Report with the trade-off in view

State the validity checks and their results first, then the primary effect with its confidence interval, the guardrails, the segments, the time trend, and the decision with its reasoning. The reader should see why the result can be trusted before they see what it says.

## Worked example, end to end

An experiment reports a +5.2% lift in sign-ups, significant at p = 0.01, and the team wants to ship. Before celebrating, the skill checks the sample ratio. The design was 50/50, but the variant received 252,100 users and the control 247,900, a total of 500,000.

```python
from scipy.stats import chisquare
observed = [252100, 247900]          # variant, control
expected = [250000, 250000]          # 50/50 of the total
chi2, p = chisquare(f_obs=observed, f_exp=expected)
print(round(chi2, 1), p)
# chi2 = 35.3, p = 2.8e-09  -> far below 0.0005: SRM present
```

The split fails the SRM check by a wide margin (p is about 3e-09), so the +5.2% lift cannot be trusted. Diagnosis shows the variant had a redirect that dropped slow mobile connections, so the extra users in the variant were systematically more engaged to begin with. The honest output is not "ship the 5.2% win"; it is "the test is invalid because of a sample ratio mismatch caused by the redirect, fix the redirect and rerun". Shipping on the raw number would have rolled out a change whose measured lift was an artefact of who was dropped, not what the change did.

## Templates you can use verbatim

```python
# 1. Sample ratio mismatch check (run this first, before any winner)
from scipy.stats import chisquare
def srm_check(n_variant, n_control, split=(0.5, 0.5)):
    total = n_variant + n_control
    expected = [total * split[0], total * split[1]]
    chi2, p = chisquare(f_obs=[n_variant, n_control], f_exp=expected)
    return {"chi2": chi2, "p": p, "srm": p < 0.0005}   # Microsoft ExP threshold
```

```python
# 2. Primary metric: lift with a 95% CI (two proportions) -- see hypothesis-testing
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
stat, p = proportions_ztest([conv_var, conv_ctrl], [n_var, n_ctrl])
low, high = confint_proportions_2indep(conv_var, n_var, conv_ctrl, n_ctrl)
lift_abs = conv_var/n_var - conv_ctrl/n_ctrl
```

```text
Decision checklist (run in order)
1. SRM check passes (p >= 0.0005)                 -> if not, STOP, diagnose
2. Ran to planned sample size and full cycles     -> if not, inconclusive
3. Not stopped early on a peek                     -> if it was, distrust
4. Primary: effect, 95% CI, p; CI lower bound vs worthwhile threshold
5. Guardrails: each with a CI, none degraded past tolerance
6. Segments: pre-registered or exploratory, corrected for comparisons
7. Effect stable over time, not a novelty spike
8. Decision from the table, with reasoning and the trade-off stated
```

## Failure modes and anti-patterns

1. **Trusting a result with a sample ratio mismatch.** A broken split produces a clean, significant, false lift. Spot it with the chi-square SRM check at p < 0.0005. Fix: stop, find the cause (redirects, logging, assignment), rerun; never ship on an unexplained SRM.
2. **Shipping on the primary while a guardrail degraded.** A sign-up lift that slows the page or cuts revenue per user is often a net loss. Spot it by reading every guardrail with its own CI. Fix: do not ship on the primary alone; weigh the trade-off and escalate.
3. **Peeking and early stopping.** A test stopped the first day it looked significant has a false positive rate far above its stated level. Spot it from a stop date well before the planned sample size. Fix: run to the planned size, or use the platform's sequential method, and treat an early stop as a reason to distrust.
4. **Data-dredged segment wins.** Scanning twenty segments and reporting the one that is significant invents a finding. Spot it when a headline result lives only in an unplanned subgroup. Fix: pre-register segments, correct for the number of comparisons, confirm on fresh data.
5. **Reading a flat result as proof the versions are equal.** A non-significant test with a wide CI could not tell; it is not evidence of no difference. Spot it from a wide confidence interval around zero. Fix: call it inconclusive, extend or rerun, or run an equivalence test (see `hypothesis-testing`).

## Quality bar

- The sample ratio was checked with a chi-square test and passed (p at or above 0.0005) before any winner was reported.
- The test ran to its planned sample size over full business cycles and was not stopped early on a peek.
- The primary metric is reported with an effect size and a 95% confidence interval, and judged against the smallest worthwhile effect, not against 0.05 alone.
- Every guardrail metric was checked with its own interval, and a primary win with a degraded guardrail was not shipped on the primary alone.
- Segment results are pre-registered or labelled exploratory and corrected for multiple comparisons, and the effect was checked for stability over time.
- The decision comes from the full table, states its reasoning, and never reads a flat result as proof of equality.

## Data handling and safety

Experiment results sit on top of user-level data, so do not log raw user records, paste them into prompts or send them to a third-party service; work with the aggregated counts the platform already produces. Shipping, pausing or rolling back a change affects real users and revenue, so it is an irreversible decision that needs explicit human confirmation: this skill produces a recommendation with its reasoning, it does not roll anything out. Never fabricate a lift, a p-value, a confidence interval or an SRM result; every figure comes from the actual experiment data. When the test touches money, health, legal standing or employment, the read is evidence for a human decision, not the decision itself, and the trade-offs and uncertainty must be shown.

## Appendix: tool-specific notes (verify before relying on these)

- **scipy.stats:** `chisquare(f_obs, f_exp)` for the SRM goodness-of-fit test; the SRM flag is p < 0.0005. Use `chi2_contingency` or `proportions_ztest` for the primary metric, and the test choices in `hypothesis-testing` for other metric types.
- **statsmodels:** `proportions_ztest`, `confint_proportions_2indep`, and `multipletests` for correcting segment and multi-metric comparisons.
- **Optimizely and VWO:** their Stats Engine uses sequential testing and always-valid p-values, so their "significance" is designed to survive continuous monitoring; read their reported significance and confidence intervals rather than running a fixed-horizon test on top of their numbers, and still run the SRM check (Optimizely docs).
- **GA4, Firebase and warehouse data:** export per-variant users and conversions by day; confirm the randomisation unit is the user, deduplicate across sessions, and reconcile the totals against the assignment log before trusting any split.
- **R:** `prop.test`, `chisq.test` for SRM, and `p.adjust` for corrections; the `pwr` package for the power follow-up in `sample-size-calculation`.

## Sources

- Kohavi, Tang and Xu, *Trustworthy Online Controlled Experiments* (2020), and Kohavi et al., "Seven Rules of Thumb for Web Site Experimenters" (KDD 2014): that trust comes before the verdict, that effects are usually small, that tests should run over full business cycles, and the use of guardrail metrics and an overall evaluation criterion. Verified 7 November 2026.
- Microsoft Experimentation Platform, "Diagnosing Sample Ratio Mismatch in A/B Testing": the chi-square SRM check, the p < 0.0005 threshold, that about 6% of experiments have an SRM, that missing users are usually not random, and the rule not to trust a result with an unexplained SRM. Verified 7 November 2026.
- Evan Miller, "How Not To Run an A/B Test" (2010): that stopping a fixed-horizon test early on a significant peek inflates the false positive rate well above the stated level. Verified 7 November 2026.
- Optimizely documentation, "How long to run an experiment" and the Stats Engine notes: running over at least one business cycle, the role of baseline rate and minimum detectable effect, and that sequential methods change how significance is read. Verified 7 November 2026.
- SciPy and statsmodels documentation: `chisquare`, `chi2_contingency`, `proportions_ztest`, `confint_proportions_2indep`, `multipletests`. Verified 7 November 2026.
- Existing skills reviewed and improved on: `phuryn` ab-test-analysis and `openclaudia` ab-test-setup name sample ratio mismatch but give no test or threshold, apply no multiple-comparison correction, give no peeking rule, and read a flat result as "no difference"; `wdavidturner` trustworthy-experiments gives the principles but ships the detailed thresholds as placeholders. This skill supplies the SRM test and threshold, the guardrail and segment handling, and a complete decision table. Verified 7 November 2026.