Statistics and experimentation skills
Statistics and Experimentation Skills-md
Statistics and experimentation skills-md are SKILL.md files that teach an AI agent to tell a real effect from noise, size a test before it runs, and read a finished experiment without fooling itself. Each one gives decision rules, thresholds and verification steps rather than vague advice, and every figure names its source.
3 of 6 skills published, version 1.0.0.
Where to start: begin with hypothesis-testing, because every experiment decision rests on reading a p-value, an effect size and a confidence interval correctly. Then size the test with sample-size-calculation, and read the result with ab-test-analysis.
Hypothesis Testing
Decides whether a difference between two or more groups is real, picking the right test and reporting an effect size and confidence interval beside the p-value, with equivalence tests and multiple-comparison corrections built in.
A/B Test Analysis
Reads a completed online A/B test and decides ship or not, checking sample ratio mismatch, guardrails, segments and peeking before the winner, with a complete decision table rather than a bare p-value.
Sample Size Calculation
Works out how many users per group a test needs and how long to run it, from baseline, MDE, alpha and power, with a validator that recomputes power and a corrected reference table where popular calculators go wrong.
No skill matches that search. Clear it or choose All.
What statistics and experimentation skills do for an AI agent
A general model can run a t-test, but left to itself it tends to quote a p-value as if it were the chance the result is wrong, call a flat test a proven tie, and stop a test the moment it looks significant. Each of those is a known way to ship a change that does nothing, or to kill one that works.
These skills encode the parts models get wrong: the right test for the data, the effect size and confidence interval that sit beside the p-value, the sample size fixed before the test starts, and the validity checks that decide whether a result can be trusted at all.
Where to start
Begin with hypothesis-testing, since it sets the frame every other skill uses. Move to sample-size-calculation before you launch anything, so the test is powered to find the effect you care about. Use ab-test-analysis to read a completed online experiment, including the sample ratio and guardrail checks that catch a broken test before its numbers mislead you.
What the agent needs from you
Give the agent the metric and what one row represents, the two or more groups being compared, and whether the data is counts, proportions or continuous values. Say what size of effect would change your decision, and whether the sample size was fixed before the test ran. For a live experiment, add the traffic split and any guardrail metrics. Without these the agent should ask, not guess.
Frequently asked questions
What is the difference between these skills and a sample size calculator?
A calculator returns one number. These skills choose the right test for your data, size it before launch, and read the finished result with the validity checks a calculator never runs, such as sample ratio mismatch and guardrail metrics. The maths is the easy part; the judgement around it is where tests go wrong.
Do I need to know statistics to use them?
No. Each skill asks for its inputs in plain terms, names the test it is running and why, and explains the result against the decision that matters to you. It will say when a result is inconclusive rather than dress up noise as a win.
Which tools do they work with?
The core of each skill is tool-agnostic. The appendices give concrete code for Python (scipy and statsmodels), with notes for R where it differs. You can follow the method in a spreadsheet, a notebook or a query, and the decision rules do not change.
Can an agent run a real experiment with these?
The skills size, run and read experiments, but they never ship, pause or reallocate on their own. Any action that affects real users or spend is left for a person to confirm. The default output is a recommendation with its reasoning, not an executed decision.
Sources and more data skills
Each skill lists its own sources with the date each was last checked, drawn from the official documentation of the tools it uses and from primary writing on experimentation, including Kohavi, Tang and Xu on trustworthy online controlled experiments and the American Statistical Association statement on p-values. For more, see the other Data sets: data analysis, dashboards and reporting and analytics and tracking.
