--- name: descriptive-statistics description: >- Produces honest summary statistics for one variable: it reads the distribution's shape, then reports the centre and spread that fit it (median and IQR for skewed or outlier-heavy data, mean and standard deviation only when roughly symmetric), always with the sample size, the missing count and a five-number summary, at a precision the data supports. Use when someone asks for the average, the typical value, the spread, the five-number summary, or to summarise a column: "what's the average order value", "mean and standard deviation", "give me the five-number summary", "summarise this variable", or hands you a numeric column to describe. Do NOT use for: the shape of the distribution itself (histograms, skewness, kurtosis, modality, normality: use data-distributions), a first-pass profile of a whole dataset (use exploratory-data-analysis), the relationship between two variables (use correlation-analysis), modelling, forecasting or A/B test read-outs. --- # Descriptive statistics A summary statistic is a promise: that one number stands in honestly for a whole column of values. The promise breaks the moment the number stops matching the shape of the data. The mean of a right-skewed column sits above most of the values in it. A standard deviation reported for a column with one large outlier describes the outlier more than the data. A mean taken across an ordinal scale or a set of ID codes is a number that should never have been computed. This skill produces summary statistics that keep the promise: it reads the shape first, picks the centre and the spread that fit, reports them with the sample size and the missing count, and rounds to a precision the data can actually support. The discipline is to match the statistic to the distribution, never to report a centre without a spread, and never to let a tidy number hide a messy column. ## When to use this, and when not to Use this skill when: - Someone wants the summary numbers for one variable or a handful of variables: the typical value, how spread out it is, and the key quantiles. - A report or a decision is about to quote "the average X" and the average needs to be the right average. - A first-pass profile (handed over from `exploratory-data-analysis`) flagged a column worth summarising properly. - Someone asks "what's the typical order value", "what's the average and the spread", "give me the five-number summary", or "summarise this column". Do not use this skill for: - The shape of the distribution itself: histograms, skewness and kurtosis, modality or normality. Use `data-distributions`. This skill reads the shape only far enough to pick the right centre and spread. - Profiling a whole dataset for the first time. Use `exploratory-data-analysis`. - The relationship between two variables. Use `correlation-analysis`. - Modelling, forecasting, or reading out an A/B test. ## Inputs you must have before starting Ask for anything missing. Do not assume it. 1. **The variable (or variables) and the dataset they live in.** 2. **What one row represents.** A mean across rows only means something if each row is one comparable unit. One row per order, per customer and per day give three different averages. 3. **The type of the variable:** continuous, discrete count, ordinal (ordered categories), or nominal (unordered categories or codes). The type decides which statistics are even valid. 4. **The unit and a plausible range.** Knowing the variable is a price in pounds, or an age in years, lets you catch impossible values before you summarise them. 5. **The decision behind the question.** A headline figure for a slide and an input to a pricing model need different precision and different handling of outliers. 6. **Sensitivity.** If the variable is personal, health, financial, legal or employment data, follow the PII rules below. If the person hands you a single column and nothing else, confirm the grain (input 2) and the type (input 3) before computing anything, because both change the answer. ## Workflow Work on a copy of the data. This skill does not alter the source. ### Step 1: Count what you have before you summarise it 1. Record `n`, the number of non-missing values, and the count and rate of missing values separately. Every statistic below rests on `n`, and a mean over 40 present values in a 1,000-row column is a different claim from a mean over 1,000. 2. Note that most tools compute summaries on the non-missing values silently. In pandas, the `count` row of `describe()` is the non-missing count, and statistics exclude `NaN` (pandas `DataFrame.describe`, verified 2026-10-27). Report the missing count so the reader knows what was dropped. 3. If missingness is high (over roughly 20% as a flag, not a law), say so: the summary describes the values that are present, which may not represent the column. ### Step 2: Read the shape far enough to choose Before picking a centre, look at the distribution: a quick histogram, or the output of `data-distributions`. You are not doing the full shape analysis here; you are answering one question. Is the column roughly symmetric with no heavy outliers, or is it skewed or outlier-heavy? Decision rule, using sample skewness as a quick guide (these bands are a common convention, not a law; editorworld, verified 2026-10-27): - Skewness roughly within -0.5 to 0.5, and no extreme outliers: treat as symmetric. The mean is a fair centre. - Skewness beyond about +/-1, or clear outliers: treat as skewed. The median is the honest centre and the mean will mislead. - Between the two: report both the mean and the median and let the gap speak. Skewness itself is unstable on small samples, so on fewer than about 100 rows lean on the histogram rather than the number (spcforexcel, verified 2026-10-27). ### Step 3: Choose the centre from the type and the shape | Situation | Centre to report | |---|---| | Continuous or count, roughly symmetric | Mean (report the median alongside) | | Continuous or count, skewed or outlier-heavy | Median leads; report the mean too and name the gap | | Ordinal (ordered categories) | Median or the modal category, never the mean of the codes | | Nominal (unordered categories, IDs, postcodes) | Mode and a frequency table; no mean | | Clearly two or more peaks | No single centre; report the groups and hand the shape to `data-distributions` | The reason the median leads for skewed data is mechanical: the mean is pulled toward the long tail, so for a right-skewed column the mean sits above the median and above most of the actual values. Report both and state the gap, for example "median 47, mean 84; the mean is lifted by a small number of large values". ### Step 4: Choose the spread to match the centre A centre with no spread is half a summary. Pair them, and pair them consistently. | Centre | Spread to report | |---|---| | Mean | Standard deviation (state the ddof, see below) | | Median | Interquartile range (IQR = Q3 - Q1), or the median absolute deviation | | Any skewed column | Prefer IQR; the range (max - min) is owned by a single outlier and is not a spread measure | Two traps to handle explicitly: - **The ddof trap.** The standard deviation depends on whether you divide by `n` or `n - 1`. `numpy.std` defaults to `ddof=0` (the population standard deviation, divisor `n`); pandas `Series.std` defaults to `ddof=1` (the sample standard deviation, divisor `n - 1`). The two tools give different numbers for the same column unless you set `ddof` to match (numpy `std`, verified 2026-10-27). Report the sample standard deviation (`ddof=1`) for sample data and say which you used. - **The coefficient of variation.** To compare spread across variables with different units, the coefficient of variation (standard deviation / mean) is useful, but only for ratio-scale data with a positive mean. It is undefined or meaningless when the mean is near zero or the scale has an arbitrary origin (temperature in Celsius, for example). Use it with that caveat or not at all. ### Step 5: Report the five-number summary Give the minimum, first quartile, median, third quartile and maximum. For a long-tailed column, add the 5th and 95th percentiles so the reader sees the tail without the single most extreme row. Note that quantiles vary slightly by method: `numpy.quantile` defaults to linear interpolation and offers nine documented methods, so quantiles can differ a little between tools and settings (numpy `quantile`, verified 2026-10-27). The difference is rarely material but is worth knowing when two tools disagree in the last digit. ### Step 6: Sanity-check against the unit and the range 1. Check for impossible values: negative ages, prices below zero, percentages above 100. Flag them rather than summarising over them. 2. Check for a constant or near-constant column: if every value is the same, the standard deviation is zero and the coefficient of variation is undefined. Say so instead of reporting a spread of zero as if it were a finding. 3. Check that the min and max are plausible for the unit. An implausible extreme is usually a data error or a mixed unit, and it will distort the mean and the standard deviation. ### Step 7: Write it up at a precision the data supports Report `n` and the missing count, the centre and spread matched to the shape, and the five-number summary. Round to a precision the sample can bear: a mean quoted to four decimal places on 80 rows is false precision. A good default is two or three significant figures for a headline, and the unit of the data for currency or counts. ### Step 8: Verify before handing over Recompute the headline centre and spread a second way (for example the mean by `sum / count` against the library function, or on a fresh load of the data), confirm `n` matches the non-missing count, and confirm the ddof you reported is the one you computed. If you reported a mean for a column you called skewed, that is a contradiction to fix before handing over. ## Worked example **Input:** "What's the typical order value? I want a headline number for the board deck." Data: one row per order, 2 years, column `order_value` in pounds, 2,000 rows, 60 missing. **Step 1, count.** n = 1,940 non-missing; 60 missing (3.0%). Low enough to proceed, noted. **Step 2, shape.** A quick histogram is right-skewed with a long tail of large orders; sample skewness about 2.3. Treat as skewed. **Step 3, centre.** Median leads: median = 47. Mean = 84, reported alongside because it is the figure people expect, with the gap named: the mean is lifted by a small number of large orders. **Step 4, spread.** IQR = 73 (Q1 = 22, Q3 = 95), reported with the median. The standard deviation (ddof=1) is 96, which is larger than the median itself, a sign that the mean-and-standard-deviation pair is the wrong summary here. **Step 5, five-number summary.** min = 3, Q1 = 22, median = 47, Q3 = 95, max = 2,480; 95th percentile = 310, which describes the tail far better than the single 2,480 maximum. **Write-up (shortened):** > **Order value, per order, 2 years (n = 1,940; 60 missing, 3%), 2026-10-27.** Typical order **47 (median)**, middle half between **22 and 95** (IQR). The mean is **84**, lifted by a small number of large orders, so the median is the fairer headline. Orders run from 3 to 2,480; the 95th percentile is 310. **For the board deck:** quote the median (47), not the mean (84), or the mean will overstate what a typical customer spends. ## Templates ### Single-variable summary (use verbatim) ``` # , , n (non-missing): Missing: (%) Type: Shape (quick read): Centre: () Other centre for reference: , gap: <...> Spread: Five-number summary: min <>, Q1 <>, median <>, Q3 <>, max <> Tail (if long): p5 <>, p95 <> Read in one line: Caveats: ``` ### Opening questions when inputs are thin ``` Three checks so the summary is honest, not just quick: 1. What does one row represent, and is each row one comparable unit? 2. Is this variable continuous, a count, an ordered category, or an unordered category or code? 3. What unit is it in, and what is a plausible range (so I can catch errors)? ``` ## Failure modes and anti-patterns 1. **Reporting the mean for skewed or outlier-heavy data.** The headline failure. The mean sits above most values and overstates the typical case. Spot it when the mean is well above the median. Fix it by leading with the median and the IQR, and reporting the mean only with the gap named. 2. **Reporting a centre with no spread.** A mean or median alone hides whether the values are tight or wild. Spot it when a summary has one number per variable. Fix it by always pairing the centre with a matching spread (mean with standard deviation, median with IQR). 3. **The ddof mismatch.** The same column gives two standard deviations because one tool divides by `n` and another by `n - 1`. Spot it when numbers disagree between numpy and pandas, or between a spreadsheet's STDEV and STDEVP. Fix it by reporting the sample standard deviation (`ddof=1`) and stating it. 4. **Using the range as the spread.** Max minus min is set entirely by the two most extreme rows, so it grows with sample size and one bad value owns it. Spot it when "spread" is reported as a single range on skewed data. Fix it with the IQR, and add the 5th and 95th percentiles for the tail. 5. **Averaging what should not be averaged.** The mean of a 1-to-5 satisfaction scale, of postcodes, or of category codes is a number with no meaning. Spot it when the variable is ordinal or nominal. Fix it with the median or modal category for ordinal data and a frequency table for nominal data. ## Quality bar Run this against your own output. Every item must be verifiable. - [ ] `n` (non-missing) and the missing count are both reported. - [ ] The centre matches the shape: the median leads for skewed or outlier-heavy data, the mean only for roughly symmetric data. - [ ] No ordinal or nominal variable was summarised with a mean. - [ ] Every centre is paired with a matching spread (mean with standard deviation, median with IQR). - [ ] The standard deviation states its ddof, and the sample value (`ddof=1`) was used for sample data. - [ ] The five-number summary is reported, with tail percentiles added for long-tailed columns. - [ ] Impossible values, near-constant columns and mixed units were checked and flagged. - [ ] Numbers are rounded to a precision the sample size supports; no false precision. - [ ] The headline centre and spread were reproduced a second way and `n` matches the non-missing count. - [ ] If the variable is sensitive, no raw personal values were printed or sent anywhere. ## Guardrails - **Do not fabricate.** Every statistic comes from the data. Conventions (the skewness bands, the missingness flag, rounding choices) are labelled as conventions with a source and a date, not presented as facts about this dataset. - **Work on a copy; confirm before changing data.** Removing outliers or recoding a variable before summarising is a change the person approves first. Report with and without rather than silently dropping rows. - **PII and credentials.** If the variable is personal, health, financial, legal or employment data, never print raw values, never paste them into a third-party service, and keep them in the person's own environment. Report the summary, not the records. Summaries of protected attributes (age, sex, race, disability and the like) can feed discrimination; flag the fairness and legal risk and route decisions to a human. - **Not advice.** When the variable concerns health, money, legal standing or employment, the output is a statistical summary only. It is not medical, financial, legal or HR advice, and any decision needs a qualified human. - **Scope.** This skill summarises the centre and spread of one variable. For the shape of the distribution (histograms, skewness, kurtosis, modality, normality) use `data-distributions`; for two variables use `correlation-analysis`; for a whole-dataset first pass use `exploratory-data-analysis`. It will not model, forecast, or make the decision. - **Cost.** Summary statistics are cheap and local. There is no API cost and no reason to sample; compute on the full column. ## Tool appendix (Python) ``` This section is tool-specific. The workflow above is tool-agnostic and works the same in R, SQL (AVG, PERCENTILE_CONT, STDDEV_SAMP) or a spreadsheet. ``` - Quick summary: `df["x"].describe()` returns count (non-missing), mean, std (`ddof=1`), min, the 25/50/75 percentiles and max; pass `percentiles=[.05,.25,.5,.75,.95]` to add the tail (pandas `DataFrame.describe`, verified 2026-10-27). - Median and IQR for skewed data: `df["x"].median()`, and `df["x"].quantile(.75) - df["x"].quantile(.25)`. - Standard deviation, ddof made explicit: `df["x"].std()` is `ddof=1`; `numpy.std(a)` is `ddof=0`. Pass `ddof=1` to numpy to match pandas (numpy `std`, verified 2026-10-27). - Skewness as a shape cue only: `scipy.stats.skew(x, bias=False, nan_policy="omit")`; `bias=False` applies the small-sample correction (scipy `skew`, verified 2026-10-27). - Quantiles with a known method: `numpy.quantile(a, [.25,.5,.75])` uses linear interpolation by default and offers nine methods; use `numpy.nanquantile` when NaNs are present (numpy `quantile`, verified 2026-10-27). - Coefficient of variation (ratio-scale, positive mean only): `df["x"].std() / df["x"].mean()`. - Mind NaN defaults: numpy functions propagate NaN by default, pandas skips it; this is a common source of two different answers. ## Sources - pandas, `DataFrame.describe` API reference (pandas 3.2.0.dev), pandas.pydata.org (last verified 2026-10-27). - NumPy, `numpy.std` API reference, numpy.org (last verified 2026-10-27). - NumPy, `numpy.quantile` API reference (nine quantile methods, Hyndman and Fan 1996), numpy.org (last verified 2026-10-27). - SciPy, `scipy.stats.skew` API reference (SciPy 1.18.0), docs.scipy.org (last verified 2026-10-27). - "Skewness and Kurtosis: What Your Data's Shape Tells You", editorworld.com (skewness bands as rough guidelines) (last verified 2026-10-27). - "Are Skewness and Kurtosis Useful Statistics?", spcforexcel.com (small-sample instability of shape statistics) (last verified 2026-10-27). Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link to afeezadebayo.com and to this skill's page. LinkedIn: linkedin.com/in/afeez-adebayo.