Data distributions

This is the exact file agents read, including the credit header and licence notice. Save it as data-distributions/SKILL.md.

---
name: data-distributions
description: >-
  Characterises the shape of one numeric variable: it plots the distribution
  with a bin rule that fits (Freedman-Diaconis by default, not the tool's
  Sturges default), checks at a second bin width and with an ECDF or QQ plot,
  reads modality and skew direction from the picture, quantifies skewness and
  excess kurtosis with the correct definitions (kurtosis is tail weight, not
  peakedness), assesses normality by eye rather than a large-sample p-value, and
  says what the shape means for the next step. Use when someone asks "is this
  normal", "is it skewed", "plot a histogram", "are there two groups in here",
  "how heavy are the tails", "should I log-transform this", or when a summary
  flagged a column's shape. Do NOT use for: the centre and spread numbers (use
  descriptive-statistics), a first-pass dataset profile (use
  exploratory-data-analysis), two-variable relationships (use
  correlation-analysis), formal hypothesis tests, experiment read-outs or
  forecasting.
---

# Data distributions

The shape of a variable decides almost everything you can honestly do with it: which average to trust, whether a standard deviation means anything, whether a model's normality assumption holds, and how often you should expect an extreme value. Most tools skip the shape and jump to a number, and the number quietly misleads. A histogram drawn with the default bin rule can hide a second peak or invent one. A kurtosis figure gets read as "how pointy the peak is" when it measures the tails. A normality test on fifty thousand rows rejects a distribution that is normal enough for any practical purpose. This skill characterises the shape of one variable properly: it plots the distribution with a bin rule that fits the data, reads modality and skew from the picture, quantifies skewness and kurtosis with the right definitions, assesses normality by eye rather than by a p-value, and says what the shape means for the next step.

The discipline is to look before you measure, to describe tails and peaks as separate things, and to let the shape, not habit, choose the method.

## When to use this, and when not to

Use this skill when:

- Someone wants to know what a variable's distribution looks like: its shape, its peaks, its tails.
- A method is about to assume normality (a t-test, a control chart, a confidence interval) and the assumption needs checking.
- A summary from `descriptive-statistics` or a first pass from `exploratory-data-analysis` flagged a column as skewed, heavy-tailed or oddly shaped.
- Someone asks "is this normal", "is it skewed", "plot a histogram of this", "are there two groups in here", "how heavy are the tails", or "should I log-transform this".

Do not use this skill for:

- The summary numbers themselves: the centre, the spread, the five-number summary. Use `descriptive-statistics`. This skill explains the shape those numbers sit inside.
- Profiling a whole dataset for the first time. Use `exploratory-data-analysis`.
- The relationship between two variables. Use `correlation-analysis`.
- Formal hypothesis testing, experiment read-outs, or forecasting a time series.

## Inputs you must have before starting

Ask for anything missing. Do not assume it.

1. **The variable and the dataset it lives in.** Shape analysis is for numeric variables (continuous or counts); an unordered category has a bar chart of frequencies, not a distribution shape.
2. **What one row represents.** A distribution of "amounts" means one thing per order and another per customer. Mixed grains produce fake multimodality.
3. **The type and unit of the variable,** and a plausible range, so impossible values are caught before they distort the picture.
4. **Why you need the shape.** Choosing a summary, checking a normality assumption, hunting for a mixture of groups, and deciding on a transformation are four different questions, and they change where you spend effort.
5. **The sample size,** because skewness, kurtosis and normality tests all behave very differently at n = 50 and n = 50,000.
6. **Sensitivity.** If the variable is personal, health, financial, legal or employment data, follow the PII rules below.

If the person gives you a column and nothing else, confirm the grain (input 2) and ask why they need the shape (input 4) before plotting, because the answer decides what to show.

## Workflow

Work on a copy of the data. This skill does not alter the source.

### Step 1: Plot the distribution first, with a bin rule that fits

A histogram is the workhorse, but the bin width changes the story, so never trust a single one. Choose the rule by the data:

| Rule | Bin width or count | Use it when | Avoid it when |
|---|---|---|---|
| Freedman-Diaconis | h = 2 x IQR / n^(1/3) | The default for skewed, heavy-tailed or outlier-prone data; it uses the IQR, so the tails do not set the bin width | Rarely the wrong choice; for very small n it can give too few bins |
| Scott | h = 3.5 x sd / n^(1/3) | Roughly normal, moderate samples | Outliers are present: they inflate the standard deviation and widen the bins (one billionaire turns an income histogram into two bins) |
| Sturges | k = ceil(log2(n)) + 1 | Small, near-normal samples only | Large n (it gives about 21 bins at n = 1,000,000) or skewed data; it assumes normality and oversmooths |

Freedman-Diaconis is the right default (metricgate, verified 2026-10-27). Sturges is R's default bin rule and one of NumPy's named options, though NumPy's own default is 10 equal-width bins; the Sturges rule oversmooths and assumes normal data, so it is a poor choice above roughly n = 10,000 or for skewed columns (Scott 2009, via Wikipedia; metricgate, verified 2026-10-27).

Then do two things every time:

1. Redraw the histogram at a second bin width. If a feature (a second peak, a gap) survives both, it is real; if it appears at one width and not the other, it is an artefact of binning.
2. Add a bin-free view: an empirical cumulative distribution function (ECDF), which has no bins at all, or a kernel density estimate as a smoothed overlay. For long-tailed data, a log scale on the x-axis often makes the shape legible.

### Step 2: Read modality and symmetry from the picture

1. **Count the peaks.** One peak is unimodal. Two or more peaks usually mean the column is a mixture of groups (two customer segments, two machines, weekday and weekend). A single mean, standard deviation, or skewness describes none of the groups. Decision rule: if the data is multimodal, split it by the likely grouping variable and summarise each group, and hand the per-group centres to `descriptive-statistics`. Do not report one summary for a mixture.
2. **Read the direction of skew.** A long right tail is positive (right) skew; a long left tail is negative (left) skew. Name the direction from the plot before you compute a number for it.

### Step 3: Quantify skewness as direction and degree, not precision

Compute the sample skewness (Fisher-Pearson coefficient; 0 means symmetric, positive means a heavier right tail) with the small-sample correction on (scipy `skew`, verified 2026-10-27). Read it against these bands, which are a common convention and not a law (editorworld, verified 2026-10-27):

- Within about -0.5 to 0.5: roughly symmetric.
- 0.5 to 1 or -0.5 to -1: moderate skew.
- Beyond +/-1: substantial skew; the mean and median diverge meaningfully.
- Beyond +/-2: severe skew; a transformation or a non-parametric method is usually more appropriate.

Skewness is unstable on small samples: in repeated 30-point samples from a symmetric population it has ranged from -1.3 to +1.3 (spcforexcel, verified 2026-10-27). Below roughly 100 to 300 rows, read the direction from the plot and treat the number as a rough cue, not a measurement.

### Step 4: Quantify kurtosis, and read it as tails, not peak

Compute excess kurtosis (Fisher's definition, where a normal distribution scores 0 because 3 is subtracted; scipy `kurtosis`, verified 2026-10-27). Confirm which convention your tool uses before reporting: excess kurtosis gives the normal a 0, while Pearson's raw kurtosis gives it a 3.

Read it correctly. Kurtosis measures tail weight and the propensity to produce outliers, not the peakedness of the distribution. Values near the mean contribute almost nothing to it; it is driven by the tails (Westfall 2014, via Wikipedia, verified 2026-10-27).

- Positive excess kurtosis (leptokurtic): heavier tails and more frequent extreme values than a normal distribution. Student's t, the Laplace distribution and financial returns are examples.
- Negative excess kurtosis (platykurtic): lighter tails and fewer extremes than normal. The uniform distribution is the example.

Decision rule: high positive excess kurtosis means a normal-based model will understate how often extreme values occur, which matters for risk, latency, quality control and any confidence interval that assumes thin tails. Kurtosis is even more unstable than skewness on small samples, so it needs large n to be worth reporting at all (spcforexcel, verified 2026-10-27).

### Step 5: Assess normality by eye, not by a p-value

1. **Draw a normal quantile-quantile (QQ) plot first.** It shows the size of a departure from normality, not merely whether one is statistically detectable. Read the shape: an S-curve means heavy tails, a J or banana shape means skew, and a few isolated points at one end mean outliers (metricgate normality, verified 2026-10-27).
2. **Treat a normality test as a secondary anchor, weighted by sample size.** The power of every normality test rises with n, so on large samples a test such as Shapiro-Wilk rejects departures too small to matter: a barely-bent line on 10,000 points is a non-issue even if the test screams significant. On small samples (n under about 50) the test is useful and Shapiro-Wilk is the usual default; above roughly a thousand rows, lean on the QQ plot (metricgate normality, verified 2026-10-27).
3. **Remember that normality is often not required.** The central limit theorem makes many methods, including the t-test, robust to mild departures at reasonable sample sizes. Judge a departure by whether it threatens the specific method you plan to use, not by a p-value alone.

### Step 6: Say what the shape means for the next step

Translate the shape into a decision:

- Symmetric with light tails: the mean and standard deviation are fair; normal-based methods are reasonable.
- Skewed: lead the summary with the median and IQR (hand to `descriptive-statistics`); a log or Box-Cox transform may help, but it changes what the variable means, so interpret results on the transformed scale and transform only for a reason.
- Heavy tails (high excess kurtosis): expect outliers; use robust methods and do not rely on normal-based intervals.
- Multimodal: investigate the mixture and summarise the groups separately.

### Step 7: Write it up with the plot, the verdict and the implication

Report the plot (or describe it), the modality and skew direction, the skewness and excess kurtosis with their convention and a note on sample-size reliability, the normality read from the QQ plot, and the one-line implication for what to do next.

### Step 8: Verify before handing over

Confirm you drew the histogram at two bin widths, produced a QQ plot (not just a test), stated whether kurtosis is excess or raw, and flagged skewness and kurtosis as unstable if n is small. If the only evidence for or against normality is a p-value, the analysis is not finished.

## Worked example

**Input:** "What does our API response time look like? I need to set a latency SLA." Data: one row per request, 7 days, `response_ms`, 50,000 rows.

**Step 1, plot.** A Freedman-Diaconis histogram is strongly right-skewed with a long thin right tail; unimodal. The same shape holds at a second bin width and on a log x-axis, so it is real. An ECDF confirms most requests are fast with a small fraction very slow.

**Step 2, modality.** One peak. No mixture visible.

**Step 3, skewness.** Skew = 3.1: severe right skew.

**Step 4, kurtosis.** Excess kurtosis = 18: very heavy tails, so extreme response times happen far more often than a normal model would predict. This is the point for an SLA.

**Step 5, normality.** The QQ plot curves sharply up at the right: heavy right tail. Shapiro-Wilk returns p far below 0.001, but at n = 50,000 that is expected and says little; the QQ plot shows the departure is large and real, not trivial.

**Step 6, implication.** Do not use the mean and standard deviation, and do not set the SLA from a normal assumption. Summarise with the median and the 95th and 99th percentiles, because the tail is the whole point of a latency target.

**Write-up (shortened):**

> **API response time, per request, 7 days (n = 50,000), 2026-10-27.** Strongly right-skewed (skew 3.1) with very heavy tails (excess kurtosis 18); unimodal. Median 90 ms, but p95 = 650 ms and p99 = 2,100 ms. The QQ plot confirms a genuine heavy right tail, not a borderline one. **For the SLA:** set it on a percentile (p95 or p99), not on a mean, and expect slow outliers more often than a normal model suggests. Mean-and-standard-deviation summaries would badly understate the tail.

## Templates

### Distribution write-up (use verbatim)

```
# <variable>, <grain>, <date>   (n = <n>)
Plot: histogram (FD bins), checked at a second width; ECDF/QQ drawn
Modality: <unimodal / bimodal / multimodal -> mixture of ...>
Skew: <direction> (skewness <value>, <band>)
Tails: excess kurtosis <value> (<heavier / lighter> than normal)
Normality (QQ): <normal enough / heavy tails / skewed / outliers at ... end>
  Test (secondary): <Shapiro p, with sample-size caveat>

What the shape means: <which summary / which method / transform? / split groups?>
Reliability: <skew/kurtosis stable at this n? / read from plot on small n>
```

### Opening questions when inputs are thin

```
Three checks so the shape read is honest:
1. What does one row represent (so any second peak is real, not mixed grains)?
2. Why do you need the shape: to choose a summary, to check a normality
   assumption, to find groups, or to decide on a transform?
3. Roughly how many rows are there (it changes how much to trust skew,
   kurtosis and any normality test)?
```

## Failure modes and anti-patterns

1. **Trusting a single histogram at the default bins.** R defaults to the Sturges rule and NumPy defaults to 10 equal-width bins, and neither suits large or skewed columns; the Sturges rule in particular oversmooths and assumes normal data, so it hides modes. Spot it when a conclusion rests on one histogram at the default width. Fix it with the Freedman-Diaconis rule, a second bin width, and a bin-free ECDF.
2. **Reading kurtosis as peakedness.** Kurtosis is about tails and outliers, not how pointy the peak is. Spot it when a write-up describes high kurtosis as a "sharp peak". Fix it by interpreting kurtosis as tail weight and confirming whether the figure is excess or raw.
3. **Reading a normality p-value on a large sample.** On tens of thousands of rows a normality test rejects deviations too small to matter. Spot it when "not normal, p < 0.001" is the only evidence on a big dataset. Fix it with a QQ plot and a judgement about the size of the departure.
4. **Trusting skewness and kurtosis on small samples.** Both swing wildly below a few hundred rows. Spot it when a shape claim rests on a single skew or kurtosis value from a small column. Fix it by reading the plot and flagging the number as unreliable.
5. **Transforming to tidy the shape, or flattening a mixture.** Logging a variable to make it look normal changes what it means, and reporting one summary for a two-peaked column describes neither group. Spot it when a transform has no stated reason, or a bimodal histogram gets one mean. Fix it by transforming only for a purpose and interpreting on the new scale, and by splitting mixtures into groups.

## Quality bar

Run this against your own output. Every item must be verifiable.

- [ ] The histogram was drawn with a rule that fits the data (Freedman-Diaconis by default), not left on the tool's Sturges default without thought.
- [ ] The histogram was checked at a second bin width, and a bin-free view (ECDF or KDE) was produced.
- [ ] Modality was stated, and any multimodal column was flagged as a likely mixture rather than given one summary.
- [ ] Skew direction was read from the plot and the skewness value reported with its band.
- [ ] Kurtosis was reported as excess or raw explicitly, and interpreted as tail weight, not peakedness.
- [ ] Normality was judged from a QQ plot, with any test treated as secondary and weighted by sample size.
- [ ] Skewness and kurtosis were flagged as unreliable when the sample is small.
- [ ] The write-up ends with what the shape means for the next step (which summary, which method, whether to transform).
- [ ] If the variable is sensitive, no raw personal values were printed or sent anywhere.

## Guardrails

- **Do not fabricate.** Every value comes from the data. The bin rules, the skewness bands, the sample-size cautions and the normality thresholds are labelled as conventions with a source and a date, not presented as facts about this dataset. Anything version-dependent (a tool's default bin rule) is worth verifying before relying on it.
- **Work on a copy; confirm before changing data.** A transformation (log, Box-Cox) or removing outliers is a change the person approves first, and results are then read on the transformed scale. Report the original shape, not a quietly transformed one.
- **PII and credentials.** If the variable is personal, health, financial, legal or employment data, never print raw values, never paste them into a third-party service, and keep them in the person's own environment. A histogram or density of a sensitive variable can still expose rare individuals in the tails; aggregate enough that no single person is identifiable.
- **Not advice.** When the variable concerns health, money, legal standing or employment, the output describes a distribution only. It is not medical, financial, legal or HR advice, and any decision needs a qualified human.
- **Scope.** This skill characterises the shape of one variable. For the centre and spread numbers use `descriptive-statistics`; for two variables use `correlation-analysis`; for a whole-dataset first pass use `exploratory-data-analysis`; for formal tests or an experiment read-out, route to the proper method. It will not model or forecast.
- **Cost.** Histograms, ECDFs and QQ plots are cheap and local. A kernel density estimate on millions of rows can be slow; sample the rows for the plot and say the plot is on a sample.

## Tool appendix (Python)

```
This section is tool-specific. The workflow above is tool-agnostic and works
the same in R (hist with FD, qqnorm, moments) or a stats package.
```

- Histogram with a sensible rule: `matplotlib.pyplot.hist(x, bins="fd")`, or compute edges with `numpy.histogram_bin_edges(x, bins="fd")`; NumPy also accepts `"scott"`, `"sturges"` and `"auto"` (which takes the larger of the Sturges and Freedman-Diaconis counts); its plain default is 10 equal-width bins (verified 2026-10-27, NumPy 2.5). Redraw at a second width to test any feature.
- Bin-free views: `matplotlib.pyplot.ecdf(x)` (Matplotlib 3.8+), or `numpy.sort` against `numpy.arange(1, n+1)/n`; a KDE via `seaborn.kdeplot(x)` or `scipy.stats.gaussian_kde` (the bandwidth matters, so do not read a KDE as ground truth).
- Skewness: `scipy.stats.skew(x, bias=False, nan_policy="omit")`; `bias=False` applies the small-sample correction (scipy `skew`, verified 2026-10-27).
- Excess kurtosis: `scipy.stats.kurtosis(x, fisher=True, bias=False, nan_policy="omit")`; `fisher=True` gives the excess form where a normal distribution is 0 (scipy `kurtosis`, verified 2026-10-27).
- QQ plot: `scipy.stats.probplot(x, dist="norm", plot=plt)`, or `statsmodels.api.qqplot(x, line="s")`.
- Normality test (secondary, small n): `scipy.stats.shapiro(x)`; read it alongside the QQ plot, and discount a tiny p-value on a large sample.
- A log x-axis (`plt.xscale("log")`) often makes a heavy right tail readable.

## Sources

- SciPy, `scipy.stats.skew` API reference (SciPy 1.18.0), docs.scipy.org (last verified 2026-10-27).
- SciPy, `scipy.stats.kurtosis` API reference (SciPy 1.18.0; Fisher excess form, normal = 0), docs.scipy.org (last verified 2026-10-27).
- "Kurtosis", Wikipedia (Westfall 2014: kurtosis measures tailedness, not peakedness), en.wikipedia.org (last verified 2026-10-27).
- "Sturges's rule", Wikipedia (oversmoothing; Scott 2009 recommends Freedman-Diaconis or Scott instead), en.wikipedia.org (last verified 2026-10-27).
- "Histogram Bin Width: Sturges vs FD", metricgate.com (bin-rule formulas and recommendations) (last verified 2026-10-27).
- "Normality Tests vs QQ Plots: What to Trust", metricgate.com (large-sample over-rejection; QQ plots first) (last verified 2026-10-27).
- "Skewness and Kurtosis: What Your Data's Shape Tells You", editorworld.com (skewness bands as rough guidelines) (last verified 2026-10-27).
- "Are Skewness and Kurtosis Useful Statistics?", spcforexcel.com (small-sample instability of shape statistics) (last verified 2026-10-27).

Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link to afeezadebayo.com and to this skill's page. LinkedIn: linkedin.com/in/afeez-adebayo.