Correlation analysis

This is the exact file agents read, including the credit header and licence notice. Save it as correlation-analysis/SKILL.md.

---
name: correlation-analysis
description: >-
  Measures and interprets the relationship between two variables correctly: it
  plots the pair first, picks the right coefficient (Pearson, Spearman or
  Kendall) from the data's shape, reports the strength with a sample size and a
  confidence interval or p-value, checks for confounders and Simpson's paradox,
  and states in plain words what the number does and does not mean. Use when
  someone asks "is X correlated with Y", "what drives Y", "does A affect B",
  "which factors relate to churn/revenue/conversion", "run a correlation",
  "build a correlation matrix", "is this relationship significant", or hands you
  two columns and asks how they move together. Do NOT use for: the first-pass
  profile of a whole dataset (use exploratory-data-analysis), causal proof,
  regression modelling or prediction, A/B test read-outs, or forecasting.
---

# Correlation analysis

A correlation is one of the easiest numbers to produce and one of the easiest to misread. A single coefficient can look strong when the relationship is an artefact of one outlier, look like nothing when the relationship is a clear curve, or point the opposite way once you account for a group the data was hiding. This skill measures the relationship between two variables in a way that survives those traps, and it says out loud what the number means for a decision.

The discipline is simple: look at the picture before you trust the number, choose the coefficient that matches the shape, report the uncertainty, and never let a correlation stand in for a cause.

## When to use this, and when not to

Use this skill when:

- Someone wants to know whether two variables move together, and how strongly.
- A correlation matrix from a first-pass profile flagged a pair worth a proper look (handed over from `exploratory-data-analysis`).
- A report or decision is about to lean on "X is correlated with Y" and the claim needs checking.
- Someone asks "what relates to" an outcome (churn, revenue, conversion) and wants the relationships ranked by strength and uncertainty.

Do not use this skill for:

- Profiling a whole dataset for the first time. Use `exploratory-data-analysis`.
- Proving that one thing causes another. Correlation cannot do that; say so and point to an experiment or a causal design.
- Building a predictive model or a regression. A correlation is a starting signal, not a model.
- Reading out an A/B test or forecasting a series.

## Inputs you must have before starting

Ask for anything missing. Do not assume it.

1. **The two variables** (or the set of variables for a matrix), and the dataset they live in.
2. **What each row represents.** Correlating two columns across a table only means something if each row is one comparable unit. Mixed grains produce nonsense.
3. **The type of each variable:** continuous, ordinal (ordered categories), or nominal. The type decides the method.
4. **The decision behind the question.** "Is ad spend correlated with sales" for a budget decision needs a confounder check that a curiosity question does not.
5. **Candidate confounders:** any variable that could drive both (season, region, customer size, time). Ask for the obvious ones up front.
6. **Sensitivity:** if either variable is personal, health, financial, legal or employment data, follow the PII rules below.

If the person gives you two columns and nothing else, confirm the grain (input 2) and the types (input 3) before computing anything, because both change the answer.

## Workflow

Work on a copy of the data. This skill does not alter the source.

### Step 1: Plot the pair first

Before computing any coefficient, draw the scatter plot of the two variables (for continuous or ordinal data). For a matrix, draw a grid of scatter plots, not only a heatmap.

Why first: datasets with identical coefficients can have completely different shapes. Anscombe's quartet is four datasets that share the same mean, variance, correlation (0.816) and regression line, yet one is linear, one is a curve, one is a line dragged by a single outlier, and one is a vertical stack where one far point creates the whole correlation (Anscombe, 1973). The Datasaurus Dozen pushes this further: twelve datasets, including a dinosaur, all sharing the same means, standard deviations and Pearson correlation to two decimal places (Matejka and Fitzmaurice, 2017). The plot tells you which coefficient is even valid.

Decision rules from the plot:

- Roughly straight-line cloud: Pearson is appropriate.
- Curved but always rising or always falling (monotonic): use Spearman, not Pearson.
- U-shape or any non-monotonic curve: no single correlation coefficient describes it. Report that and stop reaching for one number.
- One or two points far from the rest: note them now; they will drive step 4.

### Step 2: Choose the coefficient from type and shape

| Situation | Coefficient |
|---|---|
| Two continuous variables, straight-line relationship | Pearson's r |
| Monotonic but curved, or either variable is ordinal, or heavy outliers | Spearman's rho |
| Ordinal with many ties, or small samples | Kendall's tau |
| One variable is nominal (unordered categories) | Not a correlation coefficient; use group comparisons or Cramer's V |

Pearson measures the strength of a straight-line relationship on the raw values. Spearman is Pearson computed on the ranks, so it measures any monotonic relationship and is far less sensitive to outliers and to curvature. Because it works on ranks, extreme values pull it much less than they pull Pearson. In a curved-but-monotonic case, Spearman's magnitude is usually higher than Pearson's, because the curvature makes Pearson understate the link (statisticsbyjim.com, verified 2026-10-26).

Decision rule: if the plot in step 1 is monotonic but not straight, or either variable is ordinal, or there are influential outliers you are not removing, use Spearman. When in doubt, compute both and report the gap; a large gap between Pearson and Spearman is itself a finding about shape.

### Step 3: Compute the coefficient, the sample size and the uncertainty

1. Compute the coefficient on the pairs where both values are present. Record the number of complete pairs (n) used, because that is what the result rests on.
2. Report a confidence interval (preferred) or a p-value with it. A coefficient without n and without uncertainty is not a finding.
3. Significance rules:
   - The p-value answers only "could this coefficient have come from no relationship at all". It does not measure strength. A tiny, useless correlation can be highly significant on a big dataset.
   - Pearson's p-value assumes the data is drawn from normal distributions; it is treated as reasonably robust once n is about 25 or more (statisticsbyjim.com). For small samples, prefer a confidence interval, and consider a permutation test.
   - Spearman's p-value is only accurate for large samples (roughly above 500 observations); for smaller samples use a permutation test (SciPy `spearmanr` docs, verified 2026-10-26).
4. Do not attach fixed "weak / moderate / strong" labels to coefficient sizes as if they were universal. What counts as a strong correlation differs by field. Report the number and the interval, and interpret it against the domain, not a generic table.

### Step 4: Stress-test the number

A coefficient you have not stress-tested is a guess with decimal places.

1. **Outliers.** Recompute the coefficient with and without the influential points from step 1. If one or two rows move the coefficient materially (for example from 0.2 to 0.7), the relationship is about those rows, not the population. Report both numbers and the row count.
2. **Constant or near-constant inputs.** If either variable barely varies, the coefficient is undefined or unstable. Tools return NaN for a constant input and warn for a near-constant one. Check variance before trusting the result.
3. **Non-linearity recheck.** If Pearson is near zero, confirm from the plot that there is truly no relationship rather than a symmetric curve. "No correlation does not imply independence unless the pair is jointly normal"; a V-shape (y = |x|) gives a Pearson correlation near zero while being perfectly determined (SciPy `pearsonr` docs, verified 2026-10-26).

### Step 5: Check for confounders and Simpson's paradox

This is the step most correlation work skips, and it is the one that flips conclusions.

1. For each candidate confounder from input 5, compute the correlation within each group of that confounder, not only on the pooled data.
2. If the pooled correlation disagrees with the within-group correlations (different sign, or present in the pool and gone in every group), you have Simpson's paradox: the pooled number is an artefact of how the groups are mixed. The classic case is a kidney-stone study where one treatment was better for both small and large stones separately, yet looked worse overall because it was given to the harder cases (Charig et al., via metricgate.com, verified 2026-10-26).
3. Decision rule: when a confounder changes the story, report the within-group relationships as the real finding and the pooled number as misleading. Adjusting properly is a regression job; name it as the next step rather than forcing it here.

### Step 6: Write it up and state what it does not mean

Use the template below. Every correlation write-up ends with the causation clause: a correlation shows that two things move together, not that one causes the other. A third variable, reverse causation or coincidence can all produce it. If the person needs a causal answer, the next step is an experiment or a causal design, not a bigger correlation.

### Step 7: Verify before handing over

Run the quality bar. Recompute the headline coefficient a second way (for example Pearson by formula vs the library, or on a fresh load of the data) and confirm n matches the number of complete pairs. If the plot was not produced, the analysis is not finished.

## Worked example

**Input:** "Is ad spend correlated with sales? I want to set next quarter's budget." Data: one row per week for 2 years (104 rows), `ad_spend` and `sales`, both continuous. Candidate confounder offered: `is_promo_week`.

**Step 1, plot.** Scatter shows a rising cloud, roughly linear, with three weeks sitting far top-right.

**Step 2, method.** Continuous and roughly straight-line: Pearson, and Spearman as a cross-check because of the three far points.

**Step 3, compute.** Pearson r = 0.74 (95% CI 0.64 to 0.82), n = 104, p < 0.001. Spearman rho = 0.69. Close, so the shape is sound.

**Step 4, stress-test.** Remove the three far weeks: Pearson drops to 0.58. The relationship holds without them but is weaker, so those three weeks inflate it. Neither variable is constant. Pearson is not near zero, so the non-linearity trap does not apply.

**Step 5, confounder.** Split by `is_promo_week`: in non-promo weeks r = 0.41; in promo weeks r = 0.39; the three far weeks were all promo weeks. The pooled 0.74 overstates the normal-week relationship because promo weeks have both high spend and high sales. This is a milder Simpson's effect: the within-group links are real but weaker.

**Write-up (shortened):**

> **Ad spend vs sales, weekly, 2 years (n = 104), 2026-10-26.** Overall Pearson r = 0.74 (95% CI 0.64 to 0.82), Spearman 0.69. The pooled figure is inflated by three promotional weeks with unusually high spend and sales. Within ordinary (non-promo) weeks the relationship is moderate: r = 0.41. **What this means:** spend and sales move together, more so during promotions. **What it does not mean:** this does not show that extra spend causes proportional extra sales; promotions, seasonality and demand could drive both. **Next step for a budget decision:** a spend test (hold out regions or weeks) or a regression that adjusts for promotions and season, not a larger correlation.

## Templates

### Correlation write-up (use verbatim)

```
# <var A> vs <var B>, <grain>, <date>
n (complete pairs): <n>   Method: <Pearson/Spearman/Kendall> (why: <shape/type>)
Coefficient: <r> (95% CI <lo> to <hi>)   p: <p>   Cross-check: <other method = x>

Shape (from the plot): <linear / monotonic curve / non-monotonic / outlier-driven>
Outlier test: with = <r>, without <k> points = <r>
Confounders checked: <var: within-group coefficients>
Simpson's paradox: <none / detected, pooled misleads>

What this means: <plain-language reading against the domain>
What it does NOT mean: this is association, not causation; <plausible third factors>
Next step: <experiment / regression / stratified view>
```

### Opening questions when inputs are thin

```
Quick checks so the correlation is meaningful, not just a number:
1. What does one row represent, and is each row one comparable unit?
2. Is each variable continuous, ordered-category, or unordered-category?
3. What are you deciding with this, and what else could move both variables
   (season, region, promotions, customer size)?
```

## Failure modes and anti-patterns

1. **Trusting the coefficient without a plot.** The headline failure. Anscombe and the Datasaurus exist because a number cannot see a dinosaur. Spot it when you have an r but no scatter plot. Fix it by making step 1 non-negotiable.
2. **Using Pearson on a curve or on ranks.** Pearson near zero on a clear U-shape, or Pearson on Likert-scale data, both mislead. Spot it when the plot is curved or a variable is ordinal. Fix it with Spearman or a group comparison.
3. **Reading significance as strength.** On 100,000 rows almost everything is "significant". Spot it when a tiny r is reported as important because p is small. Fix it by leading with the coefficient and interval and interpreting the size against the domain.
4. **Ignoring confounders and Simpson's paradox.** A pooled correlation that reverses within groups is the most expensive mistake here, because people act on the pooled number. Spot it by always splitting on the obvious confounder. Fix it by reporting within-group relationships when they disagree with the pool.
5. **Letting "correlated" become "causes".** The slide from association to cause happens in the summary sentence, not the maths. Spot it when the write-up recommends an action as if the cause were proven. Fix it with the mandatory causation clause and a named causal next step.

## Quality bar

Run this against your own output. Every item must be verifiable.

- [ ] A scatter plot (or scatter grid for a matrix) was produced and read before any coefficient was trusted.
- [ ] The coefficient matches the data's type and shape (Pearson only for straight-line continuous data).
- [ ] n (complete pairs) is reported, and a confidence interval or p-value is attached.
- [ ] The coefficient was recomputed with and without influential outliers, and both are reported if they differ.
- [ ] Constant and near-constant inputs were ruled out.
- [ ] At least one obvious confounder was checked by computing within-group correlations.
- [ ] No fixed "weak/moderate/strong" label was applied as if universal.
- [ ] The write-up states plainly that this is association, not causation, and names a causal next step.
- [ ] The headline coefficient was reproduced a second way and n matches the complete-pair count.
- [ ] If either variable is sensitive, no raw PII was printed or sent anywhere.

## Guardrails

- **Association is not causation.** This is the core scope boundary. The skill never concludes that one variable causes another, and every write-up says so. A causal question is routed to an experiment or a causal design.
- **Do not fabricate.** Every coefficient, interval and n comes from the data. Conventions (CI method, the n thresholds for p-value accuracy) are labelled as conventions with their source and date, not presented as facts about this dataset.
- **Work on a copy; confirm before changing data.** Removing outliers for the main reported figure, or recoding a variable, is a change the person approves first. Report with-and-without rather than silently dropping rows.
- **PII and credentials.** If a variable is personal, health, financial, legal or employment data, never print raw values, never paste them into a third-party service, and keep them in the person's own environment. Report the relationship, not the records. Correlations that touch protected attributes (for example a score against age, sex, race or disability) can imply discrimination; flag the fairness and legal risk and route it to a human, and do not present such a correlation as a recommendation.
- **Not advice.** When the variables concern health, money, legal standing or employment, the output describes a statistical association only. It is not medical, financial, legal or HR advice, and any decision needs a qualified human.
- **Cost.** Correlations are cheap and local. If a matrix is large (hundreds of variables), note that the number of pairs grows with the square of the variables, prefer a targeted set over "everything against everything", and sample rows for the first look.
- **Scope.** This skill measures and interprets a relationship. It will not build a model, prove a cause, or make the decision.

## Tool appendix (Python)

```
This section is tool-specific. The workflow above is tool-agnostic and works
the same in R, SQL (corr functions) or a stats package.
```

- Always plot first: `import matplotlib.pyplot as plt; plt.scatter(df[a], df[b])`. For a matrix, a scatter grid (`pandas.plotting.scatter_matrix` or `seaborn.pairplot`), not only `seaborn.heatmap`.
- Pearson with p and interval: `scipy.stats.pearsonr(x, y)` returns the statistic and p-value, and `.confidence_interval()` gives the interval (Fisher method by default). It raises `ConstantInputWarning` and returns NaN for a constant input, and warns for a near-constant one (SciPy `pearsonr` docs, verified 2026-10-26).
- Spearman: `scipy.stats.spearmanr(x, y)`; its p-value is accurate only for large samples (above ~500), so use `method=stats.PermutationMethod()` for smaller ones. `nan_policy` controls NaN handling: default `propagate` returns NaN, `omit` drops them (SciPy `spearmanr` docs, verified 2026-10-26).
- Matrix: `df.corr(method="pearson", numeric_only=True)`. Traps to know, from the pandas docs (verified 2026-10-26): since pandas 2.0.0 the default of `numeric_only` is `False`, so set it; correlations use pairwise-complete rows, so different cells can rest on different n; `method` also accepts `"spearman"`, `"kendall"` or a callable. A heatmap of this matrix is a map of where to look, not a set of findings.
- Outlier recheck: filter the influential rows found in the plot and recompute; report both coefficients and the dropped row count.
- Within-group (confounder) check: `df.groupby(conf).apply(lambda g: g[a].corr(g[b]))` and compare to the pooled `df[a].corr(df[b])`.

## Sources

- SciPy, `scipy.stats.pearsonr` API reference (SciPy 1.18.0), docs.scipy.org (last verified 2026-10-26).
- SciPy, `scipy.stats.spearmanr` API reference (SciPy 1.18.0), docs.scipy.org (last verified 2026-10-26).
- pandas, `DataFrame.corr` API reference (pandas 3.1.0), pandas.pydata.org (last verified 2026-10-26).
- F. J. Anscombe, "Graphs in Statistical Analysis", *The American Statistician*, 1973 (Anscombe's quartet) (last verified 2026-10-26).
- J. Matejka and G. Fitzmaurice, "Same Stats, Different Graphs", Autodesk Research, ACM CHI 2017 (Datasaurus Dozen) (last verified 2026-10-26).
- J. Frost, "Spearman's Rank Correlation", statisticsbyjim.com (last verified 2026-10-26).
- "Simpson's Paradox examples" (kidney-stone case, after Charig et al. 1986), metricgate.com (last verified 2026-10-26).

Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link to afeezadebayo.com and to this skill's page. LinkedIn: linkedin.com/in/afeez-adebayo.