Exploratory data analysis

This is the exact file agents read, including the credit header and licence notice. Save it as exploratory-data-analysis/SKILL.md.

---
name: exploratory-data-analysis
description: >-
  Runs a structured first pass over a new tabular dataset so an AI agent
  understands its shape, quality and main patterns before any modelling,
  reporting or decision. Checks structure and types, per-column distributions,
  missing-data patterns, outliers, duplicates, cardinality and a high-level
  relationship scan, then produces a written data profile with concrete issues
  and next steps. Use when someone uploads or points to a CSV, Excel, Parquet,
  TSV or a database table and says "analyse this", "what's in this data",
  "profile this dataset", "run an EDA", "look at this before I model it", "is
  this data clean", "what should I know about this file", or asks for a summary
  of a spreadsheet or extract. Do NOT use for: deep two-variable relationship
  work (use correlation-analysis), building or evaluating predictive models,
  A/B test or experiment read-outs, dashboards, ETL pipelines, or making the
  business decision itself.
---

# Exploratory data analysis

A new dataset is a set of claims you have not checked yet. Before anyone models it, charts it or decides anything with it, someone has to look at what is actually there: how many rows, which columns, what types, where the gaps are, which values are impossible, and what the data looks like one column at a time. This skill runs that first pass in a fixed order so nothing important gets skipped, and it ends with a written profile a person can act on.

The point of the first pass is not to find the answer. It is to find the problems, the surprises and the questions, so that the real analysis is built on data you understand rather than data you assume.

## When to use this, and when not to

Use this skill when:

- Someone gives you a tabular file (CSV, TSV, Excel, Parquet, JSON records) or a database table and wants to know what is in it.
- An analysis, model or report is about to be built on a dataset nobody has profiled yet.
- A dataset has changed (a new export, a new month, a new source) and you need to confirm it still looks the way the downstream work expects.
- Numbers in a report look wrong and you need to go back to the source data to see what is actually there.

Do not use this skill for:

- Measuring and interpreting the relationship between two specific variables. Hand that to `correlation-analysis`.
- Cleaning or transforming data as the main task. This skill diagnoses quality problems and recommends fixes; it does not silently rewrite the data.
- Building, tuning or judging a predictive model.
- Reading out an experiment or A/B test.
- Making the decision. This skill describes the data and flags risks. A person decides what to do.

## Inputs you must have before starting

Ask for anything in this list that is missing. Do not proceed on assumptions about any of it.

1. **The data itself**, or a path to it, and the format. If several files, ask whether to profile them separately or as one joined table.
2. **What each row represents** (one customer, one order, one event, one day). The grain decides whether duplicates are errors or expected.
3. **The question or decision** the data is meant to support, even loosely. A profile for "which customers churned" looks at different columns from one for "is this export complete".
4. **Known quirks**: columns that are codes not numbers, sentinel values for missing (-1, 999, "N/A", 0 meaning unknown), time zone, currency, and any filtering already applied.
5. **Sensitivity**: does the data contain personal, health, financial, legal or employment information? If yes, follow the PII rules below before printing or sharing anything.

If the person cannot answer 2 or 3, do a structural profile anyway (steps 1 to 6), state what you assumed, and ask the two questions back with the profile so the next pass is sharper.

## Workflow

Work on a copy. Never overwrite, re-save or mutate the source file during profiling. If a step would change the data (dropping rows, filling gaps, recoding), stop and recommend it in writing; apply it only after the person confirms.

### Step 1: Load and confirm the shape

1. Load the data without changing types yet (read everything as text first if the loader guesses badly).
2. Record row count, column count, and file size.
3. If rows are fewer than expected or columns are merged into one, the delimiter, encoding or header row is wrong. Fix the load before going further, because every later step inherits a bad load.

Decision rule: if the loaded row count differs from what the person expected by more than a rounding error, treat the load as suspect and report it before profiling anything else.

### Step 2: Map structure and types

For every column, record: name, inferred type, and three to five example values.

- If a column is stored as text but every value parses as a number or date, flag it as a type that needs casting.
- If a numeric column holds codes (postcodes, product IDs, category numbers), mark it as categorical. Means and correlations on ID numbers are meaningless.
- If a column has one value for every row, or nearly so, it is an identifier. If it has one value for the whole dataset, it carries no information for this dataset and should be set aside.

Cardinality rule: count distinct values per column.

- Distinct count equal to row count, or above ~90% of rows: likely an identifier or free text.
- Distinct count of 1: constant, no information here.
- Distinct count between 2 and ~20 on a large table: treat as categorical and tabulate it in full.

### Step 3: Profile each column on its own (univariate)

Split columns into numeric, categorical, datetime and text, and profile each group.

Numeric columns, report: count, mean, median, standard deviation, min, max, and the 1st, 25th, 50th, 75th and 99th percentiles.

- If mean and median differ by a lot, the column is skewed; note it and prefer the median when you summarise it later.
- If the minimum is impossible (a negative age, a negative price), flag it as a data error, not an outlier.
- If a single value dominates (for example 0 appears in 60% of rows), ask whether 0 means "zero" or "missing".

Categorical columns, report: the full value counts for low cardinality, or the top 15 plus an "other" bucket for high cardinality.

- Look for the same category written several ways ("UK", "U.K.", "United Kingdom"). Flag it for cleaning; do not merge silently.
- Note any category that is a placeholder ("unknown", "test", "n/a").

Datetime columns, report: earliest, latest, and whether there are gaps or future dates. A date after today in a historical dataset is almost always an error.

### Step 4: Missing data

For every column, compute the count and percentage of missing values, and include disguised missing values (empty strings, "NA", sentinels you found in step 2) in the count, not just true nulls.

Decision rules:

- Column missing 0%: fine.
- Column missing up to ~5%: usable; note it and plan to handle it at analysis time.
- Column missing ~5% to ~40%: usable with care; the missingness itself may carry meaning. Report it prominently.
- Column missing above ~40%: flag as a candidate to drop or to treat as "present/absent" rather than as a value. Recommend; do not drop it yourself.

Then check whether missingness is random or patterned: cross-tabulate the missing flag of each affected column against a few others. If rows missing column A are also systematically missing column B, or are concentrated in one source or time window, say so. Missing-not-at-random data biases every later step, and quietly filling it hides the bias.

### Step 5: Duplicates and grain

1. Check for fully duplicated rows. Report the count.
2. Check for duplicates on the key that defines a row (from input 2). If two rows share a customer ID when one row should be one customer, the grain is broken; report it with examples.
3. Decision rule: do not de-duplicate automatically. Full duplicates are sometimes real (two identical transactions). Report and recommend.

### Step 6: Outliers and impossible values

For each numeric column, identify extreme values with two lenses:

- Statistical: values beyond 1.5 times the interquartile range from the quartiles (the standard boxplot rule), or beyond roughly 3 standard deviations for data that looks bell-shaped.
- Domain: values that cannot be true (an age of 300, a 2500% conversion rate).

Separate the two in the report. A real but extreme value (one huge enterprise customer) is kept and noted; an impossible value is a data error. Never delete either without confirmation.

### Step 7: High-level relationship scan

This is a scan, not the relationship analysis itself.

1. For numeric columns, compute a correlation matrix as a quick map of what moves with what. Read it only to decide which pairs are worth a proper look.
2. Flag any pair with a strong coefficient (roughly |r| above 0.7) as "investigate", and any cluster of near-identical columns as possible redundancy or leakage.
3. Hand every pair you will actually act on to `correlation-analysis`, which checks the shape of the relationship, the right coefficient, significance and confounders. Do not report a scan coefficient as a finding. The scan tells you where to look; it does not tell you what is true.

Guardrail: a correlation matrix on its own is one of the most over-read objects in data work. Identical summary statistics can hide completely different shapes (see the appendix on Anscombe and the Datasaurus). Treat step 7 as a to-do list, not a conclusion.

### Step 8: Write the profile

Produce the data profile using the template below. Lead with the issues that would change a decision, not with the row count. End with the specific next steps and the open questions for the person.

### Step 9: Verify before handing over

Run the quality bar at the end of this skill against your own profile. Recompute two or three headline numbers a second way (for example, confirm the missing-percentage of one column directly). If any number in the profile cannot be reproduced from the data, remove it.

## Worked example

**Input:** a file `signups.csv`, "analyse this before I build a churn model", one row is meant to be one user.

**Step 1, shape.** 48,210 rows, 14 columns. The person expected "about 50k", so the load looks right.

**Step 2, structure.** `user_id` is distinct on 48,210 rows: identifier. `plan` has 3 values: categorical. `signup_date` and `last_seen` loaded as text: cast to dates. `monthly_revenue` loaded as text because of a "£" prefix: strip and cast to number.

**Step 3, univariate.** `monthly_revenue`: mean £41.80, median £19.00, so right-skewed; one value of £0 appears in 31% of rows. `age`: minimum -2, which is impossible. `country`: "UK", "U.K." and "United Kingdom" all present.

**Step 4, missing.** `last_seen` missing 22%; those rows are almost all `plan = "free"` and `signup_date` in the last 14 days, so the missingness is patterned (new free users who have not returned), not random. `company_size` missing 61%: flagged to drop or convert to present/absent.

**Step 5, duplicates.** 0 fully duplicated rows. 18 `user_id` values appear twice: grain is broken for those; examples listed.

**Step 6, outliers.** `monthly_revenue` has 4 values above £5,000 (real enterprise accounts, kept and noted). `age = -2` on 7 rows is an error.

**Step 7, relationship scan.** `monthly_revenue` and `plan` move together (expected); `last_seen` recency and churn-to-be will need a proper look. Flagged for `correlation-analysis`.

**Profile (shortened):**

> **signups.csv data profile, 2026-10-26.** 48,210 rows, 14 columns, one row per user (18 user_ids break this). **Act on these first:** `age` has 7 impossible negative values (errors, not outliers); `monthly_revenue = £0` on 31% of rows needs a definition (true zero or unknown?); `last_seen` is missing on 22% of rows in a clear pattern (new free users), so do not fill it blindly; `company_size` is 61% missing and is probably unusable as a value. **Clean before modelling:** cast `signup_date`/`last_seen` to dates, strip "£" from revenue, merge the three spellings of "United Kingdom". **Next steps:** confirm the £0 revenue meaning; decide the grain fix for 18 duplicate users; then take revenue-vs-plan and recency-vs-churn to correlation-analysis. **Open questions:** does a missing `last_seen` mean "never returned"? Is `company_size` worth keeping at all?

## Templates

### Data profile (use verbatim, fill every field)

```
# <filename> data profile, <date>
Rows: <n>   Columns: <n>   Grain: <one row = ...>   Source/export date: <...>

## Act on these first (issues that change a decision)
- <issue>: <where, how many rows, why it matters, recommended fix (not yet applied)>

## Columns
| column | type | distinct | missing % | notes (skew, sentinels, bad values) |
|--------|------|----------|-----------|-------------------------------------|

## Missing data
- Patterned missingness: <column> missing <x%>, concentrated in <where>
- Candidates to drop or convert: <columns above ~40% missing>

## Duplicates and grain
- Full duplicate rows: <n>   Key duplicates on <key>: <n> (examples: ...)

## Outliers vs errors
- Real extremes kept: <...>
- Impossible values (errors): <...>

## Relationships to investigate (hand to correlation-analysis)
- <pair>: <why worth a look>

## Next steps and open questions
1. <next step>
?  <question for the person>
```

### Opening questions to send when inputs 2 or 3 are missing

```
Before I read too much into this, two quick things so the profile is useful:
1. What does one row represent (one user, one order, one day)?
2. What decision or question is this data meant to support?
I'll profile the structure and quality now and sharpen it once you answer.
```

## Failure modes and anti-patterns

1. **Reading the correlation matrix as findings.** The matrix in step 7 is a map of where to look. Two columns can share a correlation of 0.8 and have no usable relationship, or share 0.0 and be tightly linked in a curve. Spot it by asking "have I plotted this pair?" Fix it by sending every pair that matters to `correlation-analysis`.
2. **Filling or dropping missing data during profiling.** Imputing in step 4 destroys the evidence of why the data is missing and bakes in bias. Spot it when your "clean" dataset has no nulls but you never asked why they were there. Fix it by only recommending fixes and applying them after confirmation.
3. **Treating impossible values as outliers.** A negative age is not a tail of the distribution; it is a broken record. Spot it by checking every minimum and maximum against what is physically possible. Fix it by separating "real extreme" from "error" in the report.
4. **Profiling the wrong grain.** If you think one row is one customer but it is one order, duplicate counts, means and missingness are all wrong. Spot it when the key column is not unique. Fix it by confirming the grain (input 2) before step 5.
5. **Running the same generic profile on every dataset.** A fixed report that ignores the question produces obvious, useless findings ("customer A spent the most"). Spot it when the profile would read identically for any file. Fix it by using input 3 to decide which columns and checks get the most attention.

## Quality bar

Run this against your own output before handing over. Every item must be verifiable from the data, not aspirational.

- [ ] Row and column counts match what the person expected, or the mismatch is flagged.
- [ ] Every column has a type, a distinct count and a missing percentage.
- [ ] Disguised missing values (sentinels, empty strings) are counted as missing.
- [ ] Missingness above ~5% is reported, and patterned missingness is described, not just counted.
- [ ] Impossible values are separated from real extremes.
- [ ] Duplicates are reported against the stated grain, and nothing was de-duplicated without confirmation.
- [ ] No row, column or value in the source file was changed by the profiling.
- [ ] Two or three headline numbers were recomputed a second way and agree.
- [ ] The profile leads with decision-changing issues and ends with next steps and open questions.
- [ ] If the data is sensitive, no raw PII was printed or sent anywhere (see appendix).

## Guardrails

- **Work on a copy; never mutate the source.** Dropping rows, filling gaps or recoding values are changes the person approves in writing first. Default to recommending, not doing.
- **Do not fabricate.** Every number in the profile comes from the data in front of you. If you did not compute it, do not write it. Benchmarks and conventions (percentile rules, the IQR outlier rule, VIF thresholds) are labelled as conventions, not as facts about this dataset.
- **PII and credentials.** Datasets often hold names, emails, phone numbers, addresses, government IDs, card or account numbers, and health, financial, legal or employment details. Never print raw PII in the profile, never paste it into a prompt sent to a third-party service, and never store it outside the person's own environment. Report structure and counts ("`email` is present and unique on 99.8% of rows"), not contents. If the dataset must be summarised by an external model, sample and mask first, and say so.
- **Not advice.** When the data concerns health, money, legal standing or employment, the profile describes the data only. It is not medical, financial, legal or HR advice, and any decision needs a qualified human.
- **Cost.** Profiling is cheap and local. If you use an API-based profiling service or ask a model to summarise a large file, note the token or credit cost, and prefer sampling a few thousand rows for the first pass.
- **Scope.** This skill will not make the decision, will not claim causation, and will not build a model. It stops at a profile and a set of next steps.

## Tool appendix (Python / pandas)

```
This section is tool-specific. The core workflow above is tool-agnostic and
works the same in R, SQL, Polars or a profiling library.
```

- Load without type guessing going wrong: `pd.read_csv(path, dtype=str)` first if inference misbehaves, then cast deliberately.
- Shape and types: `df.shape`, `df.dtypes`, `df.head()`, `df.sample(5)`.
- Univariate numeric: `df.describe(percentiles=[.01,.25,.5,.75,.99])`. For skew, compare `df[col].mean()` with `df[col].median()`.
- Categorical: `df[col].value_counts(dropna=False)` so nulls show in the count.
- Missing: `df.isna().mean().sort_values(ascending=False)` for percentages; add your sentinels first, for example `df.replace(['NA','N/A','', -1, 999], pd.NA)` on the columns where those are placeholders, decided per column, not globally.
- Duplicates: `df.duplicated().sum()` for full rows; `df.duplicated(subset=[key]).sum()` for the grain.
- Outliers (IQR): `q1,q3 = df[col].quantile([.25,.75]); iqr = q3-q1; mask = (df[col] < q1-1.5*iqr) | (df[col] > q3+1.5*iqr)`.
- Relationship scan: `df.corr(numeric_only=True)`. Note: since pandas 2.0.0 the default of `numeric_only` is `False`, so set it explicitly; correlations are computed on pairwise-complete rows, so different cells can rest on different row counts (verified against pandas docs, 2026-10-26).
- A profiling library (ydata-profiling and similar) is fine for a first look, but read its output critically against the steps above: automated reports rarely flag broken grain, disguised missing values or impossible-vs-extreme, and they do not know your question.

## Sources

- pandas, `DataFrame.corr` API reference (pandas 3.1.0), pandas.pydata.org (last verified 2026-10-26).
- F. J. Anscombe, "Graphs in Statistical Analysis", *The American Statistician*, 1973 (Anscombe's quartet) (last verified 2026-10-26).
- J. Matejka and G. Fitzmaurice, "Same Stats, Different Graphs", Autodesk Research, ACM CHI 2017 (the Datasaurus Dozen) (last verified 2026-10-26).
- "Four common pitfalls to avoid in exploratory data analysis", Towards Data Science (last verified 2026-10-26).
- T. Kosir, *Common Data Science Mistakes*, github.com/tomaztk (last verified 2026-10-26).
- "Exploring data" / ydata-profiling EDA skill (arielperez82) and the EDA skills surveyed in "Claude Code for Data Scientists", aibuilderclub.com ,  reviewed as existing implementations, last verified 2026-10-26.

Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link to afeezadebayo.com and to this skill's page. LinkedIn: linkedin.com/in/afeez-adebayo.