---
name: outlier-detection
description: >
  Detect and treat outliers in a dataset so that extreme values are investigated and handled on purpose, not deleted on sight, across pandas, NumPy, SciPy, scikit-learn, SQL and R. Covers domain range validation before any statistics, the IQR (Tukey) fences, the z-score and its masking and swamping traps, the MAD-based modified z-score, Grubbs and Dixon for small samples, multivariate detection with Mahalanobis distance, Isolation Forest and Local Outlier Factor, and the correct, cap, remove or flag decision. Use it when someone says "detect outliers", "remove outliers", "find anomalies", "this value looks wrong", "cap extreme values", "winsorise", "IQR", "z-score", or "my average is being dragged by one value". Do NOT use it for missing values (see handling-missing-data), for repeated rows (see duplicate-detection), or for simply describing a skewed distribution (see descriptive-statistics and data-distributions).
license: Free to use and adapt under CC BY 4.0, with credit to Afeez Adebayo. Link afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and this skill's page at https://afeezadebayo.com/skills-md/data/data-cleaning-preparation/outlier-detection/.
---

# Outlier detection

## When to use this, and when not to

Use this skill when a dataset holds values that sit far from the rest and you need to decide what they are and what to do with them. The triggers are a request to detect or remove outliers or anomalies, a value that looks impossible or extreme, an average or model that one value is distorting, or a mention of IQR, z-score, winsorising or isolation forest. The work is to separate data-entry errors from rare but real values, pick a detection method that suits the data, then correct, cap, remove or flag each flagged point on purpose.

Do not use it for gaps in the data (`handling-missing-data`), for rows that repeat (`duplicate-detection`), or for describing a distribution that is simply skewed, which is `descriptive-statistics` and `data-distributions`. A long tail is not the same as an outlier, and treating ordinary skew as contamination throws away real data.

The core idea: an outlier is a question about cause, not a verdict. Find the cause before you act, because the same extreme value can be a typo to fix, the single most important case in the data, or noise to bound, and only the cause tells you which (metricgate, "How to detect outliers": never remove outliers automatically, always investigate first).

## Inputs the agent must have before starting

1. The tool and version, because the method differs: pandas and NumPy for the statistical fences, scikit-learn for the multivariate detectors, SQL for percentile-based bounds.
2. The column or columns and their type, and whether the question is univariate (one column at a time) or multivariate (a combination that is only strange together).
3. The distribution shape: roughly symmetric, skewed, unimodal or multimodal. The shape decides whether a z-score is even valid.
4. The sample size, because Grubbs and Dixon are for small samples while the fences and the machine-learning detectors need more data.
5. The domain limits: the physically or logically possible range for the column. An impossible value is an error, not an outlier.
6. What a flagged value is likely to be, and the downstream use. A flagged value can be an error, a rare real case, or noise, and a mean is sensitive to it while many tree models are not. Removal is irreversible, so confirm it is allowed first.

If the distribution shape or the domain limits are unknown, establish them before choosing a method. A z-score on skewed data, or a statistical test on a value that is simply impossible, gives a confident wrong answer.

## The workflow, step by step

### 1. Validate against domain limits first, before any statistics

Check each value against the range it can logically take: age 0 to about 120, a percentage 0 to 100, a non-negative quantity, a date not in the future, a body weight in a sane band. A value outside these is a data error, and it is handled as an error (corrected, or set to missing and passed to `handling-missing-data`), not run through an outlier test. This step catches the values that matter most, and it stops an impossible `3000 kg` penguin or a `-5` age from being quietly "kept as an extreme but valid point".

### 2. Look at the distribution before choosing a method

Describe and plot the column: a histogram and a box plot, the median, the quartiles, the skew. Decide whether it is symmetric or skewed, unimodal or multimodal, and whether the interesting question is univariate or multivariate. This one look rules methods in or out, because the main methods assume different things.

### 3. Choose the method by shape, dimensionality and sample size

- **Skewed or unknown distribution, one column:** the IQR (Tukey) fences. Flag below `Q1 - 1.5 × IQR` or above `Q3 + 1.5 × IQR` for outliers, and `3 × IQR` for extreme ones. It assumes no distribution; under a normal distribution `k = 1.5` flags about 0.7% of points and `k = 3` about one in 425,000 (Tukey, *Exploratory Data Analysis*, 1977; letsdatascience).
- **Approximately normal and clean, one column:** the z-score, flagging `|z| > 3` (about 0.27% under normal). Use it only when the data is roughly symmetric and not already contaminated, because of the traps in step 4.
- **Suspected contamination, one column:** the modified z-score, `M = 0.6745 × (x - median) / MAD`, flagging `|M| > 3.5`, where MAD is the median absolute deviation and `0.6745` makes MAD consistent with the standard deviation under normality (Iglewicz and Hoaglin, 1993). Median and MAD resist the masking that breaks the plain z-score.
- **Small samples:** Grubbs for a single outlier when `n > 25` and the data is roughly normal, or Dixon's Q for very small samples (`n < 25`), iterating or using the generalised ESD test for more than one (metricgate).
- **Several columns at once:** Mahalanobis distance (use the robust MCD version when the data is already contaminated), `IsolationForest` for high-dimensional data, `LocalOutlierFactor` when clusters have different densities (`n_neighbors` around 20 works in general), and `EllipticEnvelope` when the inliers are plausibly Gaussian. The `contamination` parameter sets the expected fraction of outliers and therefore the flag threshold, so set it from domain knowledge, not by default (scikit-learn, "Outlier and novelty detection").

### 4. Know the z-score traps: masking and swamping

The plain z-score uses the mean and standard deviation, which the outliers themselves inflate. A large outlier can raise the standard deviation enough to pull its own z-score back under 3, so the test misses it (masking), and a distorted mean can push ordinary points past 3 and flag them falsely (swamping). This is why the modified z-score, built on the median and MAD, or the non-parametric IQR, is the safer default whenever contamination is suspected (letsdatascience, "Stop trusting the mean"). Handle `MAD = 0` explicitly: when more than half the values are identical, MAD is zero and the modified z-score is undefined, so detect that case and switch to the IQR or report it, rather than adding a tiny constant that returns absurd scores.

### 5. Decide the treatment: investigate, then correct, cap, remove or flag

- **Correct** a value that is a fixable error with a known true value.
- **Cap or winsorise** when you must bound a real value's influence on a sensitive statistic, pulling it to a percentile rather than deleting it.
- **Remove** only with a stated reason and explicit confirmation, because removal is irreversible and can delete the very case you care about.
- **Flag and keep** is the default: add an `is_outlier` column so the value is visible and the analysis can run with and without it. A rare real value is data, not noise.

Decision rule: never remove an outlier automatically. Investigate the cause, default to flagging, and reserve deletion for confirmed errors with a human sign-off.

### 6. For modelling, fit detection on training data and scale first

When outlier handling feeds a model, fit the detector or compute the fences on the training split only, then apply them to the test set, so no test information leaks. Scale or standardise features before any distance-based method (Mahalanobis, LOF, isolation forest), or the column with the largest units dominates the distance. Remember that many tree-based models tolerate outliers, so removal may not be needed at all.

### 7. Verify

Report how many points were flagged and by which method, and show the flagged values to a human rather than acting silently. Confirm a robust method agrees with the eyeball on a plot. Confirm you have not removed the signal: fraud, equipment failure and the largest customer are often the outliers that matter. Recompute the downstream statistic with and without the flagged points so the effect of the decision is visible.

## Worked example, end to end

A monthly transactions column runs mostly from £10 to £500, with two data-entry errors of £1,000,000 and £950,000, and one genuine large payment of £8,000. A plain z-score is run first:

```python
import numpy as np
z = np.abs((x - x.mean()) / x.std())
flagged = x[z > 3]      # misses both million-pound errors
```

The two huge errors inflate the standard deviation so much that each has `|z| < 3`, so the test flags neither: classic masking. The IQR and the modified z-score both catch them:

```python
q1, q3 = np.percentile(x, [25, 75]); iqr = q3 - q1
iqr_out = (x < q1 - 1.5 * iqr) | (x > q3 + 1.5 * iqr)   # flags all three extremes
med = np.median(x); mad = np.median(np.abs(x - med))
if mad == 0:
    mod_out = iqr_out                                    # MAD undefined, fall back
else:
    mod_out = np.abs(0.6745 * (x - med) / mad) > 3.5
```

Investigation shows the two million-pound values are impossible for this account, so they are corrected or set to missing as errors. The £8,000 payment is real, confirmed against the source system, so it is kept with an `is_outlier` flag rather than removed. The report shows the mean with and without the flagged points, so a reader sees that the headline average was driven by two typos, not by real spending.

## Variations you will meet in practice

- **Multivariate outliers.** A person 2 m tall and 40 kg is plausible on each column alone but strange together. No single-column test finds this; use Mahalanobis distance or `IsolationForest` on the columns jointly, after scaling.
- **The outlier is the target.** In fraud, fault and churn detection, the rare point is the thing you want, so removing outliers deletes the answer. Switch from cleaning to anomaly detection.
- **Time series.** A value that is high for the month may be normal for December. Decompose or de-seasonalise first, then test the residual, rather than testing the raw series.
- **Winsorising versus trimming.** Capping at the 1st and 99th percentiles bounds influence while keeping every row; trimming deletes rows. Prefer capping when the rows are otherwise valid.
- **MAD of zero.** When most values are identical, MAD is zero and the modified z-score cannot be computed; fall back to the IQR and say so, rather than returning every point as an outlier.

## Templates you can use verbatim

```python
# 1. Domain validation FIRST: impossible values are errors, not outliers
import numpy as np, pandas as pd
impossible = (df["age"] < 0) | (df["age"] > 120) | (df["pct"] < 0) | (df["pct"] > 100)
df.loc[impossible, ["age", "pct"]] = np.nan      # hand to handling-missing-data

# 2. IQR fences (robust, no distribution assumption)
def iqr_outliers(s, k=1.5):
    q1, q3 = s.quantile(0.25), s.quantile(0.75)
    iqr = q3 - q1
    return (s < q1 - k * iqr) | (s > q3 + k * iqr)

# 3. Modified z-score with an explicit MAD=0 guard
def modified_z_outliers(s, thresh=3.5):
    med = s.median(); mad = (s - med).abs().median()
    if mad == 0:
        return iqr_outliers(s)                    # undefined; fall back and report
    return (0.6745 * (s - med) / mad).abs() > thresh

# 4. Multivariate, fitted on train, scaled
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
Xs = StandardScaler().fit(train[cols]).transform(train[cols])
iso = IsolationForest(contamination=0.01, random_state=0).fit(Xs)   # set contamination on purpose
flags = iso.predict(Xs) == -1       # -1 = outlier
```

```sql
-- SQL: IQR fences via percentiles (PostgreSQL)
WITH q AS (
  SELECT percentile_cont(0.25) WITHIN GROUP (ORDER BY amount) AS q1,
         percentile_cont(0.75) WITHIN GROUP (ORDER BY amount) AS q3
  FROM payments)
SELECT p.* FROM payments p, q
WHERE p.amount < q.q1 - 1.5 * (q.q3 - q.q1)
   OR p.amount > q.q3 + 1.5 * (q.q3 - q.q1);   -- flag; do not DELETE without review
```

## Failure modes and anti-patterns

1. **Plain z-score on contaminated or skewed data.** Masking hides real outliers and swamping flags good points. Spot it when an obvious extreme is not flagged, or when far more points are flagged than expected. Fix: use the modified z-score (MAD) or the IQR.
2. **MAD of zero handled with a fudge constant.** Adding `1e-10` to a zero MAD returns gigantic scores, flagging everything or nothing. Spot it when nearly all or none of the values are flagged on a column with many ties. Fix: detect `MAD == 0` and fall back to the IQR, and report it.
3. **Automatic removal.** Outliers are deleted on sight, and the rare real case (the fraud, the biggest customer) disappears. Spot it when a known important record is gone. Fix: investigate the cause, default to flagging, and confirm before any removal.
4. **Impossible values treated as statistical outliers.** A `-5` age or a future birth date is run through an IQR test and "kept as extreme". Spot it with a range check. Fix: validate domain limits first and treat impossible values as errors.
5. **Distance methods on unscaled data.** Mahalanobis, LOF or isolation forest flag only on the largest-magnitude column. Spot it when every flagged point is extreme on one column and normal on the rest. Fix: scale the features before fitting.
6. **Leakage.** Detection is fitted on the full dataset, so model metrics look too good. Spot it when offline beats production. Fix: fit the detector on the training split only.

## Quality bar

- Domain and range checks run before any statistical test, and impossible values are handled as errors.
- The method matches the distribution shape, the dimensionality and the sample size; the plain z-score is not used on skewed or contaminated data.
- `MAD = 0` is detected and handled, not patched with a constant.
- Any threshold or `contamination` value is set with a stated reason, not left at a default.
- The default treatment is to flag and keep; nothing is removed without a confirmed cause and human sign-off.
- Multivariate detection scales features first, and detection that feeds a model is fitted on training data only.
- The affected statistic is recomputed with and without the flagged points, and the flagged values are shown for review.

## Data handling and safety

Outliers are often the records that matter most: a fraudulent transaction, a failing machine, a safety incident, the single largest customer. Removing them can hide exactly what an analysis should surface, so default to flagging and keep an audit of anything removed. Removal, capping and overwriting the source are irreversible, so require explicit human confirmation and work on a copy, never the raw file. Do not present a trimmed or winsorised statistic as the full picture without saying what was excluded and why. A flagged value may be a real person's real value (a high but genuine income, an unusual but valid medical reading), so do not discard a legitimate extreme as if it were invalid. When the data affects health, money, legal standing or safety, a removed or capped value can change a decision about a real person or a real risk, so human review is mandatory and the result carries a clear note that outlier treatment was applied.

## Appendix: tool-specific notes (verify before relying on these)

- **NumPy and SciPy:** `np.percentile` for quartiles and the IQR; `scipy.stats.zscore`; `scipy.stats.median_abs_deviation` for the MAD; `scipy.stats` provides tests usable for Grubbs-style and generalised ESD detection. Compute the modified z-score by hand with the `0.6745` constant.
- **scikit-learn:** `IsolationForest` (`contamination` sets the flag rate; efficient in high dimensions), `LocalOutlierFactor` (`n_neighbors` around 20; set `novelty=True` only to score unseen data, and then do not call it on the training rows), `EllipticEnvelope` (assumes Gaussian inliers), `OneClassSVM` (sensitive to outliers, needs `nu` tuned). All expose `predict` returning `-1` for an outlier. Scale features first.
- **pandas:** `quantile` for the fences; `clip(lower, upper)` to winsorise to percentile bounds; `describe` and `plot.box` to see the shape.
- **SQL:** `percentile_cont` (PostgreSQL, standard SQL) for quartiles and the IQR fences; no built-in isolation forest, so statistical fences are the practical approach in-database, and complex detection belongs in the application layer.
- **R:** `boxplot.stats` for IQR outliers; `car::outlierTest` for a Bonferroni-adjusted test; `robustbase` for robust estimators; `isotree` for isolation forest.

## Sources

- scikit-learn User Guide, "Outlier and novelty detection": the outlier versus novelty distinction, `IsolationForest`, `LocalOutlierFactor` (with `n_neighbors` around 20, and the novelty-mode warning), `EllipticEnvelope`'s Gaussian assumption, `OneClassSVM`'s sensitivity to outliers, and the role of the `contamination` parameter as the score threshold. Verified 31 October 2026.
- Tukey, *Exploratory Data Analysis* (1977), via letsdatascience, "Stop trusting the mean": the IQR fences with `k = 1.5` and `k = 3`, their flag rates under normality, and that the method needs no fitted distribution. Verified 31 October 2026.
- Iglewicz and Hoaglin (1993), via letsdatascience and metricgate: the modified z-score `0.6745 × (x - median) / MAD` with a `3.5` threshold, that `0.6745`, the 0.75 quantile of a standard normal, makes MAD consistent with the standard deviation, and the undefined `MAD = 0` case. Verified 31 October 2026.
- metricgate, "How to detect outliers: 6 methods compared": the comparison of IQR, z-score, modified z-score, Grubbs, Dixon's Q and Mahalanobis distance, the sample-size guidance (Grubbs for `n > 25`, Dixon for `n < 25`), the robust MCD version for contaminated multivariate data, and the rule to investigate the cause before removing anything. Verified 31 October 2026.
- letsdatascience, "Stop trusting the mean: a guide to statistical outlier detection": the masking and swamping failure modes of the plain z-score and the case for median and MAD based detection. Verified 31 October 2026.