Data cleaning and preparation skills
Data Cleaning and Preparation Skills-md for AI Agents
Data cleaning skills-md are SKILL.md files that teach Claude, Codex and other AI agents to fix the defects every dataset has: missing values, duplicates, outliers and inconsistent formats. Each one carries decision rules and a verification step, so the agent cleans a real dataset on purpose and tells you what it changed.
7 of 7 skills published, version 1.0.0.
Where to start: begin with handling-missing-data, then duplicate-detection and outlier-detection, and move on to data-type-conversion, text-standardisation and date-parsing. Finish with data-validation-rules, the checks that catch bad data before it spreads.
Handling Missing Data
Finds, diagnoses and handles missing values across pandas, scikit-learn, SQL and R, from disguised sentinels and the MCAR, MAR and MNAR mechanisms to the drop, impute or flag decision, without leaking test data or quietly shrinking the variance.
Duplicate Detection
Finds and resolves duplicate records across pandas, SQL and fuzzy-matching libraries, choosing the business key, normalising before comparing, blocking for scale, validating the threshold and flagging rather than deleting.
Outlier Detection
Detects and treats outliers across pandas, SciPy, scikit-learn, SQL and R, from domain range checks and IQR and MAD fences to multivariate methods, handling the z-score masking trap and defaulting to flag, not delete.
Data type conversion
Converts columns to the right type across pandas, SQL and R: cleaning blockers before casting, choosing astype or to_numeric, keeping integers that can be missing as nullable, mapping booleans explicitly, protecting money and IDs, and counting what failed.
Text standardisation
Reduces a messy text or categorical column to one value per real thing across pandas, SQL and R: deterministic cleaning, mapping to a canonical list, fuzzy matching the leftovers with three score bands, and human review for the uncertain middle.
Date parsing
Parses messy date and time columns to the correct datetime across pandas, SQL and R: resolving the day-first ambiguity from the source, handling mixed formats, time zones and Unix timestamps, coercing failures to NaT and counting them.
Data Validation Rules
Writes, audits and places the data validation rules worth having, across Pandera, Great Expectations, dbt tests, Pydantic and SQL constraints, with the six quality dimensions, the right layer for each rule, warn against block, and the NULL traps that silently defeat CHECK and UNIQUE.
No skill matches that search. Clear it or choose All.
What data cleaning skills do for an AI agent
A general model can clean data, but left alone it guesses: it imputes over a legitimate zero, deletes rows it should flag, dedupes on the wrong key, or removes the outlier that was the whole point. These SKILL.md files give the agent the decision rules for each defect, so it fixes a real dataset the way a careful analyst would, and explains what it changed and why.
Where to start
Begin with handling-missing-data, since gaps block almost everything else. Add duplicate-detection so each entity appears once, and outlier-detection to tell a data-entry error from a rare but real value. Then fix the values themselves: data-type-conversion for columns in the wrong type, text-standardisation for inconsistent categories, and date-parsing for dates stuck as text. Install the whole set to clean a dataset end to end, or one skill at a time for a single problem.
What the agent needs from you
Point the agent at the dataset, say what each column means and where the data is going, and confirm whether rows or columns may be removed. Each skill asks for the specifics it needs, such as the business key for deduplication or the plausible range for outliers, rather than guessing and changing your numbers.
Frequently asked questions
What are data cleaning skills for AI agents?
They are SKILL.md files that turn a general model into a careful data cleaner for one task each. The agent loads the file, follows its workflow and decision rules, and runs the checks it names, such as confirming a count or a distribution after a change, before trusting the result.
Do I need to be technical to use them?
No. You describe the dataset and what it is for, and the agent applies the right treatment and explains it. The skills carry the statistical and SQL judgement, so you get a clean, auditable result without remembering which method suits which case.
Which cleaning skill should I start with?
Start with handling-missing-data, because gaps block most other steps, then duplicate-detection and outlier-detection. In a full cleaning pass, the usual order is structural fixes, duplicates, missing values, then outliers.
Will the skills change my data without telling me?
No. They default to flagging rather than deleting, and any irreversible action, such as dropping rows or merging records, needs your confirmation. Every skill logs what it changed so the cleaning is reproducible.
Sources and more data skills
Each skill lists its own sources with the date it was last checked, drawn from the official documentation of pandas, scikit-learn and the SQL engines, and from standard practice. For the rest of the data library, see the data-analysis and sql-querying subcategories.
