Analytics QA and validation

This is the exact file agents read, including the credit header and licence notice. Save it as analytics-qa-validation/SKILL.md.

---
name: analytics-qa-validation
description: Validates analytics instrumentation end to end so the collected data can be trusted, from pre-release QA in a debug environment through schema and type validation at ingestion to production monitoring for data loss and drift. Use when the request mentions QA'ing tracking, testing events before release, GA4 DebugView, Tag Assistant, GTM Preview, data layer validation, "my events dropped", duplicate or double-counted events, data observability, schema drift, a tracking audit, or checking that analytics is firing correctly. Do NOT use for designing the event taxonomy (event-tracking-plan), campaign URL tagging (utm-campaign-tracking), ad-platform conversions and the Conversions API (conversion-tracking-capi), or analysing the data once it is trusted (funnel-analysis, cohort-retention-analysis).
license: CC BY 4.0. Free to use and adapt with credit to Afeez Adebayo.
metadata:
  author: Afeez Adebayo
  website: https://afeezadebayo.com
  linkedin: https://www.linkedin.com/in/afeez-adebayo/
  source: https://afeezadebayo.com/skills-md/data/analytics-tracking/analytics-qa-validation/
  category: data
  subcategory: analytics-tracking
  version: 1.0.0
  created: 5 November 2026
  updated: 5 November 2026
---

# Analytics QA and validation

Most analytics problems are not found; they are discovered months later when a number looks wrong and nobody can say when it broke. This skill makes an AI agent validate tracking the way a careful engineer tests code: against a written spec, before release, at the point the data is ingested, and then continuously in production. The job is not to admire a dashboard. It is to prove that each event fires once, with the right properties, for the right people, and keeps doing so.

Validation has three layers, and skipping any one leaves a blind spot. Pre-release QA catches the event that never fires or fires twice. Schema validation at ingestion catches the property sent as the wrong type. Production monitoring catches the silent drop three weeks later when a deploy removes a tag. This skill works through all three and then separates a real tracking fault from a consent or modelling effect that only looks like one.

## When to use this skill, and when not to

Use this skill to QA tracking before a release, to audit live instrumentation, to debug events that dropped or doubled, to set up ongoing monitoring, or to validate data against a tracking plan. It fires on GA4 DebugView, Tag Assistant, GTM Preview, data layer validation, "my events dropped", double-counted events, data observability or schema drift.

Do not use it to design the event taxonomy, which is the sibling `event-tracking-plan`, and which this skill validates against. Do not use it for campaign URL tagging, which is the sibling `utm-campaign-tracking`. Do not use it for ad-platform conversions and the Conversions API, which is `conversion-tracking-capi`. Do not use it to analyse data that is already trusted, which is `funnel-analysis` or `cohort-retention-analysis`. If there is no tracking plan to validate against, stop and build one first with `event-tracking-plan`; QA without a spec is just clicking around.

## Inputs the agent must have before starting

Ask for these in one message, and do not invent them:

1. The tracking plan or spec the data is supposed to match. Without it there is no definition of correct.
2. The collection setup: the tool (GA4, Segment, Amplitude, Snowplow), whether a tag manager is used, and whether events fire client-side, server-side or both.
3. Access to a debug or staging environment, or permission to use debug mode, so testing does not pollute production data.
4. The specific symptom if this is a debugging request: which event, which metric, when it changed, and by how much.
5. The consent setup and the market, because consent choices change how much data arrives.
6. For production monitoring, where the data lands (the warehouse or the tool's own store) and what a normal day looks like, so anomalies have a baseline.

If this is a debugging request and no plan exists, say so and scope the work to the symptom rather than pretending to validate the whole system.

## Workflow

### Step 1. Pin down what correct means

Load the tracking plan. For each event in scope, write down what must be true: it fires on the stated trigger, exactly once, with every required property, each property the right type and within its allowed values, the right identity attached, and only after consent where consent is required. This list is the test suite. Everything below checks against it.

### Step 2. Pre-release QA in a debug environment

Test before the release reaches real users.

- Turn on debug mode. In GA4, GTM Preview routes your traffic to DebugView automatically; otherwise set `debug_mode` true on the config or use the tag assistant and realtime views. DebugView shows events in sequence with their parameters and user properties, so you can watch an event fire and inspect what it carried.
- Walk each user journey and confirm every planned event appears once, in the right order, with the right parameters. An event that fires twice, fires on page load when it should fire on click, or arrives with a missing parameter fails here.
- Check the tag-manager traps that cause most bugs: exactly one Google tag per measurement id, because a duplicate is the most common cause of doubled events; the data layer is pushed before the container reads it, not after; the previous ecommerce object is cleared before the next push; triggers are custom data-layer events rather than brittle DOM selectors; and there are no event-name typos.
- Confirm nothing fires before consent is granted where that is required.
- Remember what debug tools do not cover: DebugView validates the GA4 destination only, not other tools downstream, and the preview view does not prove the data survived every filter into reports. Use realtime and, after a short delay, the standard reports as a second check.

### Step 3. Validate schema and types at ingestion

Pre-release testing is manual and partial; schema validation is automatic and total. Check every incoming event against the plan: required properties present, each property the right type, enumerations within their allowed values, no unexpected new events. Decide what happens to a violation: record it, or block and quarantine it so bad data never reaches the warehouse. On a paid stack this is Segment Protocols, Avo Inspector or Snowplow's failed-events route; on a free stack it is a JSON Schema per event validated in CI and a check job against recent data. Blocking is powerful and irreversible for the data it drops, so default to recording and alerting, and get explicit human confirmation before turning on a hard block.

### Step 4. Monitor in production against a baseline

Most breakage happens after launch, so watch the live data on the five observability pillars:

- Freshness: is the data arriving on time. A gap where events usually flow means a stalled pipeline or a removed tag, not a quiet day.
- Volume: is the per-event count within its normal range. A sharp drop is lost data; a spike is often duplicates.
- Schema: has a property been added, removed or retyped since the last release.
- Distribution: have null rates, cardinality or the spread of values shifted, which signals a field quietly breaking even while the event still fires.
- Lineage: which downstream tables and reports depend on this event, so you know the blast radius when it breaks.

Set alerts on per-event volume against a trailing baseline and on freshness gaps, because these catch the two most common production failures: a tag removed in a deploy, and a sudden flood of duplicates.

### Step 5. Separate a real fault from a consent or modelling effect

A drop is not always a bug. Before concluding that tracking broke, check whether consent or modelling explains it. Under a consent banner, denied sessions may send no data at all in basic consent mode, or only cookieless signals in advanced mode, so a change in consent rate moves the numbers without any code changing. Platform modelling that fills some gaps only switches on above volume thresholds that are reported to be around a thousand qualifying events or ad clicks per day sustained over roughly a week; verify the current thresholds in the platform's documentation before relying on them. Below the threshold there is no modelled recovery, so the raw drop is what you see. A consent-driven change looks like data loss but is not a tracking fault, and fixing it is a consent-configuration job, not a tag fix.

### Step 6. Triage with a symptom-to-layer decision tree

When a number looks wrong, find the earliest broken layer and fix there:

- Does the event fire correctly in DebugView? If not, it is a collection problem (Step 2).
- If it fires, does it pass schema validation? If not, it is a type or property problem (Step 3).
- If the schema is clean, did consent rates or the consent setup change? If so, it is a consent effect (Step 5), not a bug.
- If none of those, is it sampling or modelling in the reporting tool rather than the raw data? Check the raw export.

Fix the earliest broken step first, because a fault there makes everything downstream look wrong.

## Worked example

A team reports purchase events down about 40% week on week and assumes the tag is broken.

Running the decision tree: Step 2 fires a test purchase, and DebugView shows the `purchase` event firing correctly with `value` and `items`, so collection works. Step 3 checks the plan, and there are no schema violations. Step 5 checks consent, and the drop lines up with a consent-banner change the week before: denied sessions now send nothing because advanced consent mode was never enabled, and the site is below the modelling threshold, so there is no modelled recovery. That explains most of the 40%. The fix is a consent-configuration change, not a tag change, plus an annotation on the chart so the drop is not read as a product failure.

While validating, Step 2 also catches a second, smaller problem: a duplicate Google tag on the checkout template was firing `purchase` twice for one traffic segment, which had been inflating the previous week and made the drop look worse. Removing the duplicate corrects both weeks. The headline finding is that the scary 40% was mostly consent, not breakage, which is the opposite of the team's first guess.

## Templates you can use verbatim

Per-event QA checklist (run in a debug environment):

```
event: ____________
[ ] Fires on the stated trigger, and only then
[ ] Fires exactly once (no duplicate tag, no double push)
[ ] Every required property present
[ ] Each property the right type and within allowed values
[ ] Correct identity attached (anonymous vs user_id as planned)
[ ] Does not fire before consent where consent is required
[ ] Visible in DebugView, then in realtime, then in standard reports
```

Production monitor spec (one per critical event):

```
event: purchase
freshness: alert if no events for > 2x the normal gap
volume: alert if daily count is outside [baseline - 30%, baseline + 50%]
schema: alert on any new, removed or retyped property
distribution: alert if null rate on value or items moves > 10 points
unexpected: alert on a new event name not in the plan (possible dynamic-value leak)
```

Diagnosis prompt: "Our [event] dropped by [amount] on [date]. Here is the tracking plan and access to DebugView and the export. Work the collection, schema, consent and modelling layers in order and tell me the earliest broken one before proposing a fix."

## Failure modes and anti-patterns

1. QA with no spec. You spot it when testing becomes clicking around with no pass or fail. Fix it by loading the tracking plan first and turning each event into a checklist, or build the plan with `event-tracking-plan`.
2. Testing only the happy path in preview. You spot it when an event that passed preview is missing or doubled in production. Fix it by adding ingestion-time schema validation and production monitoring, not just pre-release checks.
3. Trusting DebugView as proof the data reached reports. You spot it when DebugView looks fine but a report is empty because a filter dropped the traffic. Fix it by confirming in realtime and standard reports as well.
4. Reading a consent-driven drop as a bug. You spot it when a metric falls with no code change and the fall tracks a consent-banner or market change. Fix it by checking consent rates and modelling thresholds before touching a tag.
5. Blocking non-conforming events without confirmation. You spot it when valid-but-new data silently disappears after a strict control was switched on. Fix it by defaulting to record-and-alert and getting human sign-off before any hard block.

## Quality bar for the validation

Before handing over, confirm every item:

- Every event in scope was checked against the written plan, not against a guess.
- Each event was seen to fire once, with the right properties and identity, in a debug environment.
- The tag-manager traps were checked: one Google tag per id, data-layer order, cleared ecommerce, no typos.
- Schema validation exists at ingestion, with a stated rule for violations, and no hard block was enabled without sign-off.
- Production monitoring covers freshness and per-event volume against a baseline, at minimum.
- Any drop was traced through the collection, schema, consent and modelling layers before being called a bug.
- No personal data appears in debug logs, screenshots or shared QA artefacts.
- Any version-dependent threshold or tool behaviour was verified against current documentation.

## Tool-specific appendix

```
GA4 and Google Tag Manager
- Debug: GTM Preview routes to DebugView automatically; or set debug_mode true on the config tag or per event;
  or use the tag assistant. DebugView shows events with parameters and user properties, in sequence.
- Limits: DebugView validates the GA4 destination only and needs debug mode on during the session; preview does
  not prove data reached reports (filters can drop it). Use realtime and standard reports as a second check.
- Common bugs: more than one Google tag per measurement id (doubles events); data layer pushed after the
  container; a stale ecommerce object (push a reset before the next event); DOM triggers instead of data-layer
  events; event-name typos; tags firing before consent.

Schema validation
- Segment Protocols: violations report, with strict controls to block and quarantine non-conforming events.
- Avo Inspector: flags shipped events that do not match the plan and lists the naming variants it sees.
- Snowplow: validates each event against its JSON Schema and routes invalid events to failed events.
- Free stack: a JSON Schema per event validated in CI, plus a scheduled check job against recent warehouse data.

Consent and modelling
- Basic consent mode sends no data for denied sessions; advanced mode sends cookieless signals. Platform
  modelling only activates above sustained volume thresholds (reported near a thousand qualifying events or ad
  clicks per day over about a week); verify current thresholds before relying on them.
```

## Sources

- InfoTrust, DebugView in GA4 (how to enable debug mode, what DebugView shows and its limits). Last verified 5 November 2026.
- Practitioner GTM tracking-plan and QA guidance (one Google tag per measurement id, data-layer order, clearing the ecommerce object, DOM versus custom-event triggers, pre-consent firing). Last verified 5 November 2026.
- Segment Protocols, Avo and Snowplow documentation (violations, blocking and quarantine, Inspector, failed events). Last verified 5 November 2026.
- Conduktor, the five pillars of data observability (freshness, volume, schema, distribution, lineage). Last verified 5 November 2026.
- Consent mode and modelling guidance (basic versus advanced mode, denied-session data, modelling volume thresholds). Last verified 5 November 2026; thresholds are platform-set and version-dependent, so re-check the platform documentation before relying on specific numbers.
- Related skills: event-tracking-plan (the spec this validates against), utm-campaign-tracking (campaign data quality), conversion-tracking-capi (ad conversions), and funnel-analysis (analysis once the data is trusted).