Lifecycle Experimentation and Holdout Testing SKILL.md

This is the exact file agents read, including the credit header and licence notice. Save it as lifecycle-experimentation/SKILL.md.

---
name: lifecycle-experimentation
description: "Measure what lifecycle marketing really adds: flow, channel and global holdouts, A/B tests on meaningful changes such as offers, timing and channel mix, hypotheses, sample size and duration planning, randomisation, guardrail metrics, honest analysis with confidence intervals and a test log, for Klaviyo, Braze, Customer.io and similar platforms. Use whenever someone asks about email A/B testing, holdout groups, incrementality, whether a flow works, testing discounts, sample size, or why attributed email revenue looks too high."
license: CC-BY-4.0
metadata:
  author: Afeez Adebayo
  author_url: https://afeezadebayo.com
  author_linkedin: https://www.linkedin.com/in/afeez-adebayo/
  author_x: https://x.com/za_icon
  source: https://afeezadebayo.com/skills-md/marketing/crm-lifecycle/lifecycle-experimentation/
  category: marketing
  subcategory: crm-lifecycle
  version: "1.0.0"
  created: "2026-10-10"
  updated: "2026-10-10"
---

# Lifecycle Experimentation and Holdout Testing

Most lifecycle teams test subject lines and button colours, then report flow revenue that the platform attributes to every email someone opened before buying. Neither answers the question leadership cares about: what would have happened without this programme? Holdout tests answer it. They are the single most useful habit a lifecycle team can build.

This skill designs experiments that measure incremental impact: holdouts for flows and channels, A/B tests for meaningful changes, sample size and duration planning, and honest reporting of results, including when the answer is "no effect".

## Operating principles
1. **Incrementality over attribution.** The difference between treated and untreated groups is the effect; attributed revenue is not.
2. **Test decisions, not decorations.** Offers, timing, channel mix, message angle and whole flows change outcomes; tiny style tweaks rarely do.
3. **Plan size and duration first.** Small tests on small lists produce noise that looks like results.
4. **Choose the metric before the test.** Conversions, revenue, retention or activation, defined in advance.
5. **Report every result.** Null and negative results save money and prevent repeating mistakes.

## Test types
| Type | Question | Example |
|---|---|---|
| Flow holdout | Does this flow add value at all? | 10% of cart abandoners get no cart emails |
| Channel holdout | Does adding SMS increase conversions after cost? | 20% of eligible customers get email only |
| Global holdout | What does the whole lifecycle programme add? | 5% of customers receive no marketing messages for a quarter |
| A/B test | Which version performs better? | Offer versus no offer in a win-back email |
| Multivariate | Which combination works best? | Rarely worth it for lifecycle; needs very large audiences |

## When to use, and when not to
Use it before scaling a new flow, when attributed revenue looks too good, when deciding on discounts, when adding paid channels like SMS, and to set a testing roadmap.

Don't use it to write the messages (`lifecycle-email-copy-design`) or to build reporting dashboards (`lifecycle-reporting-attribution`).

## Inputs to ask for, in one message
1. The decision the test should inform.
2. Audience size per week or month and baseline conversion rate.
3. Platform capabilities: random splits, holdout groups, conversion tracking.
4. The primary metric and how it is measured.
5. Past tests and results.

## Workflow

### Step 1. Write a hypothesis
"Adding a post-purchase education sequence will increase 90-day repeat purchase rate for first-time buyers, because they'll get better results from the product."

### Step 2. Choose the design
Holdout for "does it work?"; A/B for "which is better?". Randomise at the person level.

### Step 3. Estimate sample size
Use the baseline rate, the smallest effect worth detecting and standard settings (often 95% confidence, 80% power) in a sample size calculator. If the required audience exceeds what you'll get in a reasonable time, test a bigger change or a higher-volume flow.

### Step 4. Set duration
Run for at least one full business cycle (often two to four weeks), and long enough for the metric to happen (repeat purchase may need 60 to 90 days).

### Step 5. Protect the test
No peeking-driven early stops, no changes mid-test, consistent exclusions, and check that groups are balanced at the start.

### Step 6. Analyse
Compare the primary metric with confidence intervals. Check guardrail metrics (unsubscribes, complaints, margin).

### Step 7. Decide and document
Ship, iterate or stop. Record the hypothesis, design, result and decision in a test log.

## Sample size reality check
| Baseline conversion | Relative lift to detect | Approximate people needed per group |
|---|---|---|
| 2% | 10% | Tens of thousands |
| 2% | 30% | A few thousand |
| 10% | 10% | Over ten thousand |
| 10% | 30% | Around a thousand to two thousand |
These are rough orders of magnitude at common settings; always calculate for your own numbers.

## Holdout sizing
- Flow holdouts of 5 to 20% are common; larger holdouts give answers faster but cost more in lost conversions if the flow works.
- Global holdouts are usually small (for example 2 to 5%) and run for a quarter.
- Rotate global holdout members periodically so the same customers aren't excluded forever.

## Testing roadmap: what to test first
| Priority | Test | Why first |
|---|---|---|
| 1 | Holdout on the highest-attributed flow (often abandoned cart or welcome) | Reveals how much attributed revenue is real |
| 2 | Discount versus no discount in win-back and cart flows | Margin impact is large and often unknown |
| 3 | Channel holdout for SMS or WhatsApp | Costs per message make incrementality essential |
| 4 | Onboarding sequence versus minimal onboarding | Activation and retention effects compound |
| 5 | Frequency test on campaigns (for example 3 vs 5 per week) | Balances revenue against unsubscribes and complaints |
| 6 | Global holdout for a quarter | Shows total programme contribution |

## Test log template
```
| ID | Date | Flow or campaign | Hypothesis | Design | Primary metric | Result (with range) | Guardrails | Decision | Owner |
```
Keep the log where the whole team can see it, and review it quarterly to spot patterns.

## Analysing a holdout
1. Confirm groups were similar at the start (size, value, recency).
2. Calculate the conversion rate and revenue per person in each group.
3. Incremental conversions = (treated rate minus holdout rate) multiplied by treated group size.
4. Incremental revenue = incremental conversions multiplied by average order value, or compare revenue per person directly.
5. Subtract costs (discounts, SMS and WhatsApp fees) for incremental profit.
6. Report the range, not just the point estimate.

## Guardrail metrics
- Unsubscribe and complaint rates (deliverability risk).
- Margin per order (discount tests).
- Support contacts (confusing messages).
- Long-term retention (short-term wins that hurt loyalty).

## Edge cases and judgement calls
| Situation | What to do |
|---|---|
| Small list | Test bigger changes, combine similar flows, or use longer durations |
| Leadership resists holdouts | Start with one flow and a small holdout; show the value of the answer |
| Transactional or legally required messages | Never hold out; test only the marketing elements |
| Results differ by segment | Pre-register segments of interest; avoid fishing for a winning slice |
| Platform reports opens as the metric | Use clicks or conversions; opens are unreliable after Apple Mail Privacy Protection |

## Handoffs to related skills
- Programmes to test: `onboarding-activation`, `retention-churn-prevention`, `win-back-reactivation`
- Channel tests: `sms-whatsapp-lifecycle`
- Message variants: `lifecycle-email-copy-design`
- Reporting incrementality: `lifecycle-reporting-attribution`

## Worked example
A beauty retailer's platform credited its abandoned cart flow with a large share of email revenue. A four-week test held out 15% of cart abandoners from the flow. Purchase rates were higher in the treated group, but the incremental lift was about a third of the attributed revenue. A second test found the third email, which carried a 15% discount, added almost nothing beyond the first two, while cutting margin. The team removed the discount email, kept a permanent 5% holdout on the flow and reported incremental revenue to leadership from then on.

## Output format
```
# Test plan: <name>, <date>
Decision it informs:
Hypothesis:
Design (holdout or A/B), randomisation unit, groups and split:
Primary metric and guardrails:
Sample size calculation and duration:
Risks and protections:
# Test result
Result with confidence interval, guardrails, decision, next test
```

## Common mistakes
- Reporting attributed revenue as impact.
- Stopping tests early when results look good.
- Tests too small to detect realistic effects.
- Testing trivial changes while big decisions go untested.
- Not logging results, so lessons are lost.

## Questions to ask before you test
- What decision depends on this result?
- What is the smallest effect that would change the decision?
- How many people enter this flow each week?

## Self-check before you answer
- Is the design matched to the question (holdout or A/B)?
- Are sample size and duration calculated?
- Are primary and guardrail metrics defined in advance?
- Is the result logged with a decision?

## Further reading
- Evan Miller sample size calculator: https://www.evanmiller.org/ab-testing/sample-size.html
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020)

---
Created by Afeez Adebayo, Marketing, SEO & Growth Expert
Website: https://afeezadebayo.com
LinkedIn: https://www.linkedin.com/in/afeez-adebayo/
X: https://x.com/za_icon
Source: https://afeezadebayo.com/skills-md/marketing/crm-lifecycle/lifecycle-experimentation/

Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). You may copy, share and adapt this skill, including for commercial use, as long as you credit Afeez Adebayo and link to the source URL above. Keep this notice in any copy or adapted version.