Concept testing

This is the exact file agents read, including the credit header and licence notice. Save it as concept-testing/SKILL.md.

---
name: concept-testing
description: Design and run the cheapest experiment that tests whether people want a product idea and will act on it, before building, using Testing Business Ideas. Carries the real rules: start from the riskiest assumption and its pass signal; pick what you are validating (problem, demand, willingness to pay, delivery); choose the cheapest test that gives strong-enough evidence; sequence problem interview, landing page or fake door, pre-sale, then concierge or wizard of oz; weight behaviour over stated intent; set the pass threshold and sample before running and do not peek. Use it when someone says concept testing, demand or desirability test, validate an idea, fake door or painted door, smoke test, landing page test, pre-sale, concierge MVP, wizard of oz, or will people pay for this. Do NOT use it to rank which assumption to test (see assumption-mapping), to run a usability test (see usability-testing), to plan generative interviews (see customer-interviews), or to launch a deceptive test without a human sign-off.
licence: CC BY 4.0. Free to use and adapt with credit to Afeez Adebayo, linking afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and the skill's download page at https://afeezadebayo.com/skills-md/product-management/product-discovery/concept-testing/.
---

# Concept testing

Concept testing is how you find out whether an idea is wanted before you spend months building it. It puts the idea, or a convincing stand-in for it, in front of real people and watches what they do, because what people do is a far better predictor than what they say. The job is not to prove the idea right. It is to design the cheapest test that could prove it wrong, run it without flinching, and let the result decide. Done well, it turns "we think customers will love this" into "forty per cent of them acted on it, here is the evidence".

The discipline has a clear backbone. Bland and Osterwalder's Testing Business Ideas sorts experiments by what they test and how strong their evidence is, and tells you to start with the smallest test that can validate or invalidate the belief. The evidence hierarchy is the spine: opinions are weak, stated intentions are moderate, and observed behaviour is strong, so a sign-up, a pre-order or a click beats any amount of "yes, I would use that". This skill makes an AI agent take a ranked assumption, choose the right test, set the pass signal and sample before running, keep the test honest, and read the result against behaviour rather than enthusiasm. It does not decide which assumption to test, run usability studies, or make the build decision. It produces the evidence those decisions need.

## When to use this skill, and when not to

Use it when someone wants to validate an idea, test demand or desirability, run a fake door or painted door, a smoke test, a landing-page test, a pre-sale or letter of intent, a concierge or wizard-of-oz test, or find out whether people will pay before building. Use it when you have a specific, important belief to check and need the cheapest honest way to check it with real behaviour.

Do not use it to decide which assumption is riskiest or worth testing; that is assumption-mapping, and this skill expects a ranked assumption with a pass signal as its input. Do not use it to test whether people can use a design they already have; that is usability-testing. Do not use it to plan and run generative interviews; that is customer-interviews, though a problem interview is often the first test in the sequence below. And do not use it to launch a deceptive test, a fake door or a wizard of oz, without a human deciding the ethics first. Validation that misleads real customers is not a shortcut; it is a reputation risk.

## What the agent needs from you before starting

Ask for these before designing a test. Do not proceed on assumptions.

1. The assumption to test and its pass signal. The specific, important belief, written as a hypothesis with a measurable signal, ideally from the assumption-mapping skill. If you only have "we should test the idea", stop and get the riskiest assumption named first.
2. What the belief is about: a problem, demand or desirability, willingness to pay, or whether you can deliver. This picks the test.
3. The segment and how to reach them. One segment at a time, and the channel (your users, ad traffic, a list, sales contacts) with rough volume, because volume decides which tests are even possible.
4. How strong the evidence needs to be. Directional (a quick read to steer), moderate (stated intent), or strong (observed behaviour). An important, idea-killing belief needs strong evidence.
5. The decision and the constraints. What a pass or fail will trigger, and the time, budget and engineering you have, which caps how expensive a test you can run.

If there is no specific assumption or no way to reach the segment, stop and ask. Do not invent a result, a conversion rate or a benchmark to fill the gap.

## How to design and run a concept test, step by step

Follow these in order. Steps 1 to 5 design the test; steps 6 to 9 run and read it.

### Step 1. Start from the riskiest assumption and its pass signal

Take the ranked assumption and its hypothesis from assumption-mapping. If the pass signal is missing, write it now: the observable thing that would count as the belief holding, with a number. A test without a pass signal written in advance cannot be failed, because the goalposts will move.

### Step 2. Name what you are validating

Sort the belief into one of four:

- Problem: does this problem exist, and does it matter to them?
- Demand or desirability: do they want this solution enough to act?
- Willingness to pay: will they pay, at your price, now?
- Delivery or feasibility: can you actually deliver the outcome?

The category picks the test and its place in the sequence.

### Step 3. Choose the cheapest test that gives strong-enough evidence

Run the cheapest test that could kill the riskiest assumption, and test one segment at a time so a weak result does not hide a strong one. The usual sequence, cheapest and earliest first:

- Problem interview (hand to customer-interviews): talk to about five people per segment about past behaviour. Move on when eight to ten people in one segment describe the same costly problem unprompted.
- Landing page or smoke test: a page that describes the offer and measures sign-ups or clicks to a call to action, usually driven by a small ad spend. Tests demand from strangers.
- Fake door or painted door: a button or entry point for the not-yet-built feature inside a real product; count the clicks, then show an honest "coming soon" and a way to stay informed.
- Pre-sale or letter of intent: ask people to pay, pre-order, or sign a non-binding intent at your real price. The strongest signal short of launch.
- Concierge or wizard of oz: deliver the outcome by hand (concierge, openly) or behind a working-looking front (wizard of oz) for a few customers, to test delivery and value before building the software.

Use the decision table in the templates to map category to test. Match the test to the evidence strength you need: a landing page gives behaviour, a survey gives only stated intent.

### Step 4. Set the pass threshold before you run

Write the threshold the metric must clear to count as a pass, and do it before the test goes live. Use real benchmarks as a sanity check, not as a target handed down from nowhere. For landing pages, Unbounce's 2024 Conversion Benchmark Report (over 57 million conversions) puts the median conversion around 6.6 per cent across industries and about 3.8 per cent for software; a test page far below that is a warning sign, not a pass. Set the threshold from your own economics where you can (what conversion makes the idea worth building), and treat benchmarks as the floor for "did anything happen at all". Verify current benchmark figures before relying on them.

### Step 5. Decide the sample size and the stop rule

Decide how much data settles it, before running:

- Interviews: about five per segment, across two or three segments, then deeper in the sharpest one.
- Quantitative tests (landing page, fake door, pre-sale): pre-set the number of visitors or the time window that gives you enough signups to tell a pass from a fail, and do not stop early because the number looks good. Peeking and stopping on a good day is how teams fool themselves.
- One variable at a time, so you know what caused the result.

This skill does not compute formal statistical power; for a test that needs a significance calculation, say so and use a proper sample-size tool. The rule here is simpler: enough data to distinguish pass from fail, fixed in advance.

### Step 6. Build the test as a draft and get sign-off

Produce the test as a draft: the landing-page copy, the fake-door placement and its honest follow-up, the pre-sale script, or the concierge plan. Anything that reaches real customers, ad spend, an email, a fake door in the live product, a pre-sale, needs explicit human confirmation before it goes live. Default to draft, never to launch.

### Step 7. Run it for the fixed window, without peeking

Run the test to the sample or time you set, keeping conditions steady. Do not change the page, the price or the audience mid-test, and do not call it early.

### Step 8. Read the result against the threshold and the evidence hierarchy

Compare the metric to the pre-registered threshold and classify: clear pass, clear fail, or inconclusive. Weight the evidence by what it is: behaviour (a sign-up, a payment, a click) counts; stated intent counts for little, because a 2006 meta-analysis found a medium change in intention produced only a small change in behaviour. An inconclusive result is a real outcome, not a failure to try harder; it usually means the test was too small or the signal too weak, so redesign rather than reinterpret.

### Step 9. Decide the next action and write it up

Turn the result into a decision: build, kill, reshape, or run a stronger test. Record it in a learning card (template below): the hypothesis, the test, the metric, the result against threshold, the decision, and the next test. Then hand the assumption back to the map so it moves on the importance-against-evidence grid, and let a human make the build-or-kill call.

## Worked example

Carried over from assumption-mapping: a project tool is considering a shared client inbox, and the riskiest assumption is that clients will reply inside the tool rather than their own email. The pass signal, set earlier, is that at least half of clients on pilot projects send their next substantive reply in-tool within two weeks.

The agent names the category as demand and delivery, and picks the cheapest test that could kill the belief: a concierge pilot on five real projects, where the team routes client replies into the tool by hand instead of building the routing. It sets the sample (five projects, two-week window), the pass threshold (50 per cent of next substantive replies in-tool), and the stop rule (read at two weeks, no peeking), all before starting, and drafts the pilot plan and the client-facing note for the team to approve and send themselves.

After two weeks, three of five projects cross the threshold and two do not, and the agent reads the behaviour rather than the warm feedback: a clear signal in agencies with a single client contact, a weak one where several people reply. It classifies the result as a conditional pass, narrows the segment, and recommends a second, cheaper landing-page test on willingness to pay before any build. It writes the learning card and hands the decision to a human. It does not decide to build the inbox.

## Templates you can use verbatim

Test card (fill before running):

```
Assumption (from assumption-mapping): <belief + why it is risky>
We are validating: <problem | demand | willingness to pay | delivery>
Test: <interview | landing page | fake door | pre-sale | concierge | wizard of oz>
Segment / channel: <one segment; how you reach them>
Metric: <the observable behaviour you will count>
Pass threshold (set now): <number the metric must clear>  Evidence needed: <directional|moderate|strong>
Sample / window (set now): <visitors or time; stop rule = no peeking>
```

Learning card (fill after running):

```
Result: <metric value> vs threshold <value>  ->  <clear pass | clear fail | inconclusive>
Evidence type: <behaviour | stated intent>  (weight behaviour; discount intent)
Decision: <build | kill | reshape | run a stronger test>
Next test: <what, or none>   Assumption moved on the map: <yes/no>
```

Decision table (what to validate -> cheapest sufficient test):

```
Problem exists?            -> problem interview (customer-interviews), past behaviour
Do they want it?           -> landing page / smoke test, or fake door in-product
Will they pay, at price?   -> pre-sale / letter of intent / crowdfunding
Can we deliver the value?  -> concierge (by hand, openly) or wizard of oz (manual behind a front)
Rule: cheapest test that can kill the riskiest assumption; one segment at a time; behaviour over intent.
```

Fake-door specification (if chosen, needs sign-off):

```
Placement: <where the button/entry sits in the live product>
Honest follow-up: "coming soon" + opt-in to hear more (never a dead end, never a charge)
Exposure cap: <small % of users / short window>   Count: <clicks / opt-ins>
Ethics: disclose it is not yet built; honour the implied promise; a human approves before launch.
```

## Common mistakes and how to avoid them

1. Testing with opinions instead of behaviour. Asking "would you use this?" and counting the yeses. Spot it when the evidence is what people said, not what they did. Fix by designing a test where people act (sign up, click, pay) and weighting behaviour over intent.
2. No threshold set in advance. The result comes in and the team decides, after the fact, that it counts as a pass. Spot it when the pass signal was never written down. Fix by setting the threshold and sample before the test goes live.
3. Peeking and stopping early. Calling the test the moment the number looks good. Spot it when the test ended off-schedule on an up day. Fix by fixing the sample or window in advance and reading only at the end.
4. Testing many things at once. Changing the offer, the price and the audience together, so no one can say what caused the result. Spot it when more than one variable moved. Fix by testing one segment and one variable at a time.
5. Deceptive tests with no ethics. A fake door that leads to a dead end, a pre-sale for something you cannot deliver, a wizard of oz that quietly keeps people's money. Spot it when the test would embarrass you if the customer knew. Fix by disclosing, capping exposure, honouring the implied promise, and getting a human to approve the ethics first.

## Quality bar: check the output against this before handing it over

- The test starts from a specific, important assumption with a pass signal, ideally from assumption-mapping.
- What is being validated (problem, demand, willingness to pay, delivery) is named, and the test is the cheapest one that gives strong-enough evidence for it.
- The pass threshold and the sample or window are set before the test runs, with a real benchmark used only as a sanity check.
- The metric counts behaviour, and stated intent is discounted, not treated as proof.
- Anything that reaches real customers is produced as a draft for human sign-off, not launched.
- Deceptive tests (fake door, wizard of oz) carry an ethics note: disclosure, an exposure cap, and the implied promise honoured.
- The result is classified as clear pass, clear fail or inconclusive against the threshold, and turned into a decision and a learning card.
- Nothing is invented: no fabricated conversion rates, sample sizes, benchmarks or results; where a figure is unknown, the output says so.
- Ranking the assumption, usability testing and generative interviews are left to their own skills; the build-or-kill call is left to a human.

## Guardrails

- Anything reaching real customers needs a human. Launching a landing page with ad spend, placing a fake door in the live product, emailing a pre-sale, or running a concierge pilot all reach real people. Produce them as drafts and get explicit confirmation before they go live.
- Deceptive tests carry duties. Fake doors and wizard-of-oz tests show people something that does not fully exist. Disclose that it is not built yet, cap exposure to a small share and a short window, never charge for or promise what you cannot deliver, and have a human approve the ethics before launch. If a test would embarrass the team if the customer knew, do not run it.
- Never fabricate results. A concept test with no real data is worthless, and invented conversion rates are worse than none. If the test did not run or did not reach enough people, the honest output is "inconclusive, here is what to run next", not a number.
- Protect personal data. Sign-ups, pre-sale payments and pilot customers bring names, emails and payment detail. Collect only what the test needs, keep it where your organisation already keeps customer data, use anonymised labels in write-ups, and never paste contact or payment details into prompts or third-party tools.
- Mind the cost. Ad spend, engineering for a fake door and customer time all cost. Reserve the expensive tests for important, low-evidence assumptions, and start with the cheapest test that could kill the belief.
- Stay in scope. This skill designs and reads validation experiments. It does not rank assumptions (assumption-mapping), test usability of a design (usability-testing) or plan generative interviews (customer-interviews). It produces evidence and a recommendation; a human owns the build-or-kill decision. It is not a guarantee that a passing test means a successful product.

## Tool-specific appendix

The method is tool-agnostic, but the tests need some plumbing. Build landing pages and fake doors with a page builder or your own product, and wire the metric to analytics with the goal defined before launch, not reconstructed after. Use a scheduling tool for problem interviews, as in the customer-interviews appendix. Run wizard-of-oz and concierge tests with whatever no-code or manual process delivers the outcome, and keep a clear line between the manual back end and what the customer sees. Keep pre-sale payments in a real, refundable mechanism so you can honour or return them. For ad-driven tests, cap the spend in advance. Whatever the tools, the rules do not change: a pass signal and sample set before running, behaviour over intent, one variable at a time, and a human sign-off for anything that reaches a customer.

## Sources

- David J. Bland and Alexander Osterwalder, "Testing Business Ideas", and Strategyzer's summary: the experiment library sorted by what it tests and its evidence strength, discovery versus validation experiments, the evidence hierarchy (opinions weak, stated intent moderate, behaviour strong), and "start with the smallest possible test". Last verified 13 November 2026.
- Koji, "Which validation test to run first", "Fake door testing", "Wizard of Oz testing", "Smoke tests": run the cheapest test that can kill the riskiest assumption; the sequence problem, demand, willingness to pay, delivery; definitions of landing page, fake door, concierge and wizard of oz; about five interviews per segment and eight to ten unprompted problem mentions before demand testing; test one segment at a time; write the pass signal before the test. Last verified 13 November 2026.
- Unbounce, 2024 Conversion Benchmark Report (over 57 million conversions, 41,000-plus pages): median landing-page conversion around 6.6 per cent across industries and about 3.8 per cent for software. Verify current figures before relying on them. Last verified 13 November 2026.
- Webb and Sheeran, 2006 meta-analysis of intention and behaviour: a medium change in intention (d = 0.66) produced only a small change in behaviour (d = 0.36), so stated intent is a weak predictor. Last verified 13 November 2026.
- CB Insights, 2021 analysis of start-up post-mortems (111 cases): 35 per cent cited "no market need" as a reason for failure, the case for testing demand before building. Last verified 13 November 2026.
- Marty Cagan, Silicon Valley Product Group, "Product Risk Taxonomies": value and viability risk as what demand and pricing tests address. Last verified 13 November 2026.
- Existing skills reviewed for structure and gaps: borghei "brainstorm-experiments" (strong on the XYZ hypothesis and an experiment list, but puts sample size out of scope and is thin on choosing between tests and reading results against benchmarks), majiayu000 "product-discovery" (broad market-research scope, no concrete experiment design or thresholds). This skill adds the sequencing rule, pre-registered thresholds with real benchmarks, sample and stop rules, and an ethics guardrail for deceptive tests. Last verified 13 November 2026.

---
1.0.0, 13 November 2026: first release. Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link afeezadebayo.com and the skill's download page. This is decision-support guidance, not a substitute for testing with your own customers.