Usability testing

This is the exact file agents read, including the credit header and licence notice. Save it as usability-testing/SKILL.md.

---
name: usability-testing
description: Plan, run and analyse a usability test that shows whether real people can use an existing design or prototype, using think-aloud and standardised metrics rather than opinion. Scope two to five scenario tasks that give context without naming the button; set method and sample before running (about five per user group for a qualitative round, twenty or more to benchmark); let the participant work unaided; measure task success, time and errors, read ease and usability with SEQ and SUS, and rate each issue on Nielsen's 0 to 4 severity scale by frequency, impact and persistence. Use it when someone says usability test, user test, think-aloud, moderated or unmoderated test, test this prototype, write test tasks or a moderator script, SUS or SEQ, or analyse findings. Do NOT use it to test whether people want the idea (see concept-testing), to plan discovery interviews (see customer-interviews), to pick which assumption to test (see assumption-mapping), or to run a live A/B test (see product-experimentation).
licence: CC BY 4.0. Free to use and adapt with credit to Afeez Adebayo, linking afeezadebayo.com, https://www.linkedin.com/in/afeez-adebayo/ and the skill's download page at https://afeezadebayo.com/skills-md/product-management/product-discovery/usability-testing/.
---

# Usability testing

Usability testing is how you find out whether people can actually use a design before you ship it, by watching them try. You give a real user a realistic task, ask them to think aloud, and stay quiet while they work, because what they do with the interface tells you far more than what they say about it afterwards. The job is not to prove the design works. It is to put it in front of people who have never seen it, let them struggle without rescue, and collect the struggles as evidence. Done well, it turns "the team thinks this flow is clear" into "four of five users missed the save button, three gave up, here is the recording".

The discipline has a clear backbone. Jakob Nielsen's work at the Nielsen Norman Group set the shape: watch a handful of users attempt realistic tasks, because about five users per group reveal most of the problems, and several small rounds beat one large study. Steve Krug's Rocket Surgery Made Easy made it cheap and routine. Measurement comes from Jeff Sauro and James Lewis: task success, time on task and errors for behaviour, the Single Ease Question and the System Usability Scale for perception, each with a known benchmark. This skill makes an AI agent scope the tasks, write them so they do not lead, set the method and sample before running, keep the session honest, measure both behaviour and perception, rate issues by severity, and separate a real finding from a one-off. It does not decide whether the idea is wanted or which assumption to test; it finds out whether the thing in front of the user can be used.

## When to use this skill, and when not to

Use it when someone wants to run a usability test or user test, a think-aloud or task-based session, a moderated or unmoderated remote test, or a test of a prototype, wireframe, live product or competitor's product. Use it when they ask to write test tasks, a moderator script or a test plan, to pick a sample size, to run SUS or SEQ, to rate issue severity, or to analyse and report usability findings. Use it when there is a design that people are meant to use and a real question about whether they can.

Do not use it to find out whether people want the idea at all or will act on it; that is concept-testing, which measures demand and desirability, not whether an existing design can be operated. Do not use it to plan and run generative discovery interviews about people's lives and past behaviour; that is customer-interviews. Do not use it to decide which belief is riskiest and worth testing first; that is assumption-mapping. Do not use it to run a live, randomised A/B test on production traffic; that is product-experimentation. Heuristic evaluation, where an expert reviews a design against usability principles with no users, is a useful cheap complement before a test, but it is not a substitute for watching real people, so this skill keeps users at the centre.

## What the agent needs from you before starting

Ask for these before designing a test. Do not proceed on assumptions.

1. The design and its state. A link to or description of the prototype, wireframe, live product or competitor you want tested, and how finished it is. A clickable prototype can be tested for task flow; a static mock cannot be tested for interaction.
2. The research questions. The two to five specific things you need to learn, framed as questions a test can answer, for example "can a new user complete checkout without help?" Not "is it good?"
3. Who the users are. The target segment, and whether there are several distinct groups (new versus returning, admin versus end user), because each distinct group needs its own small sample. Say if any users rely on assistive technology.
4. The method and constraints. Whether you can moderate sessions live, your timeline and budget, the devices that matter, and the tools you have for recording or unmoderated testing.
5. The decision this feeds. What a pass or a list of problems will trigger (ship, fix and retest, redesign), so the output is sized to the decision.

If there is no testable design, or no way to reach real users of the segment, stop and ask. Do not invent a success rate, a severity count or an SUS score to fill a gap.

## How to plan and run a usability test, step by step

Follow these in order. Steps 1 to 5 design the test; steps 6 to 10 run it and read it.

### Step 1. Decide qualitative or quantitative, and the method

Name the goal first, because it sets everything after it.

- Qualitative (find problems): watch a few users, listen to think-aloud, collect issues. This is the common case and what most teams need. Use it to improve a design in rounds.
- Quantitative (benchmark): measure task success, time and satisfaction with enough users to report a number you can compare across rounds or products.

Then pick moderated or unmoderated:

- Moderated (a facilitator runs the session, in person or remote): choose it for early, rough or complex designs, when you need to ask "what made you do that?" in the moment. Costs more time per session.
- Unmoderated (the user works alone through a tool): choose it for simpler, more finished flows, for speed and scale, and when you want natural behaviour with no facilitator present. You lose the live follow-up.

If in doubt for a design still in flux, run a small moderated qualitative round.

### Step 2. Scope to a few real tasks

List the two to five tasks that matter for the research questions, and stop there. More than about five tasks in one session tires the participant and thins the data. Each task must be a thing a real user actually does, in the order they would meet it, and each must have a defined success criterion you can judge, for example "an order confirmation is reached".

### Step 3. Write the tasks as scenarios that do not lead

Give the context and the motivation, not the solution. A good task describes a situation and a goal and lets the user find the path; a bad task names the button. Write "You have a question about an order that arrived damaged. Sort it out." Do not write "Click the Help link and open a support ticket." Keep the wording neutral: no "easily", no "simply", no words that appear in the interface, because repeating the label hands the user the answer and voids the result.

### Step 4. Set the sample and the stop rule before you run

Fix the number before the first session, so a good or bad early run cannot move the goalposts.

- Qualitative, one user group: about five participants per round. Five users reveal roughly 85 per cent of the problems a single round can find, on Nielsen's model (the share one user finds averaged about 31 per cent across studies), and returns fall away after that. Spend a larger budget on several rounds of five, not one round of fifteen.
- Several distinct user groups: about three to four participants per group, each group run and read separately.
- Quantitative benchmark: about twenty or more participants for a usable completion-rate figure, and closer to forty where you need tighter confidence. If you need a formal significance test, say so and use a proper sample-size calculator; this skill fixes a planned sample, it does not compute statistical power.
- Over-recruit by 10 to 20 per cent to cover no-shows, and screen out friends, family and colleagues, whose feedback is biased.

### Step 5. Write the moderator script and the measures

Draft the session: a short warm-up to put the participant at ease and explain think-aloud, the tasks in order, a post-task question after each one, and a short debrief. Decide the measures now:

- Behaviour: task success (completed unaided, completed with difficulty, failed), time on task, and errors or wrong turns. Report success as a rate with its denominator, "8 of 10", never a bare percentage that hides the sample.
- Perceived ease, per task: the Single Ease Question, a seven-point scale asked right after each task, "How easy or difficult was it to complete this task?" The mean across tasks and studies sits around 5.5, and an average SEQ tends to go with a task completion rate near 71 per cent, so a task scoring well below 5.5 is a flag.
- Perceived usability, whole design: the System Usability Scale, ten items scored into a single 0 to 100 number. The average SUS is about 68, so 68 is a C, not a good score; 80.8 and above is an A, and around 80 is a common industrial goal. Note that SUS benchmarks come from finished products, so treat a prototype's SUS as directional.

### Step 6. Pilot the session with one person

Run the whole script with one colleague or spare participant before the real sessions. A pilot catches tasks that confuse, instructions that leak the answer, prototypes that break, and timing that overruns. Fix those, then start the real sessions. A pilot is not a test participant and its data does not go in the results.

### Step 7. Run the session: let them struggle, do not lead and do not help

Keep the facilitator to roughly a fifth of the talking; the participant does the rest. The two hardest rules, and the ones that make or break the test:

- Do not lead. Ask open, neutral prompts, "what are you trying to do now?", "what did you expect?", never "did you see the big blue button?" A leading prompt plants the behaviour you then record.
- Do not help. When the user is stuck, sit with the silence. Being stuck is the finding. Only step in if they are completely blocked and the session would otherwise end, and note that you did. Rescuing a user erases the problem you came to find.

Record with consent (see guardrails). In unmoderated tests, the task wording has to carry the whole session, so it must be even more carefully non-leading, since no facilitator is there to recover a misread task.

### Step 8. Capture issues per participant, tagged to evidence

Write one short record per participant: which tasks they completed, where they struggled, their SEQ per task, and the SUS if used. Tag every observed problem to the participant and the task, and separate what they did (behaviour, strong evidence) from what they said (opinion, weaker). Do this straight after each session while it is fresh; do not wait until the end and reconstruct it.

### Step 9. Cluster issues and rate severity, with more than one rater

Pool the per-participant records, merge duplicate issues, and count how many participants hit each one, reported as a rate with the denominator. Rate each issue on Nielsen's severity scale:

- 0: not a usability problem.
- 1: cosmetic, fix if time allows.
- 2: minor, low priority.
- 3: major, important to fix.
- 4: catastrophe, fix before release.

Judge severity by frequency (how many hit it), impact (how badly it blocks them) and persistence (a one-off hurdle or a repeated one; persistence is the factor teams most often under-weight). Have at least two or three people rate severity independently and reconcile, because a single rater's severity is unreliable. Remember that frequency from only five participants is a rough signal, not a precise rate, so describe it as such.

### Step 10. Turn findings into a prioritised, decision-ready report

Report the task success rates with denominators, the SEQ and SUS against their benchmarks, and the issues ranked by severity, each with the evidence (how many, which task, a clip or quote). Keep severity and priority in separate columns: severity is harm to the user; priority is a business call that also weighs reach and effort, and feeds a framework like the one in prioritisation-frameworks. Recommend the fixes and a retest of the ones that mattered, and leave the ship-or-hold decision to a human. Krug's rule applies: fix the most serious problems first, and make the smallest change that fixes each one.

## Worked example

Carried over from the product's discovery work: a project tool has redesigned its invoice-sending flow and needs to know whether freelancers can send an invoice without help before it ships. The research question is exactly that, the segment is solo freelancers who already use the tool, there is one user group, and the design is a clickable prototype.

The agent scopes three tasks (create an invoice, add a client, send it), writes them as scenarios ("A client has just asked you to bill them for last month's work. Invoice them through the tool."), and fixes the method as moderated qualitative with five participants plus one over-recruit, set before any session. It drafts the script with a warm-up, the three tasks, an SEQ after each, and a SUS at the end, then pilots it with a colleague and finds that the first task's wording named the "New invoice" button, so it rewrites the task to remove the label.

Across the five sessions the agent stays near a fifth of the talking, asks "what did you expect to happen?" when a participant pauses, and does not point to anything. It records that four of five participants could not find how to add a client mid-invoice (severity 4, hit by four of five, blocks the core task), two of five mistyped the amount because the currency field was unlabelled (severity 3), and SEQ for the add-client task averaged 2.8 against the 5.5 benchmark while SUS came in at 59. Two team members rate severity independently and agree. The report ranks the add-client failure first with the clips, recommends a fix and a five-user retest of that task, and hands the ship decision to the product lead. It does not declare the flow ready.

## Templates you can use verbatim

Test plan (fill before running):

```
Design under test: <prototype | wireframe | live product | competitor> + link/state
Research questions (2 to 5): <specific, answerable by a test>
Users: <segment; distinct groups?; any assistive-technology users?>
Method: <moderated | unmoderated> + <qualitative | quantitative> + <remote | in person> + devices
Sample (set now): <about 5 per group qual; 20+ quant; +10 to 20% over-recruit; no friends/colleagues>
Tasks (2 to 5): <scenario + success criterion each>
Measures: <task success as rate; time on task; errors; SEQ per task; SUS if benchmarking>
Decision this feeds: <ship | fix and retest | redesign>
```

Task card (one per task):

```
Scenario (context + motivation, no UI labels, no "easily"/"simply"): <...>
Goal the user must reach: <...>
Success criterion (observable): <e.g. order confirmation reached>
SEQ asked after: "How easy or difficult was it to complete this task?" (1 to 7)
```

Moderator do and do-not:

```
DO: explain think-aloud; ask "what are you trying to do?" / "what did you expect?"; stay ~20% of talking; sit with silence
DO NOT: name buttons; ask leading questions; help a stuck user (that is the finding); react to right/wrong
Only intervene if fully blocked; note it; it invalidates that task for that user
```

Findings record (one per participant, filled right after):

```
Participant <id/segment>  Device <...>
Task 1 <completed unaided | with difficulty | failed>  SEQ <1-7>  Notes <behaviour vs opinion>
Task 2 ...   Task 3 ...
SUS (if used): <0-100>   Standout quotes/clips (tagged to task): <...>
```

Issue + severity table (after clustering, 2 to 3 raters):

```
Issue | Task | Frequency (n of N) | Impact | Persistence | Severity 0-4 | Evidence (clip/quote)
Rule: severity by frequency + impact + persistence; report frequency with denominator;
keep severity separate from priority; fix severity 4 before release.
```

## Common mistakes and how to avoid them

1. Leading the participant. Naming the button, asking "wasn't that easy?", or writing a task that repeats an on-screen label. Spot it when the task tells the user where to click or the prompt suggests an answer. Fix by writing scenario tasks with neutral wording and asking only open questions.
2. Rescuing a stuck user. Jumping in the moment they hesitate, so the problem never gets recorded. Spot it when the facilitator is doing most of the talking. Fix by keeping to about a fifth of the talking, sitting with silence, and treating being stuck as the finding.
3. Confusing opinion with behaviour. Reporting "users said they liked it" when four of five could not finish the task. Spot it when the headline is a quote, not a success rate. Fix by leading with what people did, measured, and treating what they said as secondary.
4. Over-reading five users as numbers. Turning "three of five" into "60 per cent of users" and ranking issues on it as if it were precise. Spot it when a small qualitative sample is reported as a confident percentage. Fix by reporting the rate with its denominator, calling five-user frequency a rough signal, and benchmarking with twenty or more when you need a real number.
5. Testing usability to answer a demand question. Running a polished usability test on an idea nobody has shown they want, and reading task success as proof the product will succeed. Spot it when the real risk is whether anyone wants it, not whether they can use it. Fix by handing demand and desirability to concept-testing and keeping this skill to can-they-use-it.

## Quality bar: check the output against this before handing it over

- The test is aimed at whether people can use an existing design, with two to five specific research questions, not whether the idea is wanted.
- There are two to five scenario tasks, each with a success criterion, worded without UI labels or leading words.
- The method (moderated or unmoderated, qualitative or quantitative) and the sample are set before running, with about five per group for a qualitative round and twenty or more for a benchmark.
- Behaviour is measured (task success as a rate with denominator, time, errors) and perception is measured where relevant (SEQ against about 5.5, SUS against about 68), with benchmarks used as a sanity check, not a target.
- The session rules are stated: think-aloud, facilitator to about a fifth of the talking, no leading, no helping, and any intervention noted.
- Issues are clustered, counted with denominators, and rated on the 0 to 4 severity scale by frequency, impact and persistence, with more than one rater.
- Accessibility is addressed where users rely on assistive technology, and the report says plainly that a user test is not a WCAG conformance check.
- Consent and recording are handled, and no participant's personal data is pasted into prompts or third-party tools.
- Nothing is invented: no fabricated success rates, severity counts, SEQ or SUS scores; where a figure is unknown, the output says so.
- Severity is kept separate from priority, fixes are recommended with a retest, and the ship-or-hold decision is left to a human.

## Guardrails

- Recording and consent come first. Get explicit, informed consent before recording a session, say what will be recorded and how it will be used, and let the participant stop at any time. Do not record covertly, and do not reuse a recording beyond what was agreed.
- Protect personal data. Sessions capture faces, voices, screens and sometimes real account data. Store recordings where your organisation already keeps personal data, use participant labels rather than names in write-ups, blur or avoid real personal and payment detail on shared screens, and never paste recordings, transcripts or participant contact details into prompts or third-party tools that are not covered by your data agreements.
- Accessibility testing is not a conformance check. Including participants who use assistive technology finds real problems, but a handful of sessions cannot tell you whether a product meets WCAG, and one disabled participant does not represent everyone with that disability. Pair user testing with a proper conformance evaluation, recruit a range where you can, and state the scope and participants in the report.
- Do not fabricate results. A usability finding with no observed session behind it is worthless, and invented success rates or SUS scores are worse than none. If the test did not run or did not reach real users, the honest output is "not yet tested, here is the plan", not a number.
- Keep the sample in proportion. Five users is enough to find most problems, not to produce a reliable percentage. Report frequencies with denominators, flag small samples as directional, and do not let a confident-looking number outrun its evidence.
- Stay in scope. This skill plans, runs and reads usability tests of a design. It does not test demand or desirability (concept-testing), plan generative interviews (customer-interviews), rank which assumption to test (assumption-mapping) or run live experiments (product-experimentation). It produces evidence and a prioritised recommendation; a human owns the ship-or-hold decision, and passing a usability test does not guarantee a successful product.

## Tool-specific appendix

The method is tool-agnostic, but a test needs some plumbing. For moderated remote sessions use a video-call tool with screen share and recording, with consent captured before you hit record; for unmoderated tests use a dedicated platform that delivers tasks, records screen and voice, and collects SEQ and SUS automatically. Test clickable prototypes from your design tool directly, and test a live or competitor product in a clean browser profile so saved logins and autofill do not distort the run. Keep the SUS as its standard ten items and score it the standard way: odd items contribute the scale position minus one, even items contribute five minus the scale position, sum the contributions and multiply by 2.5 for a 0 to 100 score. Store recordings in your organisation's existing secure location, not a personal drive or a general-purpose chatbot. Whatever the tools, the rules hold: a sample set before running, scenario tasks that do not lead, no rescuing, behaviour measured over opinion, severity rated by more than one person, and consent for every recording.

## Sources

- Jakob Nielsen, Nielsen Norman Group, "Why You Only Need to Test with 5 Users": the problems-found model N(1-(1-L)^n) with L averaging 31 per cent, so five users reveal about 85 per cent of problems in a round; run several small rounds rather than one large study; distinct user groups need their own small samples; quantitative studies need about 20 and card sorting about 15. Last verified 14 November 2026.
- Nielsen Norman Group, "Usability Testing 101": the definition and elements (facilitator, tasks, participant); moderated versus unmoderated and qualitative versus quantitative; the think-aloud protocol; the facilitator must not influence the participant and task wording must not prime; the three-day simplest-study shape and iterative design. Last verified 14 November 2026.
- Jeff Sauro, MeasuringU, "5 Ways to Use the System Usability Scale" and related SUS material: the average SUS of about 68 across a 500-product database, about 72 for consumer software, and the note that SUS benchmarks come from finished products, not prototypes. Last verified 14 November 2026.
- Jeff Sauro, MeasuringU, "The Evolution of the Single Ease Question": the seven-point SEQ, the wording "How easy or difficult was it to complete this task?", a mean that settled around 5.5, and an average SEQ associated with a task completion rate near 71 per cent. Last verified 14 November 2026.
- James R. Lewis, "The System Usability Scale: Past, Present, and Future", Journal of Usability Studies, 2018: the SUS scoring method (odd items scale position minus 1, even items 5 minus scale position, sum times 2.5, range 0 to 100) and the Sauro-Lewis curved grading scale (A from 80.8, C centred on 68, F below 51.7), with 80 a common industrial goal; SUS measures perceived usability. Last verified 14 November 2026.
- W3C Web Accessibility Initiative, "Involving Users in Evaluating Web Accessibility": testing with disabled users finds problems conformance checks miss but is not itself a conformance test, so combine it with evaluating conformance to WCAG; recruit a range and match assistive-technology experience to the audience; do not generalise one person's input to all people with that disability, and state the study's scope and participants. Last verified 14 November 2026.
- Koji, "Usability Issue Severity Ratings": Nielsen's 0 to 4 severity scale (0 not a problem, 1 cosmetic, 2 minor, 3 major, 4 catastrophe, fix 4s before release); severity judged by frequency, impact and persistence, with persistence often under-weighted; use about three independent raters; report frequency as a rate with a denominator; keep severity separate from priority and feed it into a framework like RICE. Last verified 14 November 2026.
- Lyssna, "Usability test plan": moderated (about five to eight participants) versus unmoderated (about twenty to forty) trade-offs; what a test plan contains; scenario tasks that give context without naming the solution; over-recruit 10 to 20 per cent; limit objectives and screen out friends, family and colleagues; leading questions and vague goals as common mistakes. Last verified 14 November 2026.
- Steve Krug, "Rocket Surgery Made Easy": do-it-yourself testing with a few users done regularly, a simple moderator script, and the rule to fix the most serious problems first and make the smallest change that fixes each. Last verified 14 November 2026.
- Existing skills reviewed for structure and gaps: rampstackco "usability-testing" (strong on a moderated script, think-aloud and a qualitative severity scale, but carries no standardised metrics such as task success rate, SUS or SEQ, no decision thresholds, no accessibility and thin consent, and no verification of findings), prorise-cool "usability-testing" and oakoss "usability-tester" (cover scripts, moderated and unmoderated, and severity, but show no sample-size rules, named metric benchmarks, ethics or quality bar). This skill adds standardised metrics with benchmarks, sample and stop rules set in advance, multi-rater severity tied to frequency reported with a denominator, accessibility with the WCAG caveat, consent and data guardrails, and a quality bar. Last verified 14 November 2026.

---
1.0.0, 14 November 2026: first release. Use, adapt and share this skill under CC BY 4.0: credit Afeez Adebayo, link afeezadebayo.com and the skill's download page. This is decision-support guidance, not a substitute for testing with your own users.