Skip to main content

UX and research · Intermediate · 15 min

Run an unmoderated usability test

By Padding · Updated 18 Nov 2026
An unmoderated testing tool, a clickable prototype and a spreadsheet · November 2026

You will have results from at least ten unmoderated sessions on one prototype, cleaned of low-quality sessions, with a short write-up that says what the data shows and what it cannot.

On this page

In an unmoderated usability test, participants work through your tasks on their own, in their own time, while a testing tool records what they do. Nobody is on the call to clarify a task, notice a hesitation or ask "what were you expecting there?". That is what makes it fast and cheap: twenty sessions can come back overnight. It is also what makes it easy to fool yourself.

The method is good at one kind of question: can people complete a well-defined task, and where do they go wrong? It is poor at the questions behind that: why someone went wrong, what they believed the screen meant, whether they would bother in real life. Everything below is set up to get clean answers to the first kind and to stop you claiming answers to the second.

The example throughout is a prototype of a booking flow for a town library's study rooms. The team wants to know whether people can book a room for a given day and time, and whether they can find how to cancel. Swap in your own product as you go.

Step 1 · Decide what this test can answer

Before writing any tasks, write down the two or three questions the test must answer, and check each one against what an unmoderated session can show.

Good questions for this method are about behaviour you can observe without asking: "Can people book a two-hour slot on a given day?", "Where do people look for cancellation?", "How many people pick the wrong room size?". Poor questions are about motives, trust or comparison with real life: "Why do people prefer the calendar view?", "Would people use this instead of phoning?". If most of your questions are the second kind, run a moderated test instead.

For the library flow, the team writes:

  1. Can people book a room for a stated day, time and group size without help?
  2. Can people find how to cancel a booking they have made?
  3. Which step causes the most wrong turns?

Check it worked

Your plan has two or three questions at the top, and each one names an action you could see in a screen recording or click path. If a question contains "why", "prefer", "trust" or "would", you have moved it to a list for a moderated session or an interview.

Step 2 · Write tasks that need no explanation

Each task is a short scenario the participant reads on screen and acts on alone. With nobody to ask, any ambiguity becomes a failed task that tells you about the wording, not the design.

Write each task to these rules:

  • Give a reason and all the facts needed, in the order they will be needed: "You and two friends want to study together next Tuesday from 2pm to 4pm. Book a room for the three of you."
  • Do not use the labels on the screen. If the button says "Reserve", the task says "book". Otherwise you are testing whether people can match words.
  • Say what "done" looks like, so participants know when to stop: "When you have a confirmation, move to the next task."
  • One goal per task. "Book a room and then cancel it" is two tasks.
  • Make success checkable from the recording: a specific day and time, not "a time that suits you".

Order tasks the way a real person would meet them. For the library, booking comes before cancelling, and the cancel task tells participants which booking to cancel, so it does not depend on whether they succeeded at the first one.

Keep the whole study to three to five tasks and under 15 minutes. Attention drops sharply in the second half of a long unmoderated session, and you cannot see it happening.

Check it worked

Read each task aloud to someone who has not seen the prototype and ask them to tell you, without the screen, what they would be trying to do and how they would know they had finished. If their answer matches yours for every task, and none of your tasks contains a word printed on the prototype's buttons or headings, the tasks are ready.

Step 3 · Add questions around each task

Questions are how you get a little of the "why" back. Keep them few and specific.

After each task, ask:

  • A single ease rating, for example "How easy or difficult was that?" on a 1 to 7 scale, with both ends labelled.
  • One open question tied to that task: "Was there a moment where you were not sure what to do? If so, what happened?"

At the end, ask one or two broader questions, such as "What, if anything, was confusing?". If you want a standard overall score, add the System Usability Scale here, and nowhere else, so it reflects the whole flow.

Add one attention check that has a single right answer drawn from the study itself, for example "Which day were you asked to book a room for?" with four options. It is one of your strongest signals for spotting people who clicked through without reading.

Avoid questions that invite praise ("How much did you like the new booking page?") and questions people cannot answer about themselves ("How likely are you to use this every week?").

Check it worked

Your study outline in the tool or a document lists, for every task: the task text, the ease rating, one open question. The end of the outline has no more than two open questions, an optional SUS block and exactly one attention check with a known correct answer.

Step 4 · Set up the study and the success criteria

Build the study in your testing tool. Most tools ask for the same things in different places: the prototype or page link, the task and question blocks in order, screening questions, the device type (desktop, mobile or both), whether to record screen, audio or webcam, and how many completed sessions you want.

Three settings matter more than the rest:

  • Device. Test on the device your users mostly use. If you want both, treat them as two studies with ten or more sessions each, not one mixed group.
  • Screening. Use two or three screening questions to exclude people who are clearly outside your audience. For the library, that means people who never book shared spaces of any kind. Do not screen so tightly that only experts get through.
  • Recording. If the tool supports it, ask participants to think aloud and record audio. Some will not, but those who do give you the closest thing to a moderated session.

Separately from the tool, write a success rule for every task in your spreadsheet before any data arrives. For the library: task 1 succeeds only if the confirmation shows next Tuesday, 2pm to 4pm, for three people. Writing it now stops you from relaxing the rule later to make the numbers look better.

Check it worked

Your spreadsheet has a "Success rules" tab with one row per task, each with a pass condition you could check from a recording without judgement calls. The tool's study preview runs from the first instruction to the final thank-you screen with every task and question in the order you planned.

Step 5 · Pilot with two or three people

Launch the study to two or three people first. Use colleagues outside the project or a tiny batch from your panel, never the people who built the prototype.

Watch every pilot recording in full and look for:

  • Tasks where the participant did something sensible that you did not plan for, which usually means the task wording allowed it.
  • Prototype dead ends: hotspots that do nothing, screens with no way back, links that open the wrong frame.
  • Questions answered in a way that shows they were misread.
  • The time taken. If a pilot ran longer than 15 minutes, cut a task.

Fix what you find, then pilot once more if you changed a task. Do not count pilot sessions in your results.

Check it worked

You have a short pilot log listing each issue found and the change you made. The last pilot session ran start to finish with no dead ends, no misread tasks and a total time under 15 minutes.

Step 6 · Launch and screen out low-quality sessions

Launch to enough people that you will have at least ten good sessions per device after cleaning. With panels, expect to discard some sessions, so ask for 14 to 16 to end with ten or more.

As sessions arrive, review each one against fixed rules and record the result in your spreadsheet, one row per participant. Decide the rules before you look, and apply them the same way to everyone. A reasonable starting set:

  • Failed the attention check. Exclude.
  • Too fast. A session finished in under a third of the median pilot time, with tasks "completed" without visiting the screens needed. Exclude.
  • No genuine attempt. Blank or nonsense answers to every open question ("good", "ok", "asdf"). Exclude.
  • Technical failure. The prototype did not load, or the recording is missing. Exclude, and note it, because it may be your problem rather than theirs.
  • Struggled honestly. Slow, lost, gave up. Keep. This is the data you came for.

The last rule is the important one. Never exclude someone because they failed the tasks or disliked the design. That turns cleaning into editing the results.

Check it worked

Your spreadsheet has a row per session with columns for participant ID, device, duration, attention check, kept or excluded, and the rule used when excluded. At least ten rows per device say "kept", and every "excluded" row names one of your written rules.

Step 7 · Score and summarise the results

For each kept session, mark every task as pass or fail against the success rules you wrote earlier. Add the time on task and the ease rating. Then watch the recordings of failed tasks, and a few passed ones, and note where each person went wrong: the screen, what they clicked, what they said if there is audio.

Summarise per task:

  • Completion: for example "7 of 11 booked the correct room". With ten to twenty people, report counts, not percentages with decimals, and do not compare two small groups as if a difference of one or two people meant something.
  • The main wrong turns, each with how many people made them. For the library: "4 of 11 chose a room for two people because the size filter defaulted to 'any'."
  • The median ease rating, alongside the completion count.
  • Short quotes from open answers or audio that show what people thought was happening.

Check it worked

Your summary table has one row per task with a completion count out of the number of kept sessions, the median ease rating and at least one observed wrong turn with a count. Every wrong turn you list points to at least one participant ID whose recording shows it.

Step 8 · Write down what this test cannot tell you

End the write-up with a short section on limits. It protects the team from treating the results as stronger than they are, and it is where unmoderated testing most often misleads.

Cover these, in your own words for your study:

  • Reasons are guesses. You saw what people did. Unless they said why in audio or an open answer, any explanation is your interpretation. Label it as such.
  • Participants are not your users. Panel participants are experienced at taking tests. They read task text carefully, push on through problems a real user would abandon, and often rate ease higher than their behaviour suggests.
  • The task set the goal. Real users arrive with their own goal and their own information. A high completion rate on a scripted task does not show that people would find or choose the feature on their own.
  • The prototype is not the product. Missing content, fake data and limited paths shape behaviour. Note any wrong turn caused by the prototype rather than the design.
  • Small numbers. Ten to twenty sessions find common problems well. They cannot tell you whether 70% or 80% of all users will succeed.

Finish with what you would run next to answer the open questions, for example three moderated sessions on the size filter to learn what people expected it to do.

Check it worked

Your write-up ends with a "What this test cannot tell us" section of at least four specific points about this study, not general caveats, and one recommended follow-up method with the question it would answer.

Next steps

Need this done for you?

Padding does this work for growing companies, from the first diagnosis to a system that keeps working.

Digital Transformation

Get new tutorials

New tutorials, about twice a month.