Skip to main content

UX and research · Beginner · 13 min

Rate and prioritise usability problems by severity

By Padding · Updated 18 Nov 2026
A spreadsheet such as Google Sheets or Excel, plus the note grids and recordings from your usability test · November 2026

You will have a severity table for the problems from a usability test, rated on a three-factor scale that two people apply the same way, and a ranked fix list the team can plan a release from.

On this page

After a usability test, the team usually has a list of problems sorted by how many people hit them. That is a start, but it is not a plan. A problem four people grumbled about can matter less than one that made a single person buy the wrong ticket without noticing. And if severity is "what the researcher felt", the first disagreement in a planning meeting turns into an argument about opinions.

This tutorial gives you a small, written scale with three factors: how many participants hit the problem, how badly it affected the worst-hit person, and whether that person got back on track. A fixed rule turns the three factors into one of four levels. Because each factor is read from evidence in your notes, a colleague rating the same list should reach the same answers, and you check that they do.

The running example continues from Run a five-person usability test: Northline Buses, a fictional bus company with a mobile ticket app, tested with five people on three tasks (buy a single for tomorrow, show the ticket, get a refund). Swap in your own list.

Step 1 · Set up the severity sheet

Open a new spreadsheet with one row per problem. Copy the problem list in as it stands, and add empty columns for the rating. The columns are:

Text
ID | Problem | Task | Seen in | Frequency | Impact | Recovery | Evidence | Severity

Seen in holds the participant codes, such as P1 P2 P3 P5. Evidence holds a pointer to where you saw the worst case: a participant, a task and a timestamp, such as P2 T3 06:40. You fill Frequency, Impact, Recovery and Severity in the next steps.

Give every problem a short ID (NB-01, NB-02, ...) that never changes, even when the order of the list does. People will refer to problems by ID in planning, tickets and the next round of testing.

For Northline the sheet starts with five rows:

Text
NB-01  Refund option sits under Help, not on the ticket        T3  P1 P2 P3 P5
NB-02  "Single" and "Day saver" look identical until tapped    T1  P2 P4 P5
NB-03  Bought ticket is not on Home; people reopen Buy         T2  P1 P3 P4
NB-04  Date picker defaults to today with no warning           T1  P1 P4
NB-05  "Activate" step not explained before showing driver     T2  P2

Check it worked

Your sheet has one row per problem, each with a fixed ID, the task it came from and the participant codes, and four empty rating columns plus an Evidence column. No row has a rating yet.

Step 2 · Write the scale down before you rate anything

Write the scale on a second tab, called Scale, before you look at a single row. If you define the levels after rating, you will bend the definitions to fit the ratings you already gave.

The scale has three factors, each with three values. Each value is defined by something you can see in the notes, not by how bad it felt.

Text
FREQUENCY  (count different participants, not notes)
  Few       1 of 5
  Some      2 of 5
  Most      3, 4 or 5 of 5

IMPACT  (rate the worst-affected participant)
  Blocked   Did not finish the task, gave up, needed the stuck rule or a hint,
            or finished with the wrong result
  Slowed    Finished correctly, but after a wrong turn, a backtrack or
            a restart of the task
  Bothered  Finished correctly on the first route, but hesitated,
            reread or said something negative

RECOVERY  (the same participant)
  Alone     Noticed the problem and got back on track without help
  Helped    Noticed but could not get back on track without the stuck rule,
            a hint, or giving up
  Unaware   Did not notice anything was wrong: ended with the wrong result
            and believed it was right

Then write the rule that turns them into a level:

Text
1. Start from Impact:  Blocked = 3   Slowed = 2   Bothered = 1
2. Add 1 if Recovery is Unaware
3. Add 1 if Frequency is Most
4. Cap at 4

4  Critical   Fix before the next release
3  Serious    Fix in the next planned cycle
2  Moderate   Plan it; fix when the area is next touched
1  Minor      Group with other polish work

Two consistency checks come with the scale. If Recovery is Helped or Unaware, Impact must be Blocked, because the person did not finish correctly on their own. If Impact is Slowed or Bothered, Recovery must be Alone. A row that breaks either check has been misread.

The rule makes a silent failure worse than a visible one. Someone who gives up knows to call the helpline; someone who buys a ticket for the wrong day finds out on the bus.

Check it worked

The Scale tab holds the three factors with three defined values each, the four-line rule, the four levels with what each means for planning, and the two consistency checks. Every value definition refers to something a note-taker would have written down, such as an outcome code or a wrong turn.

Step 3 · Rate frequency from the participant codes

Frequency is the quickest factor and the least open to judgement. Count the distinct participant codes in Seen in and write Few, Some or Most.

Count people, not notes. If P3 hit the refund problem twice, that is still one participant. If a problem was seen in two different tasks, keep the count of different people across both.

If your test had more or fewer than five participants, rescale the bands and write the new ones on the Scale tab. With eight people, for example: Few is 1 or 2, Some is 3 or 4, Most is 5 or more.

For Northline:

Text
NB-01  4 of 5  Most
NB-02  3 of 5  Most
NB-03  3 of 5  Most
NB-04  2 of 5  Some
NB-05  1 of 5  Few

Check it worked

Every row has a Frequency value, and for each one the number of codes in Seen in matches the band on your Scale tab. No row has a frequency higher than the number of people you tested.

Step 4 · Rate impact and recovery from the worst case

For each problem, find the participant it hurt most, and rate Impact and Recovery for that one person. Open their note grid or recording at the moment the problem happened, and write the reference in Evidence.

Rating the worst case, not the average, matters because frequency is already counted separately. If you averaged impact as well, a problem that seriously hurt one person in five would be diluted twice.

If your notes use outcome codes like the five-person test, most ratings follow straight from them:

  • Fail (finished the wrong thing and believed it was right): Blocked, Unaware.
  • Gave up (stopped, or stopped by the stuck rule): Blocked, Helped.
  • Success with difficulty: Slowed, Alone, or Bothered, Alone if there was no wrong turn.
  • Success, but the person paused or complained at this point: Bothered, Alone.

For Northline:

Text
NB-01  P2 T3 06:40  Gave up after 3 min on Help pages           Blocked  Helped
NB-02  P4 T1 03:15  Bought a Day saver, said "done, single"     Blocked  Unaware
NB-03  P3 T2 01:10  Reopened Buy, then found Tickets tab        Slowed   Alone
NB-04  P1 T1 02:05  Bought for today, did not check the date    Blocked  Unaware
NB-05  P2 T2 00:50  Read the Activate text twice, then tapped   Bothered Alone

Do not rate from memory. If nobody can point to a moment in the notes or recording, leave the row unrated and mark it "no evidence". It goes back on the ideas list.

Check it worked

Every rated row has an Evidence reference with a participant, task and time, and you can open that moment in the grid or recording. Every row passes both consistency checks on the Scale tab: no Helped or Unaware without Blocked, and no Slowed or Bothered without Alone.

Step 5 · Apply the rule to get the severity

Now fill the Severity column with the rule, one step at a time. Write the number and the level name, such as 4 Critical, so nobody has to look up what a number means.

Working it through for Northline:

Text
ID     Impact   Recovery  Frequency  Start  +Unaware  +Most  Severity
NB-01  Blocked  Helped    Most       3      0         1      4 Critical
NB-02  Blocked  Unaware   Most       3      1         1      4 Critical (capped)
NB-03  Slowed   Alone     Most       2      0         1      3 Serious
NB-04  Blocked  Unaware   Some       3      1         0      4 Critical
NB-05  Bothered Alone     Few        1      0         0      1 Minor

Look at NB-04. Only two people hit it, so it sat fourth on the list sorted by count. It is Critical because both people would have paid for a ticket they could not use. That is the gap the scale exists to close.

If you want the spreadsheet to calculate it, add helper columns that score each factor and a Severity formula that adds them and caps at 4. Keep the factor columns visible either way: the team argues with factors, not with a single number.

Check it worked

Every row with evidence has a Severity written as a number and a level name. For any row, you can show how the number came from its three factors in three steps. At least one row's severity differs from where it sat in the original list sorted by count, or you can say why none does.

Step 6 · Have a colleague rate the list on their own

A scale is only repeatable if someone else gets the same answers with it. Make a copy of the sheet with the Frequency, Impact, Recovery and Severity columns emptied, keeping the Scale tab, the Evidence references and the notes. Give it to a colleague who did not rate it, such as the note-taker or a developer who watched a session.

Ask them to rate every row from the evidence alone, without talking to you. Then put both sets side by side and count, for each factor, how many rows match exactly.

Where you disagree, do not average and do not let the senior person win. Find which factor differs and ask what each of you read in the notes. Then fix the cause:

  • If you read the same moment differently, the definition is unclear. Reword it on the Scale tab, for example by saying whether a 20-second pause counts as Bothered.
  • If you looked at different moments, one of you picked the wrong worst case. Agree which participant it was and update Evidence.
  • If one of you rated from something not in the notes, drop it.

Re-rate only the rows you disagreed on, using the reworded scale. Keep the change log on the Scale tab, with the date, so the next round of testing uses the same wording.

Check it worked

You have two independent sets of ratings for the same rows and a count of matches per factor. Every remaining difference has been settled and the reason written down, and the Scale tab shows any wording you changed, with the date. Both of you would now give every row the same Severity.

Step 7 · Turn the table into a ranked fix list

Severity says how much each problem hurts. Planning also needs to know what it costs to fix. Add two columns: Effort, from a developer, as S (under a day), M (a few days) or L (longer), and Retest, naming the task in the next test that will show whether the fix worked.

Sort the sheet by Severity (highest first), then Frequency (Most first), then Effort (S first). Within one severity level, cheap fixes go before expensive ones; across levels, severity wins. A Critical problem with L effort still sits above a Serious one with S effort, because its level says when it must be fixed.

For Northline:

Text
Rank  ID     Severity     Frequency  Effort  Retest  Fix by
1     NB-02  4 Critical   Most       S       T1      Before next release
2     NB-01  4 Critical   Most       M       T3      Before next release
3     NB-04  4 Critical   Some       S       T1      Before next release
4     NB-03  3 Serious    Most       M       T2      Next cycle
5     NB-05  1 Minor      Few        S       T2      Polish batch

Above the table, add three lines: the date and the test it came from, the number of participants and the user group, and one sentence on the scale ("Severity uses the three-factor scale on the Scale tab; rated independently by two people."). Keep the factor columns and Evidence in the shared sheet so anyone who doubts a rating can open the moment it came from.

Bring the ranked list to planning as the starting point, not the final answer. The product owner can move a row, but if they move a Critical problem below the next release, ask them to write the reason in the sheet.

Check it worked

Your fix list is sorted by severity, then frequency, then effort, and every row has an Effort, a Retest task and a Fix by value that matches its level on the Scale tab. Every Critical row is marked for the next release, or has a written reason next to it saying why not.

Next steps

Need this done for you?

Padding does this work for growing companies, from the first diagnosis to a system that keeps working.

Digital Transformation

Get new tutorials

New tutorials, about twice a month.