Testing a COVID-19 classification

Over 800 healthcare providers, academics, and members of the public sorted COVID-19 top tasks into groups. We analyzed the groups and came up with the following hypothetical classification:

Symptoms, Diagnosis, Spread
Mental & Physical Wellbeing
WHO, Government Guidance, Education, Training
Research, Statistics
Vaccine, Immunity, Treatment
Avoiding Infection
End Date, New Outbreaks

What is important to understand here is that this is a hypothesis. The results from a sort rarely give definitive data. Lots of interpretation is required. Through experience we have found that we need to test over three rounds in order to design a robust classification.

How do we measure success? By giving people instructions to see how they would use the classification to solve top tasks such as:
• When is a vaccine likely to be available?
• Find the latest updates for COVID-19.
• Can you get infected by COVID-19 through the air?
• If someone has had COVID-19 are they safe from getting it again?
• Find what the WHO recommends in relation to testing.

We recommend having a minimum of 20 instructions and not more than 35. We need at least 20 instructions because otherwise the classification will not get a robust testing. It’s easy to get high success rates if we only test a few tasks. Using lots of instructions allows the classification to be tested from multiple angles.

Over 2,000 people completed the first round. (Normally, we need about 50 for reliable results.) We had an overall success rate of 60%. In other words, the classes we expected to be selected for the instructions were selected 60% of the time. Typically, we find success rates between 40% and 60% for round one, so our hypothesis was working well.

Our target is a minimum of 80% and ideally 90%. It is impossible to achieve a 100% success rate for a complex classification. There will always be around 10% of people or tasks that behave in unusual and unpredictable ways.

To get it over 80% we need to make some adjustments. The classification causing most issues was “Symptoms, Diagnosis, Spread.” This was somewhat expected. The sorting data showed strong connections between symptoms and diagnosis-type tasks. There were some—though weaker—connections between tasks connected with transmission and spread, and symptoms and diagnosis.

There were three tasks in the Transmission cluster:
• Transmission, spread, epidemiology: 62% success
• Virus survival / viability / persistence on surfaces, in air: 30% success
• Virus mutation, new strains: 19% success

It was decided that for round two we would break this class into two:
• Symptoms, Diagnosis
• Spread, Mutation, Survival

Testing-related tasks were strongly connected to the “Symptoms, Diagnosis, Spread” class and the “Vaccine, Immunity, Treatment” class. It was decided that there should be links to testing from both these classes.

The task “At risk, vulnerable” was performing poorly. This, again, was expected. It was the most orphaned task in the sort, meaning that it was the least connected with any other tasks. The reason it was not placed at the top level as its own class was because it was a tiny task, coming 29th. However, data we received from a COVID-19 top tasks survey in Norway indicates it is a top task there. On the national Irish health COVID-19 website and on the UK NHS website, it is at the top level of the classification. It was decided to make “At Risk, Vulnerable” a top-level classification.

Here is the classification that will go into round two and will be tested using the exact same list of task instructions:
WHO, Government Guidance, Education, Training
Mental, Physical Wellbeing
Vaccine, Immunity, Treatment
Research, Statistics
Virus Survival, Spread, Mutation
Avoiding Infection
Symptoms, Diagnosis
End Date, New Outbreaks
At Risk, Vulnerable

Podcast: World Wide Waste
Interviews with prominent thinkers outlining what can be done to make digital as sustainable as possible.
Listen to episodes