Here’s the classification we tested in round two:
WHO, Government Guidance, Education, Training
Mental, Physical Wellbeing
Vaccine, Immunity, Treatment
Research, Statistics
Virus Survival, Spread, Mutation
Avoiding Infection
Symptoms, Diagnosis
News
End Date, New Outbreaks
At Risk, Vulnerable
With this classification we got a success rate of 77%, which was 17% higher than for round one. Our target is to get to 80%.
The three transmission-related tasks had an average success rate of 37% in round one. To address this problem, a new class was created: “Virus Survival, Spread, Mutation”. In round two the success rate almost doubled to 68%. There were no major indications that the new class was behaving as a dirty magnet (drawing clicks for tasks it was not supposed to draw clicks for).
The other new class, “At Risk, Vulnerable” worked even better, with a success rate of 82%. Its dirty magnet impact was negligible.
Behavior was very consistent between rounds. For example, in round one, 80% selected “Vaccine, Immunity, Treatment” for the task, “Have any drugs been approved for use on COVID-19?” In round two, it was 81%. In round one, 79% selected “Research, Statistics” for the task, “Find a chart that shows the trend / curve of coronavirus cases over time in Brazil.” In round two, it was 81%.
If you test 30 or so task instructions, you are always going to have one or two that will perform badly and that there’s nothing much you can do about. “What lessons have been learned about using ventilators?” was an instruction to test the task “Treatment lessons learned, emerging best practice, failed treatments”. We expected people would select the “Vaccine, Immunity, Treatment” class. In round one, it had a success rate of 35%, with 31% selecting “Research, Statistics” and 19% selecting “WHO, Government Guidance, Education, Training”. An argument could be made that an answer could be found under research, but this is a slippery slope. You could argue that practically anything might be found under research. You could argue that WHO guidance might address ventilators. However, ventilators are a part of the treatment process and if lessons have been learned then you should definitely be able to find that sort of information under the “Vaccine, Immunity, Treatment” class.
Specific changes we made in round two included:
• The task “Likely course of illness, outcomes, prognosis” had the instruction, “What is a typical prognosis, likely course for COVID-19?”. “Symptoms, Diagnosis” was designated at the correct path. However, in round one and two, 17% of people selected “Vaccine, Immunity, Treatment”. Seeing that prognosis is part of the treatment cycle, it was decided to make this path also correct.
• The task “Transmission, spread, epidemiology” had the instruction, “Can you get infected by COVID-19 through the air?”. “Virus Survival, Spread, Mutation” was designated as the correct path. The class “Avoiding Infection” was selected 27% of the time in round one and 33% of the time in round two. It was decided that it was logical to also make it a correct path.
• The task “Virus survival / viability / persistence on surfaces, in air” had the instruction, “How long can COVID-19 last on cardboard?”. “Virus Survival, Spread, Mutation” was designated as the correct path. (In round one it was “Symptoms, Diagnosis, Spread”.) The class “Avoiding Infection” was selected 41% of the time in round one and 22% of the time in round two. It was decided that it was logical to also make it a correct path.
When we implemented these changes the success rate moved up to 80%, our target. We also got results from a Top Tasks COVID-19 survey carried out in Canada with almost 7,000 respondents. We had a 70% overlap with the top 20 tasks, which was very positive. However, the overwhelming top task in Canada was “Financial support”, which was not a top task at all in the WHO environment. Looking at other data from countries and wanting to create a universal classification for COVID-19, we decided to add Financial Support to the top-level classification.
Here is the final classification for COVID-19:
At Risk, Vulnerable
Avoiding Infection
End Date, New Outbreaks
Financial Support
Mental, Physical Wellbeing
News
Research, Statistics
Symptoms, Diagnosis
Vaccine, Immunity, Treatment
Virus Survival, Spread, Mutation
WHO / Government Guidance, Education, Training