* The key message from the World Health Organization about COVID-19 has been “test, test, test.”
* While much is being said about testing, less is being said about test reliability.
* There are two key attributes of a medical test – test sensitivity and test specificity. Test sensitivity is the ability of a test to correctly identify those with the disease. Test specificity is the ability of a test to correctly identify those without the disease.
* There are two tests related to COVID-19 that would be extremely useful right now: Test A) Knowing who has the disease (present). Test B) Knowing who has had the disease (past).
* Test A needs to be prepared to diagnose some people with COVID-19 who don’t have it – false positives – to err on the side of caution. Test B needs to be prepared to miss some people who have had COVID-19 – false negatives – to err on the side of caution.
* There is very little evidence on the reliability of Test A. No government in the world has yet rolled out a full programme for Test B.
* “Test, test, test” is the right strategy in principle, but, in practice, test limitations need to be accounted for and other options need to be explored.
Brief update on previous note
Things have moved at quite a pace on masks since my last note. On March 31st, the World Health Organization (WHO) stood by its recommendations that members of the public should not be wearing masks (Ref 1). On April 2nd, news stations around the world reported that the WHO was reviewing the evidence (Ref 2). In the early hours of April 4th, the South China Morning Post reported that the WHO had changed course and now recommended wearing masks in public (Ref 3).
The US Centers for Disease Control and Prevention (CDC) has just issued new advice to wear cloth masks (Ref 4). At the time of writing this, the Deputy Chief Medical Officer for England said that the advice (to not wear a mask) had not changed (Ref 5).
Introduction to this week’s note
Testing is the other hot topic in COVID-19 news. The key message from the WHO conference on March 16th, 2020 was “test, test, test” (Ref 6). This makes sense. We are making huge decisions at the moment with little information and the more information we can gather, the better. However, all medical tests have limitations and we need to be aware of these limitations in the context of COVID-19. This note, therefore, is about the limitations of medical tests generally, what we would like to know about COVID-19 tests and what we may not be able to know for sure.
Test sensitivity and specificity
There are two key attributes of a medical test – test sensitivity and test specificity. They apply to any medical test that aims to diagnose if a person has a condition or not (e.g. cancer screening).
1) Test sensitivity is the ability of a test to correctly identify those with the disease. (The true positive).
2) Test specificity is the ability of a test to correctly identify those without the disease. (The true negative).
The perfect test would be positive in everyone with the disease and negative in everyone without the disease. I’m not aware of any perfect medical test. This is where the terms “false positive” and “false negative” come in. False positive means that someone tests positive who doesn’t have the disease. False negative means that someone tests negative who does have the disease.
I love a good ‘2 by 2’ table and the following one is a useful summary of the four options that can arise from a medical test. The table below has been simplified from an excellent and much cited paper on test sensitivity and test specificity in ophthalmology (Ref 7).
Person has the disease
Person doesn’t have the disease
|Person tests positive
a) True positive
b) False positive
|Person tests negative
c) False negative
d) True negative
The test sensitivity (the ability to identify those with the disease) is a/(a+c) i.e. if C is zero, then a is 100%.
The test specificity (the ability to identify those without the disease) is d/(b+d) i.e. if b is zero, then d is 100%.
Test sensitivity and test specificity are inversely related, meaning that as the sensitivity increases, the specificity decreases and vice versa. To correctly identify as many people as possible who have the disease, it is necessary to accept that there will be false positives and thus to accept a lower test specificity.
There are two tests related to COVID-19 that would be extremely useful right now:
A) Knowing who has the disease (present).
B) Knowing who has had the disease (past).
There are two types of test to assess A and B (Ref 8):
A) Antigen (nucleic acid test).
Testing for current presence of the disease requires a swab to collect a sample from inside the nose, or the back of the throat, of an individual. When we see media pictures of people attending drive through testing clinics, this is the test being done to answer the question “does this person have the virus right now?” The answer to this question can be provided within hours.
B) Antibody (serological test).
Serological tests are blood tests that look for antibodies in the blood. These tests typically involve just a finger prick of blood, which is applied to a testing strip. These tests can be done at home. This is the test being done to answer the question “has this person had the virus?” The answer to this question can be provided within 15-20 minutes.
There are many limitations of test A. The test is only as good as the sample and so the swab needs to be of high quality. There is already some evidence to show that samples from sputum (saliva and mucus coughed up) produce the most accurate result (Ref 9). The sample also needs to have been handled and stored properly. The test can only identify people with active infection; individuals tested in the incubation period, before the virus really takes hold, can test negative and individuals within the recovery period of COVID-19 will likely test negative.
There are many limitations of test B. Antibody tests generally are problematic because a number of antibodies can react to the reagent cells tested and mask the antibody of interest. Specifically, it can take approximately 14 days for antibodies to COVID-19 to appear (Ref 10). Testing someone during this window will produce a negative result.
With COVID-19, for A, it is important to know who has the virus right now, so that they can be isolated/treated accordingly. It is so important not to miss any true positives, that false positives should be accepted as a necessary consequence. The sensitivity for this test needs to be high. This does mean that some people will be diagnosed with COVID-19 who don’t have it.
With COVID-19, for B, it is important to know who has not had the disease. It would be more harmful to think that someone is immune when they aren’t. For this test, therefore, it is so important not to allow false positives, that a number of false negatives need to be accepted. The specificity for this test needs to be high. This does mean that some people will be diagnosed as not having had COVID-19 when in fact they have had it.
There is limited evidence on the accuracy of either COVID-19 test at the moment, but the evidence that we do have raises concerns.
Regarding A), the tests to confirm who currently has the virus, I have come across only two academic papers about the test accuracy and neither is robust.
The first, published on March 5th, has been withdrawn already (Ref 11). No reason has been given for the withdrawal. The full paper has not been made available at any time. The abstract reported that the positive predictive value – the probability that people with a positive test actually have the virus – was only 20%. The false positives were reported as 80%. The numbers in this paper must be disregarded, as it has been withdrawn, but the paper indicates that false positives might be high.
A second paper has been made available online, while going through peer review in parallel (Ref 12). This is happening often with COVID-19 papers. Journals have taken the view that any information is better than no information, even if the paper doesn’t subsequently get through peer review. This second paper suggests that the test for the virus may give false negatives approximately 30% of the time. An article in the New York Times by Dr. Harlan Krumholz, a professor of medicine at Yale University, advises people to assume that they have the virus if they have the symptoms, even if they test negative (Ref 13). The false negative issue has been confirmed in other articles. There have been reports of some people testing negative several times even though they were infected with the virus (Ref 14).
Regarding B), this is the test that would be invaluable to undertake in large sections of the population. This, if accurate, would help governments to understand the degree of immunity that the population has developed and to be able to identify individuals who could help others more safely (as they would not be at risk of passing on the virus).
Test B is where we have the biggest test accuracy issues. The transcript of the UK government briefing on April 3rd, quoted the Health Secretary, Matt Hancock, as saying: “And, in fact, on the G7 call earlier, it’s clear that no G7 country has yet found a home antibody test, that works. But we continue to search for one” (Ref 15).
On April 3rd, the US FDA officially authorized its first serological antibody blood test for COVID-19 (Ref 16). The UK government is reported to be working with nine companies, in the hope that at least one reliable antibody test can be put into operation (Ref 17).
“Test, test, test” is the right strategy in principle, but, in practice, test limitations need to be accounted for. Governments and medical advisors are being attacked for not testing enough, while little is being said about the limitations of those tests.
We need to know who has the virus right now and thus we need to be prepared to diagnose some people with COVID-19 who don’t have it – to err on the side of caution. The very limited academic evidence that we have so far on this test is poor and inconsistent.
We need to know who has had the virus already and thus we need to be prepared to miss some people who have had COVID-19 – to err on the side of caution. The official UK government document on testing notes that “No government in the world has yet rolled out a full COVID-19 antibody testing programme” (Ref 18). The same document cautions that “Our experts are clear that an unreliable test is worse than no test.”
I’m not sure that I agree with the “unreliable test is worse than no test” caution. Some information must be better than no information when information is so valuable right now. Provided that we know the principles of tests – true and false positives and true and false negatives – and provided that we know the side of caution to err upon, we can make relatively informed decisions from relatively little information.
We may also need to apply some common sense in the absence of robust lab tests. A young A&E doctor in our wider family, working with patients with COVID-19, developed a persistent cough, high temperature and extraordinary fatigue. She wasn’t tested for COVID-19, but you could put big money on the fact that she had it. She’s back at work already and so we can now assume that she’s in the group of people who have had COVID-19. How can we gather all such information?
As for the numbers who have had it, I come back to that tweet on March 13th: “Can anyone explain how a virus (identified in Wuhan in Dec 2019) spread to the Canadian prime minister’s wife, one of Hollywood’s top actors & his wife, a Premier League football manager, and a British member of parliament – within 16 weeks – while bypassing the majority of us?” (Ref 19). The answer is, of course, it didn’t. Many of us have had the virus already.
The UK has just issued a call to arms for people with modelling skills (Ref 20). Maybe artificial intelligence models can work out how many people have had the virus already such that Prince Charles, Idris Elba and Pink have all had it. What can the concept of “six degrees of separation” tell us right now?
Ref 1: https://edition.cnn.com/2020/03/30/world/coronavirus-who-masks-recommendation-trnd/index.html
Ref 2: https://www.smh.com.au/national/world-health-organisation-reviews-face-mask-evidence-20200402-p54gjz.html
Ref 3: https://www.scmp.com/news/china/article/3078407/coronavirus-world-health-organisation-reverses-course-now-supports
Ref 4: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/cloth-face-cover.html
Ref 5: https://metro.co.uk/2020/04/03/government-still-wont-recommend-wearing-face-masks-public-12506422/
Ref 6: https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—16-march-2020
Ref 7: Parikh et al. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2636062/
Ref 8: https://www.abpi.org.uk/medicine-discovery/covid-19/briefing-coronavirus-covid-19-testing/
Ref 9: Yang et al. Evaluating the accuracy of different respiratory specimens in the laboratory diagnosis and monitoring the viral shedding of 2019-nCoV infections. Currently in peer review. April 2020. https://www.medrxiv.org/content/10.1101/2020.02.11.20021493v2.full.pdf
Ref 10: Okba et al. SARS-CoV-2 specific antibody responses in COVID-19 patients. March 2020. https://www.medrxiv.org/content/10.1101/2020.03.18.20038059v1.full.pdf
Ref 11: Zhuang et al. [WITHDRAWN: Potential False-Positive Rate Among the ‘Asymptomatic Infected Individuals’ in Close Contacts of COVID-19 Patients] March 2020. https://pubmed.ncbi.nlm.nih.gov/32133832/
Ref 12: Yang et al. Evaluating the accuracy of different respiratory specimens in the laboratory diagnosis and monitoring the viral shedding of 2019-nCoV infections. Currently in peer review. April 2020. https://www.medrxiv.org/content/10.1101/2020.02.11.20021493v2.full.pdf
Ref 13: https://www.nytimes.com/2020/04/01/well/live/coronavirus-symptoms-tests-false-negative.html
Ref 14: https://www.medicinenet.com/script/main/art.asp?articlekey=228250
Ref 15: (At 31.52) https://www.rev.com/blog/transcripts/united-kingdom-covid-19-briefing-transcript-april-3 (The G7 countries are Canada, France, Germany, Italy, Japan, UK and US).
Ref 16: https://www.fiercebiotech.com/medtech/fda-officially-authorizes-its-first-serological-antibody-blood-test-for-covid-19
Ref 17: https://www.bbc.co.uk/news/uk-52140376
Ref 18: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/878121/coronavirus-covid-19-testing-strategy.pdf
Ref 19: https://twitter.com/DavidVidecette/status/1238487118030258180
Ref 20: https://royalsociety.org/news/2020/03/Urgent-call-epidemic-modelling/