FDA fast-tracks COVID-19 tests, but it’s unclear how accurate they are

Kendra Boroff believes she contracted the coronavirus on her 71st birthday, February 20, when her family went out for a celebratory dinner, perhaps from their waiter, who was coughing into his elbow. Four days later, she developed a fever and a raging sore throat.

“You feel like you’re suffocating,” recalled Boroff, a real estate agent in Maineville, Ohio. “You cough and breathe with the top fourth or maybe less of your chest, because everything else is in a vise.”

Over the course of the next three weeks, as Boroff started getting chills and nausea, a series of doctors would suggest that it could be the common cold, bronchitis, or pneumonia. She tested negative for the flu, and her chest X-rays showed signs of lung damage, including white patches called “ground-glass opacities” that are common in COVID-19 cases. By March 7, she was pretty sure it was COVID-19, but she couldn’t get a test until she arrived at the emergency room at the University of Cincinnati Health Center on March 19. She had a 103 degree fever and her oxygen levels were plummeting, so the doctors admitted her immediately.

Nearly a week later, as Boroff’s condition was stabilizing, the test results came back: negative.

Boroff was flummoxed, but her physician was clear that she had the virus, no matter what her test said.

“‘This is my diagnosis,’” she recalled him saying. “‘There is no other explanation.’”

Tests turning up negative even when all signs point to COVID-19 has been a common experience in American hospitals over the past month, public health experts have told ProPublica. It’s unclear what proportion of these negative results are inaccurate—known as “false negatives”—and whether that’s due to some external factor, like bad sample collection, or because of an issue inherent in the tests’ design.

Neither the major test manufacturers, the US Food and Drug Administration, nor the US Centers for Disease Control and Prevention would say how common false negatives are. While the FDA requires test-makers to report any known instances of false negatives as a condition of granting them provisional approval, known as emergency use authorizations, no such reports are visible in a database the agency maintains for that purpose.

Without much data on how COVID-19 tests are performing in the real world, concerns are mounting that a lack of accurate testing will make it more difficult for America to relax social distancing, as the ability to track and trace new infections will be critical for any strategy to reopen the country.

Those warnings have reached Capitol Hill, where Texas Democratic Rep. Lloyd Doggett had heard from a doctor in his district about the accuracy of the tests. On Thursday, he and Rep. Rosa DeLauro of Connecticut sent a letter to the FDA demanding more data about the prevalence of false negatives in both the diagnostic tests currently in widespread use, as well as inaccuracies in the coming wave of rapid blood tests that detect immunity once the infection has passed.

“I’m very concerned about it,” Doggett told ProPublica. Too many false results, he worries, could lead to a new surge of infections when people go back to work or are allowed to gather in bars, sports arenas, and restaurants. “They have to monitor this very closely to ensure that we’re not creating false expectations, and in the process ending up with an epidemic that is even worse than the one we have now.”

Lowering the bar in an emergency

In the early days of the pandemic, the FDA, which regulates diagnostic tests, was criticized for not moving quickly enough to make testing widely available. For much of February, the only available test was the CDC’s, which initially had flaws when it was sent out to public health labs. Only on Feb. 29 did the FDA announce a new policy that made it easier for private labs and academic medical centers to make tests available as well.

Since then, the ongoing need for even more testing capacity across the United States has pushed the agency to loosen its typical requirements for manufacturers to prove that their tests are accurate before allowing them onto the market.

Normally, to get FDA approval, diagnostic makers need to run trials to gather evidence on their tests’ performance, a process that can take months or even years. The agency is currently skipping a lot of those steps by issuing emergency use authorizations.

Manufacturers are now required to run their COVID-19 tests on a minimum of 30 positive samples and 30 negative samples. They must demonstrate to the agency that the test has at least a 95-percent sensitivity, meaning it must correctly identify at least 95 percent of the positive samples as having the coronavirus, and 100 percent specificity, meaning that it must accurately identify all the negative samples as not having the coronavirus.

But the manufacturers are demonstrating their diagnostics’ performance with what’s known as “contrived samples,” which are not taken from actual patients. A contrived sample is made by taking coronavirus RNA made in a lab and putting it into a medium that mimics nasal mucus.

“This is supposed to represent a swab specimen, but it’s not a positive sample from a real patient, and that does make a real difference,” said Benjamin Pinsky, medical director of the Clinical Virology Laboratory for Stanford Health Care.

It’s not clear if the concentrations of virus on the simulated samples are representative of the full range of material taken from patients’ bodies in the real world. Pinksy says that it’s reasonable for the FDA to allow the use of contrived samples, because it makes it much faster for a manufacturer to run validation studies, and the need for speed has been pressing.

“But then we need to have studies to compare these assays and see how they perform with real-world samples, and whether some are more or less sensitive and whether some are more or less specific,” Pinsky said. “We don’t know the answer to these questions at this point.”

To compensate for the lower standard up front, experts say the FDA should track data on accuracy to make sure the tests are performing as expected, but this is easier said than done.

“In diagnostic tests in particular, it’s very difficult to know if something is failing,” said Alberto Gutierrez, former director of the FDA’s Office of In Vitro Diagnostics and Radiological Health. “When are you getting more erroneous results than you should? It’s not always easy to figure out.”

Swiss manufacturer Roche, whose test was authorized by the FDA on March 12, told ProPublica it couldn’t give specific numbers about its test’s actual rate of false negatives and false positives, though it said studies have demonstrated its test could detect very low levels of the coronavirus.

“Clinical studies, which take months to run and would be part of a regular (nonemergency) test approval process, are needed to give us an exact percentage of false negatives and false positives,” Roche spokesman Mike Weist wrote in an email. “We will continue to work with the FDA on ongoing studies post-EUA that will allow us to potentially say more in the future.”

Abbott, which makes a rapid COVID-19 test, also said that “performance characteristics, including accuracy data, will continue to be collected in the field.”

Abbott and the testing firms LabCorp and Quest Diagnostics all told ProPublica that tests should be used by physicians along with other information to form a diagnosis.