The manufacturer claims 99.5% specificity based on scoring 369/371 on pre-COVID samples. That works for me.
As
@Daniel in SD has pointed out, it doesn't really work for a population with such low prevalence, when the uncertainty on the specificity is relatively high. They had 2 false positives out of 371 control samples! And then they went ahead and used that to test 3330 people and got 50 positive results (directly scaling, it seems fairly likely that ~20 of those were false positives). And it seems easily reasonable that 5-50 of those results could have been false positives, given the wide uncertainty on the false positive rate.
So how do we know that we didn't have 0 or 10
actual positives, rather than 50? That doesn't seem THAT improbable. This is the reason you have to quote confidence intervals (as I know you know).
Of course, there are also false negatives to account for (which may be significant) as well. There are a lot of uncertainties!
Really very surprising they didn't verify their positive samples for positivity.
And agreed, their corrections for selection criteria and scaling of small bins leave additional cause for concern. It's really not a random sample - there are a lot of people eager to determine their status. They do mention this in the paper, but it's not like it's headlined.
Yeah, the specificity confidence interval is wide enough to create a little doubt. We need one of these serology studies in a hot zone with 1%++ confirmed cases. Then we'll see how these 50-100x ratios hold up.
Yes. Agreed. But once that's done, they also have to make clear (would be nice to headline it in the initial publication) that New York City is a hot zone, and there will likely be much lower prevalence elsewhere. Would also be nice to follow up with ELISA testing on their positive samples!
Overall not surprised with the Santa Clara results. It seems perfectly reasonable that there is prevalence of ~1% (means an IFR of about 1%, after all the deaths have taken place), and seems like that's about what they measured in spite of being biased towards finding positive results in both their selection criteria and the performance of their test.
But something seems to have gotten lost in the messaging to claim 50-85x the measured case load - that seems way outside of what their results actually suggest.