I feel like this is one of those questions that’s somehow too basic to ask. Or maybe too political. Like wondering this stuff implies I’m siding with something.
I’d really like to just set all the politics aside, simply name my ignorance, and hear some truthful answers. Because seeing this not even discussed is part of what’s giving me a sense of “Fuck it, no one actually knows anything, everyone is just making random shit up that fits their social identities.”
People keep talking confidently about incidence rates for different Covid variants, about their death rates and likelihood of hospitalization, how we have such clear data on vaccine efficacy and safety, etc.
But all the info streams I see look extremely dubious. I don’t just mean one could doubt them. I mean, I’ve witnessed powerful evidence of blatant bias, and aside from brief mentions of the existence of those biases no one seems to care.
I’ll give some examples below that inspire my confusion. I want to emphasize that I’m honestly asking here. Less Wrong is one of the few places (the only place?) where I feel like I can seek epistemic cleaning and clarity here.
So, example confusion sources:
I’ve personally known many people who have had serious medical problems that sure looked clearly like vaccine reactions. On par with “Well now I can’t get out of bed and can’t think anymore” or “Oh shit, heart attack” kinds of reactions. But all these people I’ve known who tried to report their reactions were told “No, your reaction can’t have been due to the vaccine, because the vaccine is safe and effective.” I’ve heard lots of similar reports. Because this is about rejecting data collection, I don’t see how anyone could possibly know how common/rare this is.
I’ve never known anyone who has been tested for a variant of any kind. I don’t know anything about how variant tests look different from a generic Covid test. So where are these numbers for variant spread coming from? Maybe hospitals do have special genetic tests and reliably do those? But then isn’t there going to be a pretty strong bias based on the fact that these are only for people who are getting hospitalized?
There was, best as I could tell, active (if unintentional) data destruction quite early on in the pandemic. Maybe it’s still going on? Hospitals had financial incentives to find reasons why people who died had died of Covid. Lots of bodies got cremated before autopsies could happen. So how strong was that bias? I never heard any curiosity about the implications of this for our sense of how deadly the virus actually was. (Plenty of hand-wringing and finger-pointing though.)
I get that we can deduce something about the virus’s spread based on (a) how many hospitalizations for Covid different areas are getting plus (b) some assumptions about the exponential-flavored spread of the infections. That lets us use math to peer into how many non-hospitalized infection cases there must be, and roughly how long the incubation-to-infectious timing should be.
But how in the bajeezus can anyone possibly extrapolate from there to how long it takes to become symptomatic? In need of hospitalization, sure, but how is anyone getting not-heavily-biased data on symptom strength below the hospitalization threshold?
I don’t care about symptomatic-ness per se. This is just one of a ton of examples about info passing through the info commons that people — including here! — seem bizarrely (to me) super confident in despite the devastation and weaponization of said info commons.
I could believe there’s some solid clear reasoning going on in the background here, grounded in hard-to-refute data, that makes all these assertions about Covid’s variants and death rates and vaccine safety and so on actually quite solid. And it’s just not obvious because the main social messaging is about authority and duty and “Sacrifices to the Gods” as Zvi puts it.
If so…
…could y’all help me see the sane thing?
For the US there is a surveillance program run by the CDC in which each week they get specimens from all US states to be sequenced. CDC’s Role in Tracking Variants
“Specimens will ideally represent a variety of demographic and clinical characteristics and geographic locations. Selection of a diverse set of specimens will help ensure a representative set of sequences are generated for national monitoring.” National SARS-CoV-2 Strain Surveillance (NS3)
I guess that for other countries there are similiar surveillance programs in place.
And we have VAERS, to which individuals can report directly. Plus, the surveillance system (including our crappy contact tracing systems run by the states) means we get sub-hospitalization data. Ideally contact tracing would also help arrest spread (not so much if they call you 3 days after you test positive 3 days after you first show symptoms...sheesh), but at the very least you’re getting a survey done.
I think just from becoming aware of the surveillance and adverse event reporting systems, Valentine’s base for a high degree of skepticism is pretty shaky. Being armed with an understanding that actually, the mechanisms by which the data could be generated DO exist should help a lot. I want to note that when people exclaim we should trust the experts, I believe it is about this level of ignorance they rightly have in mind (props for identifying a knowledge gap and honestly seeking to address it!) - lacking key fundamental knowledge necessary to even begin to assess the veracity of claims, rely on the people who do have it! As we learned from the pandemic fiasco though, our “experts” having the ability to generate and interpret that information does not mean that they always do it well.
I’ll also say that even without ongoing adverse event monitoring or observational effectiveness studies, the clinical trials were gigantic and provide strong evidence supporting efficacy and safety. [Unless the researchers were selecting what data to collect in which case seeing the raw “data” would be meaningless too. Sadly, data tampering or fabrication happen, but if that fact will undermine your reliance on any data generated by anyone but you, frankly there’s no convincing you with other people’s data.] I’m not sure how large any selection biases are, but I imagine they would have to be huge to impeach such extensive data vs. a handful of anecdotes that are themselves not free of selection either.
Thanks.
I’d have to check, but I think it was the VAERS system that these folk were told to report to, and who turned down the data based on the circular logic I described in the OP.
But this is based on my recollection of that acronym looking familiar in this context. Don’t take that too seriously. Just a little seriously.
From an article about how VAERS works:
It seems unlikely to me that VAERS would not take the reports.
I think it’s more likely that they went to some doctor who works with 3 minutes per patient and the doctor didn’t want to take the time to fill the report.
Alternatively, maybe they tried to enter the data as nonmedical people and got told “You have to go to a doctor to have them file the data”?
I’m curious how that coincides with no such person reporting their experiences at https://www.lesswrong.com/posts/XnRTP4dq3dkdwwtdS/which-rationalists-faced-significant-side-effects-from-covid. For me this thread was a test about whether I should look more into vaccine site-effects and the lack of reporting suggests that they aren’t very common.
Are all those people you are talking about outside of the rationality community? One thing that I would see plausible is that people, who are generally psychologically suggestible and who believe the vaccine is dangerous, have their bodies overreact when they are faced with the more normal vaccine side-effects. Such a dynamic might produce more vaccine side-effects in people you know from a spiritual context than appear in the rationalist community.
One step that could be taken to verify an existing pattern of a lot of vaccine side-effects would be to simply hire a SurveyMonkey audience and see what people report when asked through that channel.
We have people we pay to do contact tracing. There are likely cases where that comes with both asking for symptoms and doing testing.
From a conversation I had with a doctor, it seems that our medical system generally does a lot fewer autopsies than we did 20 years ago. There seems to be a general culture change here. The question of how much resources our medical system should invest into doing more autopsies however isn’t trivial. It would raise health care costs if we give hospitals more money if they do more autopsies.
Hospitals aren’t the only places that run tests besides at-home tests. The labs that do PCR testing retest some of the positive tests with variant-specific tests. Different countries have different policies about that.
Yep. As far as I know, but I’d be pretty surprised if any of them were here.
Would you be willing to point at more details about this? I recall seeing a lot about how we weren’t doing adequate contact tracing, but not much on how we have been.
Mmm. Good to know.
Although that basically means the problem with data collection I was describing is actually a step farther up the chain. That cremation isn’t where the data are getting destroyed. If they’re not even bothering to verify the causes of death via autopsies and there was (is?) financial incentive to conclude “Covid”… well, I believe in incentive landscapes.
Ah. And some people go through the different countries’ policies and numbers and do some data crunching to extrapolate something? Okay. Who are these data crunchers then? All this is an opaque screen from where I’m standing. I just see final numbers asserted in public.
(Thank you, by the way. Gratitude for the energy you’ve put into answering this.)
I think they are basically doing a COVID-19 test and then making the claim about the cause of death based on clinical history.
From my doctor friend, the main concern was that not doing the autopsies leads to not having good statistical data about which organs get damaged in patients dying with COVID and how that differs with new varients. That would be traditionally information that’s useful for doctors who want to prevent patients from dying but it’s not structured the way the modern EBM thinking about treatment goes.
There’s a reason Zvi was focusing on Denmark’s data. They have the best data.
At this point in time, I think most countries are doing samples. You don’t need to test everyone to have statistically significant data.
Contact tracing is often getting to people to late and it’s not done in an amount that you would need to track all the cases but those contact tracers are still around. (but I haven’t looked into detail into the system; it’s not something like the autopsy topics that I discussed with a doctor who has a good understanding of what’s happening)
Hooooo boy.
Here is how I have been evaluating data, curious to know if other people are making judgments based on similar inputs:
Primary source material (CDC data tracker) is better than secondary source interpretation (CNN COVID newsfeed).
Small-scale primary source material, such as state or county data, is better than large-scale aggregate primary source material.
Secondary source interpretation can be more, not less valuable when created by a single person (as opposed to a news site) (even if that individual does not have a medical background), as an individual is more likely to look for useful information that can help them decide whether or not to take a specific action and share useful detail that explains how they came to that conclusion.
Assume something is lost/miscalculated/false-priored with every aggregation.
Assume all primary source material and all interpretations thereof are compromised for some reason (biases, incentives, etc.). Ignore actual numbers. Watch for trends.
I’ve also been doing a fair amount of on-the-ground evaluation, e.g.:
Do I know people who currently have COVID?
Are people around me (strangers in grocery stores, etc.) visibly ill?
Do I know people who have had post-vaccine health issues?
How does my experience correlate with the experience the data says I should be having?
One of the points of OP to be that aggregations like the CDC data tracker are not themselves primary source material. Like, the chain goes “person provides sample” → “sample gets processed” → “result gets recorded locally” → “result gets aggregated nationally”, and each of those steps feels like it has some possibility for error or bias or whatever. That CNN is even further from ground seems useful to know, but doesn’t tell us how connected the CDC is.
Agreed (which is why I noted that county data could be more valuable than aggregated CDC data, and that nuance has the potential to be lost with every aggregation), and I spent a good 30 minutes after writing this comment asking myself if there is a better term than “primary source,” which I probably used incorrectly above.
That said, it’s fair to note that I didn’t actually answer the question asked, because I don’t know how to determine the reliability of any given number (or any given source providing any given number). How are other people doing this?
I’m afraid I only have time for a short, partial response today. Short version: Covid surveillance is hard, and there’s lots of noise in the data. But there are lots of smart people working hard on this, and in the aggregate we actually have a pretty good idea what’s going on.
I’ll address one of the questions you asked specifically:
So where are these numbers for variant spread coming from? Maybe hospitals do have special genetic tests and reliably do those? But then isn’t there going to be a pretty strong bias based on the fact that these are only for people who are getting hospitalized?
In Washington, much of the variant prevalence data comes from UW, which sequences a subset of the samples they receive. This is a bit complicated: some samples are fully sequenced, and some are tested for S-Gene Target Failure, which is a faster, easier test that is a fairly good (but not perfect) proxy for Omicron vs Delta. The UW sequencing is a good but not perfect sample of what’s actually happening in Washington. For details on this project, the person to follow is Pavitra Roychoudhury. Details vary, but there are multiple other institutions with largely similar programs.
More general answer: you’re asking good questions. They are all important, and they’re obvious to any smart person who thinks about the issue for a moment. Although I don’t have time to answer them all, I assure you that the smart people working on Covid have thought of every single one of your questions, and have good answers to every single one. Many of the answers are in Zvi’s excellent series of Omicron updates.
I live near the UW. As far as I can tell, the UW has done a great job of pandemic response. I got a COVID test from them early in the pandemic before there were alternative tests available.
Thank you. This is clear and points me in directions that let me explore more and see through the fog of war.
There is a correlation between several types of reported data and the real situation. Raw data is not very reliable if we don’t account for biases of all kinds. For example, there is an a known difference between reported deaths and excess mortality, which is often 2-3 times larger.
Anyway, I am disturbed by your words about inability to report adverse effects to vaccines. It should not be this way.