It seems that for approving vaccines there’s a gap of weeks between the drug company finishing their trial and giving the data to the FDA and the FDA actually making the decision to approve the vaccine. What does the FDA do during that time? What takes weeks?
[Question] What does the FDA actually do between getting the trial results and having their meeting?
I asked the question at the Explain to me like I’m 5 - Reddit and if the answer is correct they do more then just looking at data. According to it they do visits at trial sites and doctors and ask for original patient data to check whether data is authentic.
Question is also on Quora but Quora isn’t what it used to be.
Related: Zvi had some notes on the general FDA approval process in his last COVID-Situation-update-post, regarding the AstraZeneca vaccine: https://www.lesswrong.com/s/rencyawwfr4rfwt5C/p/PQACEuWpkSyRgHC4p#Waiting_on_AstraZeneca
I think this post is really valuable because it pushes people to be in information-seeking mode, rather than hortatory mode. And I’m sorry to see that I’m the only one who’s tried to answer the question, because I’m sure my answer is incomplete.
I’d like to encourage other people to really try to think and learn about why the FDA is acting this way. What is happening and why? What procedures and incentives are they operating with? Don’t jump straight to criticizing them, don’t jump to what you would do if you built the system from scratch. (And no, “fire everybody and replace them with AI” is never going to fly in the federal government.)
If this is ever going to change, we need to know what’s happening first.
I’d love to hear from someone at FDA on this. I do not work for FDA, but here’s my guess.
they need to study the data they’ve been given. Although FDA will have been in communication with the drug company and seen their data all along, they probably have a rule that they can only consider data Officially Submitted as part of an Official Application. In a complex organization, most likely lots of people get involved in a big decision, including managers, lawyers, and political appointees who don’t necessarily have a lot of value to add on safety/efficacy questions but who do have enough organizational clout to ensure they don’t get bypassed even in an emergency. I’d guess for sufficiently big decisions, conversations happen with Congressional and White House staffers too.
Legal requirements for notice. FDA might have to give the public notice that they are considering the approval. And the FDA has an advisory committee on vaccines, and they are required to give notice of those meetings so that the public has a chance to attend. There are probably emergency bypasses to these notice requirements, but no one is particularly incentivized to take the risk of departing from normal process.
This comment intended as description, not justification, of the existing practices.
What kind of study do they need to do besides the statistical tests that are already done. How do you fill weeks with studying the data?
I addressed this in #1 above. Even if they’ve already seen data, they’re starting from scratch as far as evaluating the Officially Submitted Data goes. Plus the number of different people and groups involved.
No, it’s not clear why you need weeks to a bunch to check whether someone did their statistics right. Analysing clinical trial data isn’t rocket science.
even if it were rocket science, you could always just hire more rocket scientists to get it done more quickly?
I think that’s explicitly not true, especially for things of the form “come to consensus on a controversial topic.”
What bothers me about this is that in principle, before the experiment was done, the FDA has thresholds for acceptance. They already have defined mathematically what level of risk they deem “safe” and what level of effectiveness is “effective enough”. In theory, all the actual data has in it is the possibility of
a. a math error or deliberate mistake or
b. falsification of results
(this is why many have argued that truly independent organizations should run clinical trials)
...it seems like you could build a machine learning tool similar to that used for credit card fraud transaction detection, and hunt for a and b in about a minute of IRL time...
I think we have a misalignment of incentives. The FDA staffs are a few hundred people and they are most likely taking time mainly to avoid risk to their personal reputations and careers. But this extra time is having the consequence of potentially killing tens of thousands of people.
If a lot of the work is making sure that the stats / methods check out, that’s a local validity issue that scales with more people, right?
Hmm, maybe I am responding with too much pessimism for unrelated domains (coding and rationalist-philosophy). But I do have a pretty strong prior that adding more people to a cognitive process doesn’t necessarily make it faster.
I don’t think that’s the obstacle. Lots of different people are looking at the application from different angles, and no one seems to have the sense of urgency we might think is warranted.
I agree. I’m saying that if, hypothetically, this were difficult to check and the FDA couldn’t have checked the intermediate data and if… etc, then you could still hire more people.
Instead of hiring more rocket scientists, I would think it would make sense to invest in software that automates the process.
Whatever else is going on… the FDA has never been graded on speed, and plenty of ink has been spilled in non-covid times about how many people die, and how many QALYs are lost, because of delays that make little difference to the eventual outcome, but look to the general public like Serious People Making Decisions Carefully. They permit themselves the time, and with it buy a mantel of authority in the public eye.
Also… looking at the VRBPAC online meeting calendar, it looks like they plan out their meetings several months in advance. I assume normally this is to ensure that members can align their schedules. Does anyone know how far in advance the seemingly-weeks-delayed-meetings were scheduled? Before or after the data from trials gets officially submitted? I’m wondering if they just have recurring meetings on the topic and wait to talk at the next one, or if their standard practice is to always schedule meetings at least a few weeks out for their own convenience, or what.
This past year we removed many but hardly all of the more useless regulatory barriers to testing and approval the vaccines. Is there any reason the FDA couldn’t have been auditing the data and the trial process all along (other than “they’re not set up to do that or aren’t allowed to”)? “We need weeks to confirm the data are authentic and check your math” just isn’t plausible during a pandemic when there’s no scientific reason that has to wait until the very end.
The FDA is effectively never graded at all. At the same time, politicians do something push for the FDA to just approve a drug. Big Pharma who hires FDA officials likely also rewards officials for approving their drugs faster.
This is essentially a conspiracy theory. It might be true that’s what happens but as rationalists I don’t think we should believe in conspiracy theories without understanding the underlying dynamics. I asked this question because I believe the underlying dymamics are important to judge whether or not it makes sense to believe in that conspiracy theory.
Fair enough, maybe it is a conspiracy theory, though to me it seems like the default adaption of what I would expect to happen. It’s a good question regardless.
Also RE: pharma companies wanting their drugs approved faster: yes, they do, too slow lowered their ROI, but also limits their pool if competitors to those already prepared to navigate the approval process. They may not want it to be too fast either.
I think it’s just that a few weeks is the going rate for avoiding blame, as Zvi outlined in his posts Asymmetric Justice and Motive Ambiguity.