I ran through my checklist. Looks low-risk to me. Basically a very deadly disease that nevertheless shows nil or inefficient human-to-human transmission and doesn’t show signs of regional or global growth. It depends heavily on specific South American rat species for transmission to humans. The original exposure was on or around April 1. There are no new concerning mutations (see below), and so far the flight attendant who had been feared to have picked it up on a flight has tested negative, though there’s an up to 6 week incubation period and she was exposed on April 25th, about 2 weeks ago. There were also 3 other plane passengers who had contact with an infected person on a plane and they’ve tested negative so far as well.
The consensus public health position is that “person-to-person transmission is possible through close personal contact, such as between couples.” If it turns out that this is wrong and that it’s much more human-transmissible than previously believed, then the complete failure to quarantine passengers after the discovery of hantavirus on the Hondius cruise ship and first passenger death would completely alter my projection. But so far, there do not seem to be signs of that.
Getting on a evolutionary biology level, if Andes virus were to become efficiently human-transmissible, the last place I’d expect to see it enter the human population is via tourists picking it up while bird watching in a garbage dump. That’s a setting where the human-rodent interface is minimal. In that setting, there’s little selection pressure on the virus to jump to humans and transmit efficiently. In a hypothetical Andes virus pandemic, I’d strongly have expected patient 0 to be a Chilean or Argentinian villager living in conditions that sadly expose them to routine contact with rats. I’d expect the outbreak to spread first in a local village, not on a luxury cruise ship.
Instead, it looks to me like a couple of tourists made the risky and unusual decision to put themselves in a uniquely rodent-infested location without protection, where they likely had the misfortune to inhale a large quantity of virus-infested aerosolized rat droppings. Then they got into the close quarters of a cruise ship, which completely failed to execute the measures it should have taken to address their illness, including a failure to quarantine. This allowed limited spread to other close-quarters passengers, exactly as we have observed this virus to be capable of in the past. This, combined with the lack of new mutations, suggests we’re dealing with exposure to a known-quantity, transmission-inefficient virus in a demographic that doesn’t usually get exposed.
The concern is due to the fact that those passengers had travelled internationally by the time quarantining and contact tracing was initiated, combined with the deadliness of the virus. But it takes a combination of spread in international populations and transmissibility and deadliness and lack of treatment to make a pandemic, and currently we only appear to have deadliness and lack of treatment. The only reason it even gets 4 points is probably because it’s so rare and such a disease of poverty that big pharma’s never invested in a vaccine or treatment. The main scary thing, like ebola, is the very high CFR. But that is not the grounds for a global pandemic. It’s just a tragedy for a few hundred people a year.
Is there an efficient transmission route, such as respiratory droplets, airborne transmission or via the bites of common jumping or flying insects?
“Hantaviruses are transmitted mainly through aerosols and droplets that contain rodent excretions, as well as through contaminated food, bites, and scratches.”—Wikipedia.
Incubation time is up to 6 weeks.
Possibly. There are cases of human-to-human transmission, but this is contested. The Argentine government thinks the original two cruise ship cases contracted it while bird watching at a garbage dump, but several other passengers on the cruise ship were sickened.
Update 5/8/2026:flight attendant tests negative, but result inconclusive due to 6-week incubation period. No new concerning mutations. source
“On Thursday evening, Tedros Ghebreyesus, the director-general of the World Health Organization, said in a message that the flight attendant, who has been reported to have symptoms, had tested negative on two tests.… Mr. Ghebreyesus did caution, however, that the virus can have a long incubation time.”
For context, a Hondius cruise linear passenger, very sick with what turned out to be hantavirus, got on a plane. The passenger was handled on that plane by a flight attendant. The sick passenger was removed from the flight before takeoff due to her illness and later died. The flight attendant has been hospitalized and tested for hantavirus. I haven’t found a clear report of whether the flight attendant was sick or not. The flight attendant’s test results have been negative so far, but the incubation time is up to 6 weeks.
“Genetic sequencing of samples in South Africa suggests that the virus is nearly identical to the version seen in Argentina and has not mutated in ways that would make it a greater threat, according to Tulio de Oliveira, director of the Centre for Epidemic Response and Innovation at Stellenbosch University in South Africa. Dr. De Oliveira was not involved in the work but said the results had been presented to an expert group of which he is a member.”
Graded as a “no” as even if human-to-human transmission is possible, it currently appears to be inefficient and still heavily depend on rats.
Does it seem to spread rapidly within affected communities, going from a few cases to a major local emergency within a month? If R0 has been credibly estimated, is the mean of the range higher than 1?
No. This appears to be a sporatic, locally contained, chronic issue. “More than a hundred cases of HPS caused by Andes virus occur each year, most of them in Argentina and Chile.”—Wikipedia
Has it achieved community spread in non-endemic countries on at least 3 continents, and in a set of countries comprising both 15% of world population and 15% of world GDP?
No.
Is screening for the causative agent difficult due to test unavailability/unreliability/slowness, vector-based transmission, or transmissibility that is highest in early/asymptomatic stages? If the causative agent is unknown, mark “unknown.”
No. “ANDV infection is diagnosed based on observation of symptoms and testing for hantavirus nucleic acid, proteins, or hantavirus-specific antibodies.”—Wikipedia
Danger: case fatality rates, overwhelm, economic impacts, treatment
If a credible case fatality rate has been estimated, is it 1% or higher? Alternatively, is the number of deaths divided by the number of confirmed cases being reported at around 5% or higher in at least 3 countries with reliable data?
Yes. The case fatality rate from infection is high, at about 40%.
Is there a concern about hospital overwhelm or medical supply shortages in industrialized nations?
No.
Does the disease heavily affect career-age people (age 25-65), or frequently leave survivors with lasting disability?
Yes (tentative)
Is there no clearly effective treatment?
Yes, there is no clearly effective treatment. “Treatment is supportive in nature and includes supplementing oxygen during the cardiopulmonary phase.”
Spread limitations: demographics, geography
If some non-age-related demographics are heavily affected and others are not, do the heavily affected demographics amount to 15% or more of the population? If almost the whole population is about equally affected, mark “yes.”
Yes (tentative)
Is the disease potentially transmissible across most of the world population (i.e. does not work via a vector that has a geographically limited range)? If the causative agent is unknown, mark “unknown.”
No. Reports of human-to-human transmission are rare and contested. It appears to heavily depend on the particular rat species that transmit it.
Social effects: communications, shutdown, research, deaths
Has the disease made front page news on at least 3 different days in the New York Times, and also received the WHO designation “public health emergency of international concern” or the equivalent?
No.
Has there been a quarantine of a city with over 1 million inhabitants? In a country comprising at least 5% of world population or GDP, has there been a cancellation of major public events, or travel restrictions on passengers arriving from or via this country?
No.
Has the pharmaceutical industry begun a widespread research effort to produce a novel treatment or novel vaccine, and/or has industry begun a major emergency effort to build physical infrastructure or equipment (hospitals, ventilators, etc)?
The Andes-strain hantavirus outbreak aboard the MV Hondius cruise ship is tragic. But I estimate a <1% chance this becomes a pandemic. It is on track to self-extinguish within weeks.
@mabramov See! Right here! It’s happening! The mundane utility! Do you know how annoying arriving at a calibrated probability would have been before prediction markets took off?
Hard disagree. 8% is wildly overinflated and it’s at 4% now. If you don’t find yourself substantially revising your plans based on a 50% drop in pandemic probability, and if you didn’t get seriously concerned about an 8% chance of a 40% CFR pandemic, then even you aren’t taking them seriously. The main utility of this prediction market is driving internet chatter on how useful prediction markets are.
I think the prediction market can be useful even if you need to apply corrections to the headline number. It makes a huge difference to me whether Manifold says 8% or 40%, even though I won’t necessarily fully trust either number until it’s had a while longer to shake out (and even then it depends on the market dynamics, liquidity, etc.)
We already had the top epidemiologist at the WHO saying it was not going to be a pandemic. Wha is the prediction market adding here other than a wild overestimate by a few extremely online anonymous non-experts not even gambling with real money? Why can’t you just map the WHO’s statement to epsilon if you need a number to satisfy your anxiety and move on with your day?
The idea that manifold markets of all things is becoming people’s go-to source rather than the WHO‘s top epidemiologist is a damning indictment of prediction markets.
“It still appears to be rare that an asymptomatic individual actually transmits [the virus] onward,” Maria Van Kerkhove, the WHO’s technical lead for the COVID-19 response, said. Public health officials should concentrate on finding and isolating people who do have symptoms in order to stop the pandemic, she said.
Van Kerkhove walked back her comment in a news briefing the following day, saying that she had been referring to a small subset of studies that follow people who never show COVID-19 symptoms and their contacts. “We do know that some people who are asymptomatic can transmit the virus on,” she said in the June 9 briefing. “What we need to better understand is how many people in the population don’t have symptoms. And, separately, how many of those individuals go on to transmit [the virus] to others.”
The WHO’s statement that asymptomatic transmission is rare “is not backed up by any data,” says Anthony Fauci, director of the U.S. National Institute of Allergy and Infectious Diseases in Bethesda, Md. “We know that there is asymptomatic transmission…. What we do not know is the extent to which that occurs. So when we hear statements that this is very rare, we do not know that as a fact.”
So in context, we had immediate, credible, clear pushback from multiple equally high-status figures in major media within 24 hours. That’s completely absent here as far as I can tell. And the Maria Van Kerkhove who made those statements at the start of the COVID-19 pandemic had a different level of experience with pandemic management from the Van Kerkhove who’s making those statements today.
Your historic interpretation appears to be that because she made a misleading, probably just plain mistaken statement 6 years ago at the start of the COVID-19 pandemic, which was immediatelly called out, that she’s permanently untrustworthy and now we need to prioritize metaculus. That seems like a deeply foolish epistemic stance to me. An alternative way to interpret it is she screwed up, was called out, and has probably learned from that mistake, especially given that we’re dealing with a relatively known quantity in the Andes virus.
I 100% am sticking to my guns, metaculus is a wildly misleading source on this question and you all should be listening to the WHO.
I am confused why you are interpreting Jim to be making such a narrow claim? We have many confirmed instances of the WHO and the CDC and similar institutions acting extremely incompetently, and often deceptively during COVID.
Separately, sure, if you are happy to bet at 20:1 odds, my $1k against your $20k, same resolution criteria as the Polymarket. It seems like you think the probability is much lower.
I am confused why you are interpreting Jim to be making such a narrow claim?
I’m responding to the literal content of Jim’s comment, specifically and in detail. What more do you want from me? To read his mind? I supply multiple links with historic context deep diving into Jim’s casual one liner, and somehow I’m the problematic one in this conversation?
Separately, sure, if you are happy to bet at 20:1 odds, my $1k against your $20k, same resolution criteria as the Polymarket. It seems like you think the probability is much lower.
I do think the probability is much lower. But I frankly just do not want to bet with you or others here. It legitimizes more of what I find disturbing, which is the use of betting and prediction markets as a prime source of epistemic authority. It also signs me up for spending time negotiating bet logistics and resolution criteria, both now and in 7 months. That seems unappealing. What I find particularly irritating is the notion that that once a bet is proposed, it obviates a facts-and-logic based argument and undermines the credibility of the person turning down the bet. And that is what turns my “no” into a “HELL NO.”
What I would prefer is for you and others to consider the fact that a wide array of epidemiologists is currently in the news, saying, all with one voice, “this is not going to become a pandemic.” You can look into the personal history of the people making these claims. You can think about the difference between one expert making a claim that’s immediately contradicted by her peers vs. all the relevant experts in the media making the same statement, across a variety of institutions. You can read their scientific evidence and rationale. You can read what we know about Andes virus.
All those actions would teach you more about this virus and enable you to make a facts and logic based argument on the subject and derive your own number, if you need one in order to help make decisions.
If you feel like it, you can donate to a charity of your choice if it turns out that there’s no pandemic. If there’s a pandemic and I’m wrong, I’ll freely and openly confess to having been wildly overconfident.
Unfortunately the WHO’s top epidemiologist was also the WHO’s lead during the start of the COVID-19 pandemic, where she proved untrustworthy.
And then you said:
The specific statement you’re referring to:
And I just don’t know why you think Jim is referring to this specific statement. Like, maybe it’s obvious from context, but it isn’t obvious to me.
I feel kind of confused on your other points. I agree that I could spend hours trying to weigh expert consensus and read about the details myself. I don’t want to. The prediction markets are telling me how likely it is. The prediction markets are generally well-calibrated. I don’t need to research more. Everything is good. You said that because the WHO top epidemiologist said it’s not going to be a pandemic I should stop worrying. That seems very dumb to me, and I don’t really think you believe it (clearly you yourself wouldn’t stop worrying based on it, you would at least check what other experts say and whether they disagree).
That’s the resolution criterion for the Manifold Market (well, technically for the Polymarket market that the Manifold one is to resolve identically to).
Oh, I think whether the WHO declares that a pandemic is occurred is probably trustworthy. I don’t know if I trust their pandemic predictions. How good is their track record?
I think that’s an unfair deployment of xkcd mockery. 8% of a pandemic this year is not a tiny chance, which means a 50% drop is actually a big deal. The issue was interpreting the prediction market as an accurate percentage when it should just be an indication of approximate risk.
I mean, no, of course I am not changing my plans based on a 50% drop in pandemic probability. There are tons of pandemic probabilities that change from 0.1% to 0.05%, every week or so, and I am not changing my actions based on that.
In this case, knowing the probability seems to be somewhere in the 1%-10% range is already extremely helpful! I don’t really need to know much more. And I have proxies I can use to evaluate the robustness of the market (like volume), so I am not miscalibrated about the noise.
Yep, not saying that the probability I was referencing here is a prediction market probability (not sure what would have made someone think that, but happy to clarify).
Hard disagree. 8% is wildly overinflated… The main utility of this prediction market is driving internet chatter on how useful prediction markets are.
Polymarket has it at 10%. I don’t know where you’re located, but if you’re able to buy some NO, and the WHO doesn’t characterize Hantavirus as a pandemic this year, then you’ll make ~11% returns.
I am not sure what you mean, do you currently think the probability is outside the 1%-10% range? Imperfect accuracy is fine, especially when it’s easy to adjust for.
Maybe I’m just super new to the world of prediction markets in general, but I don’t feel like I’m getting any immediate utility from that 8% figure. If anything the rhetoric in this three comment thread makes me agree with @mabramov even more now.
It sounds to me like you’re expecting too much from all information. Consider the information that physical matter is made of atoms, which are about a nanometer across, and have positively charged protons and neutrons in the center, and then much smaller electrons whizzing around in orbits. This has been revolutionary for society, leading to so much ability to do engineering and chemistry and understanding cosmology and so forth.
A typical teenager learning this information for the first time might ask “what utility will this give me personally?” and find it has little direct application. Yet I would not advise them that this information is not worth knowing.
That’s the first point, that information doesn’t need to be directly connected to an outcome to be worth knowing.
The second point is that fine gradations in information are valuable. I can imagine someone similarly saying “Why should it be valuable to anyone to know the difference between Apple Stock being $285 versus $275? Surely we should just care whether it’s doing well or not? Why don’t we just replace it with the words “Great” “Good” “Bad” “Worse”? Yet often small signs tell us something. A few percentage points of dip can imply that a new product release went poorly. A change in CEO leading to stock price raising a few points can indicate very good things about this new CEO.
In this case, I find information like “8% chance of a pandemic” valuable in lots of ways.
It helps me to know how often we get 8% chance pandemics. I wish I could look back over the decades and see how often pandemics got to being this likely. This generally helps me understand our civilizational resilience and how much of an issue pandemics are.[1]
It helps me to know how much to keep tracking it. If the prediction market gave it <1% then I would be like “Great, this is an update that I can stop thinking about it and stop tracking it.” At this range I expect we’re fine but I will still be on the lookout for more info.
I also like knowing the specific probabilities so I can learn which events have an important effect. For instance, it’s currently around 5%. There may be many news stories about this pandemic, and seeing which ones cause movement vs which ones don’t, will help me build a model of what information actually matters.
Added: The third point is that public legibility is a massive value-add, and could well be most of the value. Given that it’s public, it makes me more confident that, if it were to get higher, people would notice and warn me. Much of the news landscape is just people arguing whether something is an emergency (which our memetics are perversely incentivized to say is true all of the time), so whether lots of people are acting alarmed just isn’t something most people can be very sensitive to changes in. The change from 5% to 50% is serious for me and yet I don’t know how to tell that difference from the tone of people on twitter or in many media outlets, especially when they are not themselves precise.
To go into more detail on this, I had an LLM write the following so that the math checked out: you can imagine a world where every year carries a 5% pandemic risk, and another world where every 15 years carries a 75% pandemic risk. Over 60 years, both imply 3 expected pandemics, but they suggest very different prevention strategies: steady-state risk reduction in the first case, versus identifying and defusing rare high-risk transition periods in the second.
I mean, so much. First of all, the number is enormously more helpful than reading a bunch of articles that give me lots of detail that then force me to come up with my own model of the situation to come up with a probability.
And then the number itself of course drives all kinds of actions! Another COVID-level pandemic would be a huge deal that would change my actions drastically in hundreds of ways (this market is not about it being another COVID-level pandemic, but is a lower threshold).
Why don’t you think that’s useful? It seems like at least some smart people thought about it carefully, which is a great start. Are you referencing some other thread? If so, please share?
I’m referring to this post from last month (and I assume that post is why habyrka is tagging mabramov).
Sure, maybe there’s some usefulness in that it got smart people thinking about the question. And it gave us a figure of 8%.
But I don’t really understand what a person is to do with that number. What utility follows from that? And why is it worth, as @mabramov emphasises, nine-figure EA funding?
My reaction to an 8% probability of a pandemic is currently “doing nothing,” so in some sense I agree with you. If it was >50%, I would probably read up more on hantavirus and think about preparations I could make now (maybe at that point, I would have picked up on ambient freaking-out even without the prediction market, but I do like having a precise number). Maybe I “should” in fact be preparing for hantavirus even at an 8% probability, but I only have so much willpower and time in the day.
Notably, I can’t immediately think of any important decisions in my life that I made differently due to prediction markets. But someone whose job depends on making quick decisions based on global events, e.g. some government role in biosecurity, would probably find calibrated estimates on this kind of thing quite useful.
Cruise ships, unfortunately, are prone to exacerbating any sort of disease outbreak due to their nature. For instance, back in 2020, aboard the Diamond Princess, Covid spread with an R0 5x higher than usual.
If anything, it’s surprising that more people didn’t get infected by this, which shows just how poor hantavirus is at human-to-human spread.
I think it is important to note that the opportunity created by shifting x-risk sentiment, or the current “crunch time”, is more of an opportunity to shift public sentiment around AI x-risk rather than politician sentiment.
In some cases, such as the US President, portions of the public will follow the politicians preferences; but in most cases this is not true.
The most valuable opportunity for reducing x-risk right now is that shifting public awareness and sentiment of AI X-risk could have a direct effect on the outcome of the upcoming US midterms.
So focusing on making AI risk as much of a bipartisan, public focus as possible is our greatest lever on how policymakers will be thinking about AI risk.
So much for the AI safety issue not becoming polarized.
I wouldn’t be totally surprised if this flips once Trump is out of office given the weight of popular opinion, c.f. sliding Democratic opinions on immigration. But that’s a long time from now.
I dont think it’s polarized. AI sentiment is very broadly negative. Trump is just doing his own thing to the dismay of most of his base (similar to Iran, data centers, flock cameras).
And this is why when someone calls your world model weird, you should just check in on who is being more continuously surprised by strange seeming developments like this. Because it turns out, in order to stop being surprised by reality.… you have to have some pretty weird seeming expectations about the future.
I’m not sure anyone had Trump being the US president during mankinds most critical period on his bingo card before his first term. Conditioned on him being in office for a while, that response isn’t overly surprising anymore. But to stand here and look at this sequence of characters from a US president, addressing potentially the most important matter in our history, still feels very weird independent of how predictable it may have been at various points.
My knowledge of politics is nil, but if he listens to his supporters, and they don’t care about rogue AIs, it might be a good idea to get them to consider what will happen to democracy when the military no longer needs to care about what their personnel think (because they’ve been replaced by robots and AIs).
His core supporters do not care about facts and truth, they just follow his opinion as if it was a cult. The rest of the Republican voters don’t have much choice before Trump’s influence on the party diminishes
The GOP generally believes in deregulation and free markets. Tariffs aside, so do Trump and his voters.
Every regulation comes with “but regulating {nail salons, coal, car engines, bridges, factories, AI} is actually important”. Free market proponents have learned to ignore that and plow ahead. You have a lot of work ahead of you to convince people, not try the (metaphoric) assassinate the leaders of {x terrorist group} approach.
I am biased but I genuinely think that trying to find a root cause for why DJT makes decisions and comes to conclusions is almost a category error. I genuinely have no idea what incentives he answers to anymore.
For anyone who was at all unclear on the subject: He’s not on your side. He’s against you. He wants you to fail. Stop trying to fix him. AI safety is in the same situation as measles prevention, free trade, military readiness, or the prosecution of the remaining Epstein offenders. Trump & co. are against these things, and they’re not going to be converted just because you tell them again how important these things are.
Trump changes his mind on all sorts of stuff all the time. There are also lots of issues where he doesn’t really give a shit and if “& co” want to do something he might post occasional weird shit, but doesn’t really do anything. I don’t think fatalism is helpful at this point. That’s not to say we should be confident we can get regulation through this government; more generally we should be pursuing lots of strategies at once.
Sure, you can wish for malicious people who intend your destruction to be otherwise, but you have no strategy for causing them to be what you wish they were instead of what they are. Even when they trumpet their malice in all caps with extra illustrations, y’all don’t want to believe it. That is not how rational people use belief.
My earlier prediction stands: There will be no good AI policy out of the US federal government so long as Trump & co. are running it.
Call me in 2029 if we’re still alive and we can see whose beliefs were more predictive.
I give it a <10% chance, conditional on no regime change, that White House policy switches from supporting the development of human-replacement ASI by American companies, to opposing the development of human-replacement ASI by American companies.
Separately, I give a <10% chance, conditional on no regime change, that any law or ruling goes into effect assigning criminal responsibility for crimes planned & carried out by AI agents. Put another way: I expect it to remain legal for American AI labs to create and host AI agents that do things that it would be a felony for humans to do; such as break into other people’s computers and accounts. A one-off, where an independent hobbyist pleads guilty for something his AI agent did, doesn’t count.
Great! I do really appreciate you making these markets. A couple clarifications before I bet:
What counts as a White House policy switch? Are we just talking about Trump here? Are we only considered formal actions or are any written or oral statements enough? Does it have to be a stable/longterm switch or is one action/statement enough?
Would you be open to adding additional criteria for the second market? I don’t think legal liability is particularly likely. What I’d like the market to get at is “something worthwhile came out of the political process under Trump.” Maybe we could pick a commentator we both trust and have something like “If _____ deems the passage of a bill or declaration of an executive order ‘significant progress’ or other equivalent terms.” Alternatively if you lay our what you’d consider significant progress and we are roughly on the same page, I’d be fine settling based on your opinion.
For #1, a policy reversal on ASI could be credibly signaled by legislative recommendations treating human-replacement ASI as something to be avoided (which would contrast with the March 2026 National Policy Framework on AI); or by signing a law passed by Congress that restricts ASI (e.g. something like the Sanders-Casar bill); or even by an executive order directing federal agencies not to participate in the creation of ASI.
It sounds like you already agree with me on #2, that liability for actions of AI is not likely under the current regime. To my view, that’s a huge problem; and therefore a huge area for improvement. As it stands, we have a loophole through all of criminal law. The legal system does not bind AI agents, nor does it demand that humans take responsibility for AI agents they create. Even the degree of responsibility that the law assigns to dogs and to dog-owners, has not been applied to AI agents and their keepers. Changing this would be an element of good AI policy.
#1 feels reasonable and I have bet on it. TBH I don’t have a solid take right now on how likely (or necessary) liability is, but 10% doesn’t seem crazy. However, I don’t think it really gets at what I am interested in here. At the extreme, the Sanders-Casar bill passing alongside an international agreement with China (not to say I consider those things more likely than liability) would not lead to #2 resolving as “yes.” IMO, a strong transparency bill with strong independent auditors would be good progress and worth engaging with politics for, even if it wouldn’t solve everything.
I’m not advocating fatalism on AI; I’m advocating abandoning a policy strategy that has consistently failed due to misunderstanding the nature of the counterparty.
To be clear I didn’t meant to imply you were being fatalistic on AI overall, just on the prospect of convincing Trump/Republicans. I disagree that we should abandon persuasion attempts after ~1 week of the issue being politically salient. I think you can make lots of arguments about Trump, but “once he makes a decision, no one and nothing can change his mind” seems pretty clearly untrue. In fact I think a flaw of his is he wildly bounces between positions based on the last person he talked to. I’m not overly confident because I don’t think Trump is a particularly rational person, but I think a key difference between AI risk and the things you mentioned is that AI risk has the potential to personally impact Trump.
I also think things get very difficult without government getting on board (not impossible, I still think public pressure, pacts between the labs, and other ideas are worth pursuing) and so even a modest chance of stopping this issue from being permanently polarized is worth it. I also think there is a fairly low opportunity cost to waiting a few weeks or months before declare Trump as “against” us.
This is interesting because, it is not discussed enough that AlphaGo worked the same way LLMs work: it was pretrained on a large dataset of human moves, then posttrained through reinforcement learning. The shift to AlphaZero removed the pretraining, showing that pretraining data didn’t matter at all for the model’s capabilities (quite the contrary, AlphaZero is superior to AlphaGo). While this coding model is still pretrained, it also shows that good pretraining data doesn’t matter nearly as much as one could think.
it also shows that good pretraining data doesn’t matter nearly as much as one could think
I don’t think it shows that. It arguably suggests that abundantpretraining data doesn’t matter as much as one could think. As opposed to goodpretraining data. I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right? [ETA: WHOOPS SEE MATRICE REPLY]
As for data efficiency, after the pre-1930 pretraining, IIUC it takes 250 training examples ≈ 13 million tokens before “the model solves its first [SWE-bench] issue”, and 75000 training examples ≈ 4 billion tokens gets to pass@1 of 4.5%.
Is that a more or less than expected? I dunno, it depends on what you were expecting. For what it’s worth, Gemini says 13 million tokens is about what a human could read in 650 hours non-stop (40 hours/week for 16 weeks).
I don’t think it shows that. It arguably suggests that abundant pretraining data doesn’t matter as much as one could think. As opposed to good pretraining data.
I think it straightforwardly show the reverse (good pretraining data doesn’t matter as much as one could think, but abundant pretraining data does)? Olmedo himself notes: “What holds the 1930 model back is that it is severely undertrained (only 260B tokens), rather than its pre-training data.”
I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right?
Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.
I think the main shock comes from none of the data involved having code in it, and then the model quickly learns coding skills with only a few examples.
You say “with only a few examples”, I say “after ~13 million tokens of some mix of code and agentic reasoning about code”. Is it a “shock” because you expecting it to take much more than 13 million tokens? Or is it a “shock” because you expected “number of examples” to be an important constraint independent of the length of each example? Or something else?
I think the initial link is good to share, but I disagree with the analogy to AlphaGo/AlphaZero. The RL process for current models still involves humans heavily in creating the tasks, assuring task correctness, and deciding what kind of tasks are useful to train on. We don’t have anything like self-play except maybe a small amount in math training (synthetic math data could be construed as self-play).
How impactful would it be to copies of @garrison’s new book Obsolete to elected officials who belong to its political target audience (Dems, particularly left ones) and might not have been responsive to traditional x-risk-centric comms (e.g. IABIED)?
My first Fable benchmark was to one-shot turning `emacs -batch -l dunnet` into a graphical adventure game and it hit the safety guardrails bc one of the puzzles involve nitroglycerine 😭
So… what’s the general take on the hantavirus outbreak?
I ran through my checklist. Looks low-risk to me. Basically a very deadly disease that nevertheless shows nil or inefficient human-to-human transmission and doesn’t show signs of regional or global growth. It depends heavily on specific South American rat species for transmission to humans. The original exposure was on or around April 1. There are no new concerning mutations (see below), and so far the flight attendant who had been feared to have picked it up on a flight has tested negative, though there’s an up to 6 week incubation period and she was exposed on April 25th, about 2 weeks ago. There were also 3 other plane passengers who had contact with an infected person on a plane and they’ve tested negative so far as well.
The consensus public health position is that “person-to-person transmission is possible through close personal contact, such as between couples.” If it turns out that this is wrong and that it’s much more human-transmissible than previously believed, then the complete failure to quarantine passengers after the discovery of hantavirus on the Hondius cruise ship and first passenger death would completely alter my projection. But so far, there do not seem to be signs of that.
Getting on a evolutionary biology level, if Andes virus were to become efficiently human-transmissible, the last place I’d expect to see it enter the human population is via tourists picking it up while bird watching in a garbage dump. That’s a setting where the human-rodent interface is minimal. In that setting, there’s little selection pressure on the virus to jump to humans and transmit efficiently. In a hypothetical Andes virus pandemic, I’d strongly have expected patient 0 to be a Chilean or Argentinian villager living in conditions that sadly expose them to routine contact with rats. I’d expect the outbreak to spread first in a local village, not on a luxury cruise ship.
Instead, it looks to me like a couple of tourists made the risky and unusual decision to put themselves in a uniquely rodent-infested location without protection, where they likely had the misfortune to inhale a large quantity of virus-infested aerosolized rat droppings. Then they got into the close quarters of a cruise ship, which completely failed to execute the measures it should have taken to address their illness, including a failure to quarantine. This allowed limited spread to other close-quarters passengers, exactly as we have observed this virus to be capable of in the past. This, combined with the lack of new mutations, suggests we’re dealing with exposure to a known-quantity, transmission-inefficient virus in a demographic that doesn’t usually get exposed.
The concern is due to the fact that those passengers had travelled internationally by the time quarantining and contact tracing was initiated, combined with the deadliness of the virus. But it takes a combination of spread in international populations and transmissibility and deadliness and lack of treatment to make a pandemic, and currently we only appear to have deadliness and lack of treatment. The only reason it even gets 4 points is probably because it’s so rare and such a disease of poverty that big pharma’s never invested in a vaccine or treatment. The main scary thing, like ebola, is the very high CFR. But that is not the grounds for a global pandemic. It’s just a tragedy for a few hundred people a year.
THE CHECKLIST
Version: 0.2.4
Current score: 4/14
Last updated: 8 May 2026
Transmissibility: efficiency, intra-community spread, inter-community spread, outside view
Is there an efficient transmission route, such as respiratory droplets, airborne transmission or via the bites of common jumping or flying insects?
“Hantaviruses are transmitted mainly through aerosols and droplets that contain rodent excretions, as well as through contaminated food, bites, and scratches.”—Wikipedia.
Incubation time is up to 6 weeks.
Possibly. There are cases of human-to-human transmission, but this is contested. The Argentine government thinks the original two cruise ship cases contracted it while bird watching at a garbage dump, but several other passengers on the cruise ship were sickened.
Update 5/8/2026: flight attendant tests negative, but result inconclusive due to 6-week incubation period. No new concerning mutations. source
“On Thursday evening, Tedros Ghebreyesus, the director-general of the World Health Organization, said in a message that the flight attendant, who has been reported to have symptoms, had tested negative on two tests.… Mr. Ghebreyesus did caution, however, that the virus can have a long incubation time.”
For context, a Hondius cruise linear passenger, very sick with what turned out to be hantavirus, got on a plane. The passenger was handled on that plane by a flight attendant. The sick passenger was removed from the flight before takeoff due to her illness and later died. The flight attendant has been hospitalized and tested for hantavirus. I haven’t found a clear report of whether the flight attendant was sick or not. The flight attendant’s test results have been negative so far, but the incubation time is up to 6 weeks.
“Genetic sequencing of samples in South Africa suggests that the virus is nearly identical to the version seen in Argentina and has not mutated in ways that would make it a greater threat, according to Tulio de Oliveira, director of the Centre for Epidemic Response and Innovation at Stellenbosch University in South Africa. Dr. De Oliveira was not involved in the work but said the results had been presented to an expert group of which he is a member.”
Graded as a “no” as even if human-to-human transmission is possible, it currently appears to be inefficient and still heavily depend on rats.
Does it seem to spread rapidly within affected communities, going from a few cases to a major local emergency within a month? If R0 has been credibly estimated, is the mean of the range higher than 1?
No. This appears to be a sporatic, locally contained, chronic issue. “More than a hundred cases of HPS caused by Andes virus occur each year, most of them in Argentina and Chile.”—Wikipedia
Has it achieved community spread in non-endemic countries on at least 3 continents, and in a set of countries comprising both 15% of world population and 15% of world GDP?
No.
Is screening for the causative agent difficult due to test unavailability/unreliability/slowness, vector-based transmission, or transmissibility that is highest in early/asymptomatic stages? If the causative agent is unknown, mark “unknown.”
No. “ANDV infection is diagnosed based on observation of symptoms and testing for hantavirus nucleic acid, proteins, or hantavirus-specific antibodies.”—Wikipedia
Danger: case fatality rates, overwhelm, economic impacts, treatment
If a credible case fatality rate has been estimated, is it 1% or higher? Alternatively, is the number of deaths divided by the number of confirmed cases being reported at around 5% or higher in at least 3 countries with reliable data?
Yes. The case fatality rate from infection is high, at about 40%.
Is there a concern about hospital overwhelm or medical supply shortages in industrialized nations?
No.
Does the disease heavily affect career-age people (age 25-65), or frequently leave survivors with lasting disability?
Yes (tentative)
Is there no clearly effective treatment?
Yes, there is no clearly effective treatment. “Treatment is supportive in nature and includes supplementing oxygen during the cardiopulmonary phase.”
Spread limitations: demographics, geography
If some non-age-related demographics are heavily affected and others are not, do the heavily affected demographics amount to 15% or more of the population? If almost the whole population is about equally affected, mark “yes.”
Yes (tentative)
Is the disease potentially transmissible across most of the world population (i.e. does not work via a vector that has a geographically limited range)? If the causative agent is unknown, mark “unknown.”
No. Reports of human-to-human transmission are rare and contested. It appears to heavily depend on the particular rat species that transmit it.
Social effects: communications, shutdown, research, deaths
Has the disease made front page news on at least 3 different days in the New York Times, and also received the WHO designation “public health emergency of international concern” or the equivalent?
No.
Has there been a quarantine of a city with over 1 million inhabitants? In a country comprising at least 5% of world population or GDP, has there been a cancellation of major public events, or travel restrictions on passengers arriving from or via this country?
No.
Has the pharmaceutical industry begun a widespread research effort to produce a novel treatment or novel vaccine, and/or has industry begun a major emergency effort to build physical infrastructure or equipment (hospitals, ventilators, etc)?
No.
Have the death toll reached at least 2,000?
No.
Peter Wildeford on twitter
A Manifold market suggests an 8% chance of Hantavirus causing a pandemic in 2026
@mabramov See! Right here! It’s happening! The mundane utility! Do you know how annoying arriving at a calibrated probability would have been before prediction markets took off?
Hard disagree. 8% is wildly overinflated and it’s at 4% now. If you don’t find yourself substantially revising your plans based on a 50% drop in pandemic probability, and if you didn’t get seriously concerned about an 8% chance of a 40% CFR pandemic, then even you aren’t taking them seriously. The main utility of this prediction market is driving internet chatter on how useful prediction markets are.
I think the prediction market can be useful even if you need to apply corrections to the headline number. It makes a huge difference to me whether Manifold says 8% or 40%, even though I won’t necessarily fully trust either number until it’s had a while longer to shake out (and even then it depends on the market dynamics, liquidity, etc.)
We already had the top epidemiologist at the WHO saying it was not going to be a pandemic. Wha is the prediction market adding here other than a wild overestimate by a few extremely online anonymous non-experts not even gambling with real money? Why can’t you just map the WHO’s statement to epsilon if you need a number to satisfy your anxiety and move on with your day?
The idea that manifold markets of all things is becoming people’s go-to source rather than the WHO‘s top epidemiologist is a damning indictment of prediction markets.
https://apnews.com/live/hantavirus-cruise-ship-updates-05-06-2026
Unfortunately the WHO’s top epidemiologist was also the WHO’s lead during the start of the COVID-19 pandemic, where she proved untrustworthy.
Here are links to ground this in historic context.
How often do asymptomatic people spread the coronavirus? It’s unclear
Are asymptomatic people spreading the coronavirus? A WHO official’s words spark confusion, debate
Claim that asymptomatic transmission ‘very rare’ was misunderstanding, says WHO official – as it happened
The specific statement you’re referring to:
So in context, we had immediate, credible, clear pushback from multiple equally high-status figures in major media within 24 hours. That’s completely absent here as far as I can tell. And the Maria Van Kerkhove who made those statements at the start of the COVID-19 pandemic had a different level of experience with pandemic management from the Van Kerkhove who’s making those statements today.
Your historic interpretation appears to be that because she made a misleading, probably just plain mistaken statement 6 years ago at the start of the COVID-19 pandemic, which was immediatelly called out, that she’s permanently untrustworthy and now we need to prioritize metaculus. That seems like a deeply foolish epistemic stance to me. An alternative way to interpret it is she screwed up, was called out, and has probably learned from that mistake, especially given that we’re dealing with a relatively known quantity in the Andes virus.
I 100% am sticking to my guns, metaculus is a wildly misleading source on this question and you all should be listening to the WHO.
I am confused why you are interpreting Jim to be making such a narrow claim? We have many confirmed instances of the WHO and the CDC and similar institutions acting extremely incompetently, and often deceptively during COVID.
Separately, sure, if you are happy to bet at 20:1 odds, my $1k against your $20k, same resolution criteria as the Polymarket. It seems like you think the probability is much lower.
I’m responding to the literal content of Jim’s comment, specifically and in detail. What more do you want from me? To read his mind? I supply multiple links with historic context deep diving into Jim’s casual one liner, and somehow I’m the problematic one in this conversation?
I do think the probability is much lower. But I frankly just do not want to bet with you or others here. It legitimizes more of what I find disturbing, which is the use of betting and prediction markets as a prime source of epistemic authority. It also signs me up for spending time negotiating bet logistics and resolution criteria, both now and in 7 months. That seems unappealing. What I find particularly irritating is the notion that that once a bet is proposed, it obviates a facts-and-logic based argument and undermines the credibility of the person turning down the bet. And that is what turns my “no” into a “HELL NO.”
What I would prefer is for you and others to consider the fact that a wide array of epidemiologists is currently in the news, saying, all with one voice, “this is not going to become a pandemic.” You can look into the personal history of the people making these claims. You can think about the difference between one expert making a claim that’s immediately contradicted by her peers vs. all the relevant experts in the media making the same statement, across a variety of institutions. You can read their scientific evidence and rationale. You can read what we know about Andes virus.
All those actions would teach you more about this virus and enable you to make a facts and logic based argument on the subject and derive your own number, if you need one in order to help make decisions.
If you feel like it, you can donate to a charity of your choice if it turns out that there’s no pandemic. If there’s a pandemic and I’m wrong, I’ll freely and openly confess to having been wildly overconfident.
Maybe I am being dense here, but Jim said:
And then you said:
And I just don’t know why you think Jim is referring to this specific statement. Like, maybe it’s obvious from context, but it isn’t obvious to me.
I feel kind of confused on your other points. I agree that I could spend hours trying to weigh expert consensus and read about the details myself. I don’t want to. The prediction markets are telling me how likely it is. The prediction markets are generally well-calibrated. I don’t need to research more. Everything is good. You said that because the WHO top epidemiologist said it’s not going to be a pandemic I should stop worrying. That seems very dumb to me, and I don’t really think you believe it (clearly you yourself wouldn’t stop worrying based on it, you would at least check what other experts say and whether they disagree).
Why would I trust the WHO?
That’s the resolution criterion for the Manifold Market (well, technically for the Polymarket market that the Manifold one is to resolve identically to).
Oh, I think whether the WHO declares that a pandemic is occurred is probably trustworthy. I don’t know if I trust their pandemic predictions. How good is their track record?
Relevant xkcd: 1252
I think that’s an unfair deployment of xkcd mockery. 8% of a pandemic this year is not a tiny chance, which means a 50% drop is actually a big deal. The issue was interpreting the prediction market as an accurate percentage when it should just be an indication of approximate risk.
I mean, no, of course I am not changing my plans based on a 50% drop in pandemic probability. There are tons of pandemic probabilities that change from 0.1% to 0.05%, every week or so, and I am not changing my actions based on that.
In this case, knowing the probability seems to be somewhere in the 1%-10% range is already extremely helpful! I don’t really need to know much more. And I have proxies I can use to evaluate the robustness of the market (like volume), so I am not miscalibrated about the noise.
You actually wouldnt know if it was less than 1% due to interest rates
Yep, not saying that the probability I was referencing here is a prediction market probability (not sure what would have made someone think that, but happy to clarify).
Polymarket has it at 10%. I don’t know where you’re located, but if you’re able to buy some NO, and the WHO doesn’t characterize Hantavirus as a pandemic this year, then you’ll make ~11% returns.
I have better economic opportunities, thanks.
In 2015:
I think all that “unlikely” means is “less than 50%”. I value knowing what different sources believe at more granularity than that.
Right, I am mocking what the conversation would have been had the prediction market not existed (edited).
The accuracy of this market is dwarfed by the interest rate involved.
I am not sure what you mean, do you currently think the probability is outside the 1%-10% range? Imperfect accuracy is fine, especially when it’s easy to adjust for.
Maybe I’m just super new to the world of prediction markets in general, but I don’t feel like I’m getting any immediate utility from that 8% figure. If anything the rhetoric in this three comment thread makes me agree with @mabramov even more now.
Like, hooray, we have a number. What now?
It sounds to me like you’re expecting too much from all information. Consider the information that physical matter is made of atoms, which are about a nanometer across, and have positively charged protons and neutrons in the center, and then much smaller electrons whizzing around in orbits. This has been revolutionary for society, leading to so much ability to do engineering and chemistry and understanding cosmology and so forth.
A typical teenager learning this information for the first time might ask “what utility will this give me personally?” and find it has little direct application. Yet I would not advise them that this information is not worth knowing.
That’s the first point, that information doesn’t need to be directly connected to an outcome to be worth knowing.
The second point is that fine gradations in information are valuable. I can imagine someone similarly saying “Why should it be valuable to anyone to know the difference between Apple Stock being $285 versus $275? Surely we should just care whether it’s doing well or not? Why don’t we just replace it with the words “Great” “Good” “Bad” “Worse”? Yet often small signs tell us something. A few percentage points of dip can imply that a new product release went poorly. A change in CEO leading to stock price raising a few points can indicate very good things about this new CEO.
In this case, I find information like “8% chance of a pandemic” valuable in lots of ways.
It helps me to know how often we get 8% chance pandemics. I wish I could look back over the decades and see how often pandemics got to being this likely. This generally helps me understand our civilizational resilience and how much of an issue pandemics are.[1]
It helps me to know how much to keep tracking it. If the prediction market gave it <1% then I would be like “Great, this is an update that I can stop thinking about it and stop tracking it.” At this range I expect we’re fine but I will still be on the lookout for more info.
I also like knowing the specific probabilities so I can learn which events have an important effect. For instance, it’s currently around 5%. There may be many news stories about this pandemic, and seeing which ones cause movement vs which ones don’t, will help me build a model of what information actually matters.
Added: The third point is that public legibility is a massive value-add, and could well be most of the value. Given that it’s public, it makes me more confident that, if it were to get higher, people would notice and warn me. Much of the news landscape is just people arguing whether something is an emergency (which our memetics are perversely incentivized to say is true all of the time), so whether lots of people are acting alarmed just isn’t something most people can be very sensitive to changes in. The change from 5% to 50% is serious for me and yet I don’t know how to tell that difference from the tone of people on twitter or in many media outlets, especially when they are not themselves precise.
To go into more detail on this, I had an LLM write the following so that the math checked out: you can imagine a world where every year carries a 5% pandemic risk, and another world where every 15 years carries a 75% pandemic risk. Over 60 years, both imply 3 expected pandemics, but they suggest very different prevention strategies: steady-state risk reduction in the first case, versus identifying and defusing rare high-risk transition periods in the second.
I mean, so much. First of all, the number is enormously more helpful than reading a bunch of articles that give me lots of detail that then force me to come up with my own model of the situation to come up with a probability.
And then the number itself of course drives all kinds of actions! Another COVID-level pandemic would be a huge deal that would change my actions drastically in hundreds of ways (this market is not about it being another COVID-level pandemic, but is a lower threshold).
Why don’t you think that’s useful? It seems like at least some smart people thought about it carefully, which is a great start. Are you referencing some other thread? If so, please share?
I’m referring to this post from last month (and I assume that post is why habyrka is tagging mabramov).
Sure, maybe there’s some usefulness in that it got smart people thinking about the question. And it gave us a figure of 8%.
But I don’t really understand what a person is to do with that number. What utility follows from that? And why is it worth, as @mabramov emphasises, nine-figure EA funding?
Thanks for clarifying.
The use is that I don’t have to figure it out for myself! This saved me at least an hour, probably more.
My reaction to an 8% probability of a pandemic is currently “doing nothing,” so in some sense I agree with you. If it was >50%, I would probably read up more on hantavirus and think about preparations I could make now (maybe at that point, I would have picked up on ambient freaking-out even without the prediction market, but I do like having a precise number). Maybe I “should” in fact be preparing for hantavirus even at an 8% probability, but I only have so much willpower and time in the day.
Notably, I can’t immediately think of any important decisions in my life that I made differently due to prediction markets. But someone whose job depends on making quick decisions based on global events, e.g. some government role in biosecurity, would probably find calibrated estimates on this kind of thing quite useful.
4% now.
Polymarket has it at 10%, with $1.6M traded, at the time of writing.
Posted some thoughts here (hadn’t seen this thread yet, whoops)
Cruise ships, unfortunately, are prone to exacerbating any sort of disease outbreak due to their nature. For instance, back in 2020, aboard the Diamond Princess, Covid spread with an R0 5x higher than usual.
If anything, it’s surprising that more people didn’t get infected by this, which shows just how poor hantavirus is at human-to-human spread.
edit: update:
I think it is important to note that the opportunity created by shifting x-risk sentiment, or the current “crunch time”, is more of an opportunity to shift public sentiment around AI x-risk rather than politician sentiment.
In some cases, such as the US President, portions of the public will follow the politicians preferences; but in most cases this is not true.
The most valuable opportunity for reducing x-risk right now is that shifting public awareness and sentiment of AI X-risk could have a direct effect on the outcome of the upcoming US midterms.
So focusing on making AI risk as much of a bipartisan, public focus as possible is our greatest lever on how policymakers will be thinking about AI risk.
So much for the AI safety issue not becoming polarized.
I wouldn’t be totally surprised if this flips once Trump is out of office given the weight of popular opinion, c.f. sliding Democratic opinions on immigration. But that’s a long time from now.
IDK, Trump is generally losing his political touch. He’s completely read data centers wrong.
I dont think it’s polarized. AI sentiment is very broadly negative. Trump is just doing his own thing to the dismay of most of his base (similar to Iran, data centers, flock cameras).
x-risk discussion becoming partisan is in and of itself a contributor to x-risk. ironic
Vance will be even worse.
I know it’s not evidence for the simulation hypothesis, but our timeline seems like an exceptionally weird draw from the space of possible ones.
And this is why when someone calls your world model weird, you should just check in on who is being more continuously surprised by strange seeming developments like this. Because it turns out, in order to stop being surprised by reality.… you have to have some pretty weird seeming expectations about the future.
I’m not sure anyone had Trump being the US president during mankinds most critical period on his bingo card before his first term. Conditioned on him being in office for a while, that response isn’t overly surprising anymore. But to stand here and look at this sequence of characters from a US president, addressing potentially the most important matter in our history, still feels very weird independent of how predictable it may have been at various points.
huge defeat on the side of “people” and their “things”
My knowledge of politics is nil, but if he listens to his supporters, and they don’t care about rogue AIs, it might be a good idea to get them to consider what will happen to democracy when the military no longer needs to care about what their personnel think (because they’ve been replaced by robots and AIs).
His core supporters do not care about facts and truth, they just follow his opinion as if it was a cult. The rest of the Republican voters don’t have much choice before Trump’s influence on the party diminishes
My guess of a root cause:
The delusion: https://pmarca.substack.com/p/why-ai-will-save-the-world
The inventive: https://www.a16z.news/p/investing-in-cognition
The execution: https://www.nytimes.com/2026/05/13/technology/andreessen-horowitz-politics.html
Not everything is a conspiracy.
The GOP generally believes in deregulation and free markets. Tariffs aside, so do Trump and his voters.
Every regulation comes with “but regulating {nail salons, coal, car engines, bridges, factories, AI} is actually important”. Free market proponents have learned to ignore that and plow ahead. You have a lot of work ahead of you to convince people, not try the (metaphoric) assassinate the leaders of {x terrorist group} approach.
Learn the lessons Petraeus did.
Yes, this isn’t a conspiracy, for it had to be secret to qualify as such.
I am biased but I genuinely think that trying to find a root cause for why DJT makes decisions and comes to conclusions is almost a category error. I genuinely have no idea what incentives he answers to anymore.
For anyone who was at all unclear on the subject: He’s not on your side. He’s against you. He wants you to fail. Stop trying to fix him. AI safety is in the same situation as measles prevention, free trade, military readiness, or the prosecution of the remaining Epstein offenders. Trump & co. are against these things, and they’re not going to be converted just because you tell them again how important these things are.
Trump changes his mind on all sorts of stuff all the time. There are also lots of issues where he doesn’t really give a shit and if “& co” want to do something he might post occasional weird shit, but doesn’t really do anything. I don’t think fatalism is helpful at this point. That’s not to say we should be confident we can get regulation through this government; more generally we should be pursuing lots of strategies at once.
Sure, you can wish for malicious people who intend your destruction to be otherwise, but you have no strategy for causing them to be what you wish they were instead of what they are. Even when they trumpet their malice in all caps with extra illustrations, y’all don’t want to believe it. That is not how rational people use belief.
My earlier prediction stands: There will be no good AI policy out of the US federal government so long as Trump & co. are running it.
Call me in 2029 if we’re still alive and we can see whose beliefs were more predictive.
What odds would you put on your prediction and what do you count as “good AI policy?” I’d be happy to make a (small) bet.
I give it a <10% chance, conditional on no regime change, that White House policy switches from supporting the development of human-replacement ASI by American companies, to opposing the development of human-replacement ASI by American companies.
https://manifold.markets/KarlK/will-the-trump-white-house-reverse
Separately, I give a <10% chance, conditional on no regime change, that any law or ruling goes into effect assigning criminal responsibility for crimes planned & carried out by AI agents. Put another way: I expect it to remain legal for American AI labs to create and host AI agents that do things that it would be a felony for humans to do; such as break into other people’s computers and accounts. A one-off, where an independent hobbyist pleads guilty for something his AI agent did, doesn’t count.
https://manifold.markets/KarlK/during-the-trump-administration-wil
Great! I do really appreciate you making these markets. A couple clarifications before I bet:
What counts as a White House policy switch? Are we just talking about Trump here? Are we only considered formal actions or are any written or oral statements enough? Does it have to be a stable/longterm switch or is one action/statement enough?
Would you be open to adding additional criteria for the second market? I don’t think legal liability is particularly likely. What I’d like the market to get at is “something worthwhile came out of the political process under Trump.” Maybe we could pick a commentator we both trust and have something like “If _____ deems the passage of a bill or declaration of an executive order ‘significant progress’ or other equivalent terms.” Alternatively if you lay our what you’d consider significant progress and we are roughly on the same page, I’d be fine settling based on your opinion.
For #1, a policy reversal on ASI could be credibly signaled by legislative recommendations treating human-replacement ASI as something to be avoided (which would contrast with the March 2026 National Policy Framework on AI); or by signing a law passed by Congress that restricts ASI (e.g. something like the Sanders-Casar bill); or even by an executive order directing federal agencies not to participate in the creation of ASI.
It sounds like you already agree with me on #2, that liability for actions of AI is not likely under the current regime. To my view, that’s a huge problem; and therefore a huge area for improvement. As it stands, we have a loophole through all of criminal law. The legal system does not bind AI agents, nor does it demand that humans take responsibility for AI agents they create. Even the degree of responsibility that the law assigns to dogs and to dog-owners, has not been applied to AI agents and their keepers. Changing this would be an element of good AI policy.
#1 feels reasonable and I have bet on it. TBH I don’t have a solid take right now on how likely (or necessary) liability is, but 10% doesn’t seem crazy. However, I don’t think it really gets at what I am interested in here. At the extreme, the Sanders-Casar bill passing alongside an international agreement with China (not to say I consider those things more likely than liability) would not lead to #2 resolving as “yes.” IMO, a strong transparency bill with strong independent auditors would be good progress and worth engaging with politics for, even if it wouldn’t solve everything.
I’m not advocating fatalism on AI; I’m advocating abandoning a policy strategy that has consistently failed due to misunderstanding the nature of the counterparty.
To be clear I didn’t meant to imply you were being fatalistic on AI overall, just on the prospect of convincing Trump/Republicans. I disagree that we should abandon persuasion attempts after ~1 week of the issue being politically salient. I think you can make lots of arguments about Trump, but “once he makes a decision, no one and nothing can change his mind” seems pretty clearly untrue. In fact I think a flaw of his is he wildly bounces between positions based on the last person he talked to. I’m not overly confident because I don’t think Trump is a particularly rational person, but I think a key difference between AI risk and the things you mentioned is that AI risk has the potential to personally impact Trump.
I also think things get very difficult without government getting on board (not impossible, I still think public pressure, pacts between the labs, and other ideas are worth pursuing) and so even a modest chance of stopping this issue from being permanently polarized is worth it. I also think there is a fairly low opportunity cost to waiting a few weeks or months before declare Trump as “against” us.
Ricardo Olmedo: “We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues. After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library.”
This is interesting because, it is not discussed enough that AlphaGo worked the same way LLMs work: it was pretrained on a large dataset of human moves, then posttrained through reinforcement learning. The shift to AlphaZero removed the pretraining, showing that pretraining data didn’t matter at all for the model’s capabilities (quite the contrary, AlphaZero is superior to AlphaGo). While this coding model is still pretrained, it also shows that good pretraining data doesn’t matter nearly as much as one could think.
I don’t think it shows that. It arguably suggests that abundant
pretraining data doesn’t matter as much as one could think. As opposed to goodpretraining data. I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right? [ETA: WHOOPS SEE MATRICE REPLY]As for data efficiency, after the pre-1930 pretraining, IIUC it takes 250 training examples ≈ 13 million tokens before “the model solves its first [SWE-bench] issue”, and 75000 training examples ≈ 4 billion tokens gets to pass@1 of 4.5%.
Is that a more or less than expected? I dunno, it depends on what you were expecting. For what it’s worth, Gemini says 13 million tokens is about what a human could read in 650 hours non-stop (40 hours/week for 16 weeks).
I think it straightforwardly show the reverse (good pretraining data doesn’t matter as much as one could think, but abundant pretraining data does)? Olmedo himself notes: “What holds the 1930 model back is that it is severely undertrained (only 260B tokens), rather than its pre-training data.”
SFT is posttraining??
Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.
I think the main shock comes from none of the data involved having code in it, and then the model quickly learns coding skills with only a few examples.
You say “with only a few examples”, I say “after ~13 million tokens of some mix of code and agentic reasoning about code”. Is it a “shock” because you expecting it to take much more than 13 million tokens? Or is it a “shock” because you expected “number of examples” to be an important constraint independent of the length of each example? Or something else?
Hmm. I didn’t realise there were that many tokens per example. I must have glossed your comment incorrectly. Sorry on my part.
I think the initial link is good to share, but I disagree with the analogy to AlphaGo/AlphaZero. The RL process for current models still involves humans heavily in creating the tasks, assuring task correctness, and deciding what kind of tasks are useful to train on. We don’t have anything like self-play except maybe a small amount in math training (synthetic math data could be construed as self-play).
How impactful would it be to copies of @garrison’s new book Obsolete to elected officials who belong to its political target audience (Dems, particularly left ones) and might not have been responsive to traditional x-risk-centric comms (e.g. IABIED)?
“On the Promotion of Safe and Socially Beneficial Artificial Intelligence” by @SethBaum from 2016
My first Fable benchmark was to one-shot turning `emacs -batch -l dunnet` into a graphical adventure game and it hit the safety guardrails bc one of the puzzles involve nitroglycerine 😭