I think this misses the point, since the problem is[1] less “One guy got made psychotic by 4o.” and more “A guy who got some kind of AI-orientated psychosis was allowed to continue to make important decisions at an AI company, while still believing a bunch of insane stuff.”
Conditional on the story being true
Recently I was talking to someone about Pause AI’s activities in the UK. They asked something like
“Even if Kier Starmer (UK Prime Minister) decided to pause, what would that do?”
And I responded:
“We would have the resources of an entire nation state pointed towards our cause.”
Canada (all this applies to the UK too) has a lot more resources to throw around than MIRI, Lighthaven, OpenPhil, and Pause/Control/Stop AI all put together. Canada is a founding NATO member. If Canada’s government, and military intelligence leaders, were “Doomer-pilled” on AI, they could do a lot more than we have done.
I recently, rather embarrassingly, made a post with a massive error which an LLM would have found immediately. I seriously misread a paper in a way that cut/pasting the paper and the post into Claude and asking “any egregious misreadings” would have stopped me from making that post. This is far too useful for me to turn down, and this kind of due diligence is +EV for everyone.
The presence of induction heads would be a decent answer. If you show a model …[A][B]...[A] It will often complete it with [B]. One specific mechanism which can cause this is well-studied, but the general reason is that this pattern is really common in lots of different types of pretraining data.
Books are quite long. I think we should go with the shortest possible description of his life.
I saw someone (cannot find source) say that “Eliezer wrote 300% of the book, and then Nate wrote −200%”. I’ve not read much of Nate’s writing other than on LessWrong, so I don’t know if he’s solely responsible for the change in tone, or if Eliezer was writing more normie-ish to begin with, or what.
The hypothetical ammonia-reduction-in-shrimp-farm intervention has been touted as 1-2 OOMs more effective than shrimp stunning.
I think this is probably an underestimate, because I think that the estimates of shrimp suffering during death are probably too high.
(While I’m very critical of all of RP’s welfare range estimates, including shrimp, that’s not my point here. This argument doesn’t rely on any arguments about shrimp welfare ranges overall. I do compare humans and shrimp, but IIUC this sort of comparison is the thing you multiply by the welfare range estimate to get your utility value, if you’re into that)
(If ammonia interventions are developed that are really 1-2 OOMs better than stunning, then even under my utility function it might be in the same ballpark as the campaign against cage-free eggs and other animal charities)
Shrimp stunning, as an intervention, attempts to “knock out” shrimp before they’re killed by being placed on ice, which has been likened to “suffocating in a suitcase in the antarctic”.
I think that’s a bad metaphor. Humans are endotherms and homeotherms, which means we maintain a constant internal body temperature which we generate internally. If it drops, a bunch of stress responses are triggered: shivering, discomfort, etc. which attempt to raise our temperature. Shrimp are poikilotherms, meaning they don’t regulate body temperature much at all. This means they don’t have the same stress responses to cold that we do.
(I also doubt that, given they have no lungs and can basically never “not breathe” in their normal environment, they’d experience the same stress from asphyxiation that we do, but this is weaker)
I would guess that being thrown onto ice effectively “stuns” the shrimp pretty quickly, as their metabolism—and therefore their synaptic activity—drops with their body temperature.
That seems roughly right but I don’t think chaos/revolution is necessary for this thesis.
Imagine the world sitting in a deep basin. You’re trying to get it out. You can try a revolution: launch it up into the air and hold your breath waiting for it to land.
Or you can try, sisyphus-style, to roll it out. You might get a bunch of allies, who you try and lead along a narrow and precarious ridgeline to your destination. Then, when you’re halfway to the top, some might notice that there’s a much easier path requiring much less of a push, and so the world gets pushed down off your intended route along the mountain pass and into a new crater.
The latter seems to have been the case with environmentalism and AI Safety. They pushed the world fairly gradually, but it only takes a few individuals to push the world downwards into a new equilibrium, for the upward pushers to lose that fight.
(There’s a Yudkowsky tweet somewhere which says something like this, which I’ve basically based the entire metaphor here on)
I think we have evidence that this is not the primary thing going on, because there are parts of the evaluation where the model lacks context, could easily face an identical situation in deployment, and thus can’t know it is being evaluated.
This in particular is not strong evidence, since the optimal strategy for a schemer is usually misbehave only very rarely, when it’s very confident that it isn’t in an evaluation. This is because the number of deployment runs will be >> the number of evaluations. If it’s unable to tell, then it should probably just not misbehave in that instance.
Well OpenAI, a16z, and Meta have been pretty clear that any regulation whatsoever will spell the end for the AI industry. I expect all the AI companies will be closing up shop within the next few weeks, and us lot working on X-risk can all take a well-deserved break.
Bioweapons are lousy weapons, because they aren’t effectively targetable. That may be a big part of your Great Filter.
Something just being infectuous (or deadly) doesn’t make it a good bioweapon. In fact, even if you have smallpox, you don’t have a bioweapon yet. You still have to package and deliver it. And if you want full-on apocalypse cred, you also have to make it a lot worse than the background of naturally occurring pathogens.
This is why none of the people smart enough to build and deploy one in 2025 have a good reason to, including nation states. If the number of people who can do it goes up by 3 OOMs, you might get someone crazy enough to build one despite these facts.
Your cost and “employer” filters haven’t changed. Although it’s probably not hard to create a good enough “business” to pass the employer one.
That’s true, but as they say in the army: three is two, two is one, and one is none! I’m not moving to New Zealand yet, but one of the three major reasons I wasn’t worried may no longer apply.
What happens when you trigger the 20 to 40 percent detection threshold? Do you just get told no and get to try again, with no other consequence? What if you trigger it 10 times?
If the filter assigns 40% chance that it’s viral, you (probably) just don’t trigger any filter. This is the bayesian probability assigned by a filtering model, not the probability that a binary filter gets triggered.
An early draft of a paper I’m writing went like this:
In the absence of sufficient sanity, it is highly likely that at least one AI developer will deploy an untrusted model: the developers do not know whether the model will take strategic, harmful actions if deployed. In the presence of a smaller amount of sanity, they might deploy it within a control protocol which attempts to prevent it from causing harm.
I had to edit it slightly. But I kept the spirit.
When you say “prevented” do you just mean it would have been generally off-putting, or is there something specific that you’re referring to here?
“Utility Maximizer” exists in the map, not the territory. It’s something we can apply to model the behaviour of things in the territory. As in all cases, models make a trade-off between simplicity and accuracy.
Some entities are particularly well modelled (by me) as carrying out a strategy of “Maximize [X]” where [X] is a short description of some outcome.
(The classic examples of “Stockfish” being well modelled by “Achieve wins in chess” comes to mind. Someone might well model a company as executing a strategy of “Maxmize your profits” or a politician as executing a strategy of “Maximize your popularity”.)
This isn’t perfect, obviously. We might need to add some extra information. For example, we can describe a chess player as executing “Win chess” but with an extra variable of “ELO = 1950″ which describes the power of that utility maximizer. Likewise, you might model a doctor as executing a strategy of “Cure patients” but subject to a limited set of knowledge. This isn’t really what people mean by “irrational” though, since these are mostly just limitations.
What really makes an entity “irrational” is when your model of it contains pretty much any other kind of behaviour. For example, those Go bots whose behaviour is well modelled as “Win at Go, ELO = Superhuman, EXCEPT behave as if you think cyclic patterns are unbeatable”. In that case, the Go engine is behaving irrationally under our model of the world.
(Another classic example: someone who mostly has consistent preferences, which can be simply described by a utility function, but also prefers apples to bananas, bananas to oranges, and oranges to apples. This puts an epicycle in our model if we have to model their fruit-swapping behaviour.)
An entity is irrational under some model to the degree that that there are more epicycles on its behaviour. A person might appear like a utility maximizer (and thus, very rational) to a much stupider person (who would not be able to model their behaviour in any other way), but very unlike a utility maximizer to a superintelligent AI. Since most humans don’t vary in intelligence by that much, most of the time we’re working under similar models, so we can just talk about entities being irrational.
Caveat:
We might want to talk about which agents are more or less rational in general, which then means we’re making a claim that our models reflect some aspect of reality. A more (or less) rational agent is then one which is overall considered more (or less) rational under a wide variety of high-accuracy low-complexity models.
Strong upvoted. I struggle to imagine a company which punishes its employees for speaking the truth, but where the company leadership decides to become sane and listen to them when shit hits the fan.
There’s lots of discourse around at the moment about
Will AI go FOOM? With what probability?
Will we die if AI goes FOOM?
Will we die even if AI doesn’t go FOOM?
Does the Halt AI Now case rest on FOOM?
I present a synthesis:
AI might FOOM. If it does, we go from a world much like today’s, straight to dead, with no warning.
If AI doesn’t foom, we go from the AI 2027 scary automation world to dead. Misalignment isn’t solved in slow takeoff worlds.
If you disagree with either of these, you might not want to halt now:
If you think FOOM is impossible, we’ll get plenty of warning to halt later.
If you think slow takeoff is survivable, you might want to press on if the chance of dying in a FOOM is worth the chance of getting to the stars in a slow takeoff world.
To be clear. I think that FOOM is both kinda likely, and that slow takeoff doesn’t save us. I also think that the counter-arguments are pretty weak and strained, and that try pretty much as hard as you can to get a halt or at least be honest that a halt would be good is obviously the best strategy even if you fail and have to rely on some backup strategy.
So the synthesis is:
FOOM isn’t core to the argument that we all die, BUT the possibility of it is a strong motivator for halting sooner rather than waiting later. A sane society would just halt now, obviously, but we don’t have that luxury.
Logical inductors consider belief-states as prices over logical sentences in some language, with the belief-states decided by different computable “traders”, and also some decision process which continually churns out proofs of logical statements in that language. This is a bit unsatisfying, since it contains several different kinds of things.
What if, instead of buying shares in logical sentences, the traders bought shares in each other. Then we only need one kind of thing.
Let’s make this a bit more precise:
Each trader is a computable program in some language (let’s just go with turing machines for now, modulo some concern about the macros for actually making trades)
Each timestep, each trader is run for some amount of time (let’s just say one turing machine step)
These programs can be well-ordered (already required for Logical Induction)
Each trader is assigned an initial amount of cash according to some relation
Each trader can buy and sell “shares” in any other trader (again, very similarly to logical induction)
If a trader halts, its current cash is distributed across its shareholders (otherwise that cash is lost forever)
(Probably some other points, such as each trader’s current valuation of itself: if no trader is willing to sell its own shares, how does that work? does each trader value its own (still remaining) shares proportional to its current cash stock? how are the shares distributed at the start?)
This system contains a pseudo-model of logical induction: for any formal language which can be modelled with turing machines, and for any formal statement in that language, there exists a trader with some non-zero initial value, which lists out all possible statements in that language, halting if (and only if) it finds a proof of its particular proposition. This makes a couple of immediately obvious changes from Logical Induction:
There’s a larger “bounty” (i.e. higher cash prize) for proving simpler statements given by earlier turing machines, since the earlier a particular prover turing machine is expressed in our ordering, the larger its starting cash
We don’t have to also define a particular speed/order in which logical statements proofs become available, since it’s already part of the system
The market’s “belief” in that statement can be expressed as that particular turing machine’s share price divided by its cash holdings
We can also intervene in the market by distributing cash to certain traders, or setting up a forcible halting rule on traders.
Does this actually do the logical induction thing? I have no clue.
What someone spends time writing is an important signal. More generally, what someone is interested in—and spend time on—is one of the most powerful signals as to what they’ll do in future.
“Real views” is a bit slippery of a concept. But I strongly predict that most of their outputs in future will look like it’s taken “AI will happen and we need to be there” as a primitive, and few will look like they’re asking themselves “Should AI happen?”. Because the latter question is just not a thought that interests them.