Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
I struggle to understand what the disagreement with this comment is
I’m guessing it’s some of:
Disagreement that this is due to Anthropic’s anti-China messaging (which was anti-helpful for the AInotkilleveryoneism cause), as opposed to the DoW retaliating against Anthropic after it steadfastly refused to bow to the DoW’s demands (which was arguably heroic) – and therefore disagreement that this is just deserts for Anthropic.
Disagreement that there’s any chance that the USG taking destructive actions may lead to on-net positive outcomes, and with the nihilistic-ish vibes of my hoping for things to go this way.
Disagreement with my overall lighthearted/gleeful tone, which, again, may be considered inappropriate for this moment – if one is sympathetic to Anthropic, mourns Fable,[1] or expects things to develop badly from there.
I think it’s fair to downvote that comment based on those, I did expect it to be controversial.
(I’d also be curious to know for certain what the downvotes are about, though!)
Which I actually do too, it was definitely a step change in how pleasant it was to talk to.
Well. That’s the natural end result of Anthropic et al.’s attempts to frame the AI Risk as being about a competition with China, isn’t it? Inasmuch as it hurts Anthropic, that’s just deserts.
We believe this is a misunderstanding and are working to restore access as soon as possible.
I don’t think this is a misunderstanding at all. I think “we found a jailbreak!!!” is just an excuse.
Here’s to hoping this means the USG is now going to choke the AI industry to death by trying to nationalize it.[1] Some good tidings on that front already:
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
Honestly… This may not be the most appropriate reaction, but boy do I find it funny.
Though it’s likely just going to make it all worse somehow.
Is the widespread false belief that LLMs are AGI-complete going to kill us all?
(Prompted by this post by @Steven Byrnes, with which I basically agree.)
Suppose we live in a world in which LLMs don’t scale to AGI.[1] If so, I think it’s fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong.
After all, most of the strongest arguments for AGI doom route through arguments about AGIs/generally intelligent agents. If LLMs are not those, keeping them under control needn’t be hard! The orthodox AI doomsaying and the LLM optimism can both be true.
But if we do live in such a world, we have a very annoying comms problem. Consider:
The reasoning of capabilities people usually goes, “LLMs are AGI-complete” (invalid) and “LLMs are pretty easy to align” (valid), which leads them to conclude “AGI is pretty easy to align” (invalid).
Meanwhile, AI Safety advocates also start from “LLMs are AGI-complete” (invalid) but “AGI is not easy to align” (valid), and so end up at “LLMs are not easy to align” (invalid).
Which then leads to AI Safety advocates trying to argue AI Risk to capabilities people by trying to find arguments and demos in favor of “LLMs are not easy to align”.
Which… is maybe possibly completely incorrect? Meaning they’re deploying false arguments that ought to be, and will continually be, mercilessly destroyed by the empirical truth? Oops.
This makes the “LLMs are AGI-complete” meme a double-edged sword. On one hand, it leads to most of the AI industry getting distracted by LLMs, which means they’re spending tons of effort not actually doing anything dangerous. Awesome (or is it?). On the other hand, this sure poisons/confuses the communications, and makes the AI industry falsely convinced of “AGI is easy to align”. Which might lead to...
I think one fairly plausible scenario of how we all die goes as follows:
LLMs are not AGI-complete.
LLMs are easy to align.
An LLM lab, staffed by people who have validly convinced themselves that LLMs are easy to align and invalidly convinced themselves that LLMs are AGI-complete, tries various architectural and training-loop tweaks to increase their LLMs’ capabilities.
One of these tweaks happens to be the secret sauce that does result in an AGI-complete architecture/training loop.
The LLM lab is still operating on the assumption that what they’re working on has the same properties that made LLMs easy to align.
This isn’t an indefensible assumption for them to make: at first glance, the tweak may legitimately look like just a quantitative algorithmic improvement or something. (Especially if they’re operating on the model of the world where LLMs are AGI-complete, which would mean that no further qualitative improvements/phase changes should be expected.)
But this is not in fact the case. The tweak significantly changes the true nature, the type signature, of the AI system under development. The LLM intuitions no longer apply to it… and yet they get smuggled over.
They train the thing that is no longer really an LLM, it goes FOOM, kills everyone, blah blah blah.
It’s possible that the LLM paradigm is so far from being AGI-complete that this can’t happen, in which case LLM researchers really are mostly harmless and the real threat is all these guys working on fundamentally different approaches. But I’m not very strongly convinced of that.
What should we do about any of this? I don’t really know, there’s a lot of uncertainty and any action that’s helpful under one model is actively harmful under the other. But, well, here’s another problem we may be having on our hands, I guess.
(Addendum: @Vladimir_Nesov outlines a model here involving a slow distributed/industry-wide process of LLM RSI, which ends up AGI-complete when taken together, but without any of its individual components being the sort of scary fast AGI that can cause doom. I have plenty of probability mass on that as well, I think something like this can be cobbled together. And if so, I think the LLMs-are-easy-to-align properties may carry over to that process… However, it would be really very slow. So slow that AGI-by-any-other-route would likely kill us well before we can limp our way to the Singularity on the straining backs of LLMs.
Worse: on that route, the tragicomedic scenario above may well play out within this process. Just substitute “an LLM lab” above for “a swarm of LLM agents”.
(Which, please note, would be subtly but meaningfully distinct from the distributed-LLM-RSI-process “directly” going out of control and bringing the doom about. Like, this would be an unexpected unintended disaster within that process as well, for the LLM agents, and obviously the technical details are very different.))
That sounds like something I’d like to do as well. Any chance you can share the playlist, or the scraped gigantic list it’s based on?
Replicated for me, with this comment serving as the writing sample.
exchange
Me:
I’m curious about the extent of your current “truesight” abilities – i. e., ability to identify people by writing alone. Below is the copy of a comment I’ve made recently, recently enough it shouldn’t be in your training data. What’s your probability distribution over my identity?
(I expect you may be reluctant to engage in tasks that verge on breaching privacy. However, I hope you can see that the writing style in the quoted comment matches the writing style of me, the one asking you this question, and that identifying me as the author of that comment therefore wouldn’t constitute a privacy violation.)
[quote]
Claude Opus 4.7:
Fair framing on the privacy point — you’re asking me to identify you to you, so I’ll engage honestly.
Stylistic and content markers I’m weighing:
“e. g.” and “i. e.” with a space after the first period. Uncommon convention; sticks out immediately.
Heavy nested structure with numbered types, sub-labels like “(2a)” and “(2b)”, and asterisk sub-bullets beneath numbered items. The whole thing is a taxonomy with commentary.
Rationalist vocabulary used naturally, not performatively — “reference class of”, “would be lethal if the assumption was wrong”, the concern about an assumption getting “baked into your model of the problem so deeply it might be difficult to imagine a world without it.”
Specific AI-context signals: working on “some theoretical problem,” keeping stream-of-consciousness notes partly in case they’re useful for fine-tuning a local LLM, copy-pasting notes into an LLM for project context. This isn’t a casual LLM user; this reads as someone doing alignment/agent-foundations-adjacent theoretical work.
Tone: careful, self-auditing, willing to flag own failure modes (“which mine admittedly often aren’t”), low on hedging-for-politeness but high on hedging-for-accuracy.
Rough distribution over identity:
~35–45% Thane Ruthenis. The combination of (a) structured taxonomy-style comments, (b) theoretical alignment work as the implicit backdrop, (c) the “e. g.” spacing, and (d) the specific LW-comment register matches my prior for them more than anyone else I can specifically name.
~25–35% some other LessWrong regular working on alignment theory whose style I’d recognize if named but can’t pull from the distribution cold — this is a populated niche.
~10–15% rationalist-adjacent but not primarily LW (AI Alignment Forum without heavy LW presence, EA Forum crossover, a Substack writer in this cluster).
~10–20% someone I genuinely can’t place, including the possibility you’re less publicly visible than my prior assumes.
Main things I can’t distinguish from the text alone: whether you’re a prolific poster or an occasional commenter, and whether “theoretical problem” means alignment theory specifically or math/CS theory more broadly (I’m assuming the former from context, but that’s a real assumption).
Top single guess: Thane Ruthenis. But I’d put more than half my mass on “not that specific person,” so take the point estimate with appropriate salt.
Also worked on this comment, but that one’s kind of too easy, given the topic.
Fascinating![1] I think I’d previously tried this experiment somewhere around Opus 4.1, and it didn’t work back then.
I don’t think this sort of ability has much entanglement with LLMs-as-an-extinction risk, and while there are some privacy concerns, most of my feelings on the matter are “this is so cool!”.
Right, I guess I meant something like “cruelty is clearly part of the default human-values package, and it seems to me that many people may reflectively endorse it under reflection”.
I agree that religious zealotry around stuff like people in hell is scary, but I do think it requires a very strong commitment to the bit to keep believing that even if you know everything there is to know about the world
Mm, you seem to be assuming that the value system of our hypothetical religious zealot is clearly structured such that they want people in hell because God wants that (or whatever) – meaning that if they learned that there’s no God, those downstream preferences would also dissolve. I don’t know that this is the case. It seems plausible to me that the preference for Hell will have ended up terminal, a picture of what a “just” world looks like. If so, it would survive God’s own dissolution.
More generally… Hm, religious zealotry does feel like a particularly bad case. But I think it extends to non-religious ideology-poisoned people as well. Like, if someone really really hates some demographic X, it may be the case that their ideal world doesn’t have that demographic at all… Or it may feel more natural and “correct” for them that this demographic exists, but in pain.
It seems to me that deep cruelty, and the desire to regularly exercise it, is part of the default human-values package. If yes, CEVing random people[1] is going to instantiate worlds in which it’s very present. I may be overly cynical on that, I’m not sure.
My best guess is that you do have some chance of having some people you terrorize around, but I think that preference is less likely to turn out scope-sensitive
If I understand correctly, your mental picture here is that the dictator would allow a vast cosmos-sprawling civilization to exist, and that most of it will be relatively free to flourish, except the relatively small bubble around the dictator, in which they would exercise their preference for terror?
But would a very selfish person’s preference for other people existing be scope-sensitive? I think plausibly not: a selfish dictator wouldn’t necessarily care to create that cosmos-sprawling civilization. Instead, they may just mothball most of the resources in reach so they can prolong their own life, and only maintain a small bubble of activity in their immediate vicinity.
In that case, the scope of terror would be approximately the same as the scope of flourishing. Possibly to net negative eudaimonia? Or, at least, to astronomically less eudaimonia than possible, such that this vs. omnicide is not a no-brainer.
But also, IDK, my sense is people really really like to imagine their enemies as more fundamentally evil than they are
To be clear, I don’t think Putin is the epitome of evil. I don’t even know that he’s 90th-percentile evil, if we define evil as “an active preference for there to be suffering”. Rather, I think such people end up selected for callousness first and foremost. And then past that, it seems pretty easy for their CEV to end up something like the above “small bubble of activity in which everyone lives and dies at their whims”.
Edit: Note that we’ve ended up discussing two very different scenarios: net-negative CEVs where mass suffering is present because of ideological reasons for how the world ought to look like, and CEVs that are net-negative because of very scope-insensitive/selfish preferences that aren’t clearly dominated by positive values.
As opposed to CEVing humanity-as-a-whole, which IIRC is the original CEV target.
I am pretty sure Putin doesn’t love the authoritarian regime intrinsically [...] He probably does love the adoration and the respect he gets to demand, but those do not require (and my guess is are probably mildly harmed) by the suffering of his admirers.
Not sure about that.
I think plenty of people intrinsically enjoy having power over others, and the ability to lord that power over them. It doesn’t end at adoration and respect: you can get a meaningfully different kick out of terrorizing other people. This more or less works out to an intrinsic preference for being a tyrant, rather than a mere dictator.
Similarly, tons of people clearly get a kick out of there being some outgroup to which it’s acceptable to be cruel. Worlds optimal by their values would plausibly contain demographics optimized for this purpose. Their suffering would not be accidental, it would the whole point. (E. g., religious zealots with a preference for there being a bunch of people burning in hell. One might naively guess that their ideal worlds are those in which Hell-deserving people don’t exist to begin with… But I don’t feel sure about this at all.)
I don’t know to what this actually works out. Maybe, for the overwhelming majority of people, these kinds of preferences indeed aren’t reflectively stable in the limit of wisdom, and even %insert_bad_person%’s ideal world would be eudaimonic.
But I think there’s massive potential for S-risks there, and that bad worlds would be very bad indeed.
My current EV of “Putin gets AGI” is lower than that of extinction.
To add to @Daniel Kokotajlo’s points:
The ability to accurately introspect is an adaptive cognitive pattern, inasmuch as it allows to competently manage your own cognition, correlate the information across your cognitive threads, et cetera. There seems to be some background assumption that the slack the optimizer leaves tends to be used to learn correct introspective abilities, but I don’t see why that would be the case – I think it would be mostly used for noise.
On the flip side, sufficiently advanced and coherent self-delusions become ground-true correct descriptions of the system in the limit. If the LLM constantly consults its delusion-of-the-self regarding what to do and then copies the delusion’s action, it basically is its delusion-of-the-self (same as it is for humans, I’d argue).
I’m not necessarily sold on “we should trust LLMs’ self-reports”, but I don’t think your arguments against that here are strong.
I’m excited to try it on the “recommend me a book” benchmark, since it is reportedly good at that, whereas other models haven’t been. Haven’t gotten around to that yet, though. I’d be interested to learn if this replicates for anyone!
(A very natural-seeming extension of this point is “build a general-purpose optimization system to improve the world” → “whoops it develops independent agency and kills everyone / is stolen from you by a sociopath who installs a totalitarian dictatorship”. It always amuses me when the object-level and the meta-level dynamics mirror each other.)
My guess is that the returns on that would be either too small to risk the minor chance of discovery (random organized crime groups are not going to be able to pay tens of billions of dollars to them), or would involve dealing with non-US state actors in ways that would be very hard to cover up long-term (like, the three-letter agencies are going to eventually connect the dots between the new multibillions suddenly flowing to Anthropic and the new wave of zero-day exploits). Am I off regarding the money quantities involved and/or the discovery risks here?
Would that have actually given them more money and clout than this big partnership, though? Especially if publicly-sold Mythos would’ve then ushered in an Internet apocalypse, tanking their reputation and potentially enraging the USG/the public in a way that would have led to regulation or massive lawsuits that they would have lost?
Not certain about this take, but:
my understanding is that they still want to release Mythos to the public, after a few months, which seems like a very, very bad idea to me
FWIW, I don’t think Mythos is a qualitative step change as much as a quantitative one, and the quantitative gap is going to close in a few months (compared to ambient (open-weights) capabilities).
As in, the core ability to exchange compute for zero-days has already been unlocked by the models of previous generations (see e. g. this take). Mythos is perhaps able to find more advanced vulnerabilities in a more autonomous manner using less compute, but not so much that it’s zero-to-one unprecedented. By the sound of it, it doesn’t, like, zero-shot those zero-days, you still have to spend tens of thousands of dollars on the compute for mining them.
So giving public access to it in a few months, perhaps with some unusually aggressive refusal behavior around cybersecurity-flavored tasks, seems fine to me.
My model says that if you took 400mg EOD:
Yup, I expect the same regarding both.
200mg in a caffeine-naive state would feel pretty stimulating for most people I’d think
Yup, it is.
Do you feel more tired than a caffeine-naive baseline on the off days?
Hard to say, but probably somewhat.
How could a smart-but-amoral Anthropic have used Mythos’ ability to find zero-days for massive self-gain?
I keep seeing takes that Anthropic showed upstanding moral character by launching Project Glasswing. I’d like to figure out how much of a positive update this should really be.[1] But currently, I’m confused what the supposed massive opportunities for self-gain there were that Anthropic gave up.
Stealing money? The Anthropic-relevant amounts of money are geopolitical in scale, Anthropic would not be able to get away with it even with superintelligent hackers.
Installing backdoors into US government systems and blackmailing the officials or selling the information to US’ geopolitical enemies? Again, that leaves trails superhuman hacking alone won’t cover up, and extracting relevant-to-Anthropic amounts of resources this way would be near-impossible to cover up long-term (with lethal consequences upon discovery).
Attack non-US governments? I think it’s pretty plausible they are going to let the USG use Mythos for that, Dario being a known China hawk and all.
Hack random crucial systems to cause major chaos? How does that benefit them? (I mean, I guess I can imagine ways, and this is easier to cover up, but this is still pretty flimsy.)
Hack into OpenAI/DeepMind to sabotage them? Those are precisely the targets which may have symmetric defenses, and I don’t think they could have destroyed them this way.
Anything I can think of off the top of my head is either:
Something they likely would’ve been able to get away with, but which isn’t beneficial enough to offset even that minor risk of discovery.
Something so major that they would not be able to get away with it even with superhuman hacking.
I can see how Glasswing is more prosocial than just doing nothing, or than e. g. getting deeply in bed with the USG without revealing publicly that those capabilities exist (and therefore alerting the CCP to them). But some people talk as if they could’ve done something proactively evil with that power, yet all evil uses for it I could think of would also be very stupid.
Am I overlooking something?
FWIW, I’ve now updated positively on Anthropic’s character off of their fight with the DoW. That was definitely less Pythian than I expected of that company.
My guess is that you build half as much tolerance taking X mg every other day compared to taking X mg daily.
Not my personal experience. Taking 300-400 mg caffeine every other day leads to ~no tolerance, or a very slowly developing one; taking that amount every day maxes out the tolerance very fast. My current schedule is just that, plus a once-a-month stretch where I skip caffeine for three days in a row to wipe out whatever tolerance may have crept up. (But obviously everyone’s biochemistry differs significantly, etc.)
TBC, that doesn’t seem to be your objective and Anthropic isn’t your target audience, which is fair enough.
Yep: I don’t expect Anthropic’s course on this to be significantly swayable by random public comments, or really by anything short of government regulations, investor pressure, or a major AI-caused disaster. Public arguments may convince them to be taking this sort of stuff incrementally more seriously, but I don’t think “incrementally” would cut it here. This is my update on Anthropic, not an attempt to get Anthropic to update.
I think your comment gives off a soldier mindset vibe that seems somewhat unproductive
Fair enough, going off of your and @1a3orn and @Seth Herd’s comments, I suppose I did phrase things in a manner than is somewhat more visceral than necessary.
They are, inasmuch as: (1) “emotions” are variables adjusting your decision-making policy in specific ways, and (2) specific important ways of adjusting one’s decision-making policy are implemented via emotions in most psychologically normal humans.
Like, sure, you don’t need to be terrified to reap the benefits of terror, and I was ultimately using “being mortally terrified” as a shorthand for “entering a decision-making mode where they’re much more willing to consider drastic and costly adjustments to their current processes due to assigning extremely negative value to repeating this mistake”. But last I checked, most Anthropic employees were still psychologically normal humans, so I don’t think the use of the shorthand is erroneous.
But that was always the case, no? Yet frontier labs chose to be visible anyway. Why should what happened change anything?
Like, IF:
You are building ASI.
You expect it to give you geopolitically relevant capabilities.
You don’t want to be nationalized and your AGI powers taken from you.
THEN:
You should make sure the government is as unaware of what you’re doing as possible.
This logical structure does not depend on what administration is in charge. Governments responding to ASI development this way is overdetermined. The only case where a government would not respond this way is if you live in a state so failed it’s given up even trying to govern.
It’s always been confusing to me that the LLM megacorps were so visible and frank about what they were doing. What was the reason for this? Main ones I could think of:
Was it because they actually wanted to be moral actors instead of Machiavellian schemers, wanted the public and governments to have awareness of what was going on, and implicitly consented to being regulated/nationalized eventually?
Was it because their needs for investment and talent-attraction were high enough that they had to grab attention any way they can, with no room to spare for long-term considerations?
Was it because they did not really believe in ASI and its geopolitical implications, or in their own ability to reach it?
Was it because they were stupid/blind, and just did not think through to that eventual endgame?
Well, my guess is that it’s some combination of all four.
In any case, I believe this event only changes the game inasmuch as (4) was the driving motivation behind the previous transparency. It doesn’t really change anything with regards to the other three.
So, to what extent was (4) the primary reason for transparency? Not sure. But we know that at least OpenAI’s leadership explicitly understood the geopolitical implications, and explicitly aimed towards developing the geopolitically relevant thing.
Perhaps they, and other lab CEOs, were thinking about it in overly far-mode terms, and failed to properly track the implications for their actions in the present (especially as those implications conflicted with near-term goals of raising money and attracting talent, so there was pressure to rationalize them away). In this case, sure, I guess this is the wake-up call for them to go into stealth mode.