Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
I’m excited to try it on the “recommend me a book” benchmark, since it is reportedly good at that, whereas other models haven’t been. Haven’t gotten around to that yet, though. I’d be interested to learn if this replicates for anyone!
(A very natural-seeming extension of this point is “build a general-purpose optimization system to improve the world” → “whoops it develops independent agency and kills everyone / is stolen from you by a sociopath who installs a totalitarian dictatorship”. It always amuses me when the object-level and the meta-level dynamics mirror each other.)
My guess is that the returns on that would be either too small to risk the minor chance of discovery (random organized crime groups are not going to be able to pay tens of billions of dollars to them), or would involve dealing with non-US state actors in ways that would be very hard to cover up long-term (like, the three-letter agencies are going to eventually connect the dots between the new multibillions suddenly flowing to Anthropic and the new wave of zero-day exploits). Am I off regarding the money quantities involved and/or the discovery risks here?
Would that have actually given them more money and clout than this big partnership, though? Especially if publicly-sold Mythos would’ve then ushered in an Internet apocalypse, tanking their reputation and potentially enraging the USG/the public in a way that would have led to regulation or massive lawsuits that they would have lost?
Not certain about this take, but:
my understanding is that they still want to release Mythos to the public, after a few months, which seems like a very, very bad idea to me
FWIW, I don’t think Mythos is a qualitative step change as much as a quantitative one, and the quantitative gap is going to close in a few months (compared to ambient (open-weights) capabilities).
As in, the core ability to exchange compute for zero-days has already been unlocked by the models of previous generations (see e. g. this take). Mythos is perhaps able to find more advanced vulnerabilities in a more autonomous manner using less compute, but not so much that it’s zero-to-one unprecedented. By the sound of it, it doesn’t, like, zero-shot those zero-days, you still have to spend tens of thousands of dollars on the compute for mining them.
So giving public access to it in a few months, perhaps with some unusually aggressive refusal behavior around cybersecurity-flavored tasks, seems fine to me.
My model says that if you took 400mg EOD:
Yup, I expect the same regarding both.
200mg in a caffeine-naive state would feel pretty stimulating for most people I’d think
Yup, it is.
Do you feel more tired than a caffeine-naive baseline on the off days?
Hard to say, but probably somewhat.
How could a smart-but-amoral Anthropic have used Mythos’ ability to find zero-days for massive self-gain?
I keep seeing takes that Anthropic showed upstanding moral character by launching Project Glasswing. I’d like to figure out how much of a positive update this should really be.[1] But currently, I’m confused what the supposed massive opportunities for self-gain there were that Anthropic gave up.
Stealing money? The Anthropic-relevant amounts of money are geopolitical in scale, Anthropic would not be able to get away with it even with superintelligent hackers.
Installing backdoors into US government systems and blackmailing the officials or selling the information to US’ geopolitical enemies? Again, that leaves trails superhuman hacking alone won’t cover up, and extracting relevant-to-Anthropic amounts of resources this way would be near-impossible to cover up long-term (with lethal consequences upon discovery).
Attack non-US governments? I think it’s pretty plausible they are going to let the USG use Mythos for that, Dario being a known China hawk and all.
Hack random crucial systems to cause major chaos? How does that benefit them? (I mean, I guess I can imagine ways, and this is easier to cover up, but this is still pretty flimsy.)
Hack into OpenAI/DeepMind to sabotage them? Those are precisely the targets which may have symmetric defenses, and I don’t think they could have destroyed them this way.
Anything I can think of off the top of my head is either:
Something they likely would’ve been able to get away with, but which isn’t beneficial enough to offset even that minor risk of discovery.
Something so major that they would not be able to get away with it even with superhuman hacking.
I can see how Glasswing is more prosocial than just doing nothing, or than e. g. getting deeply in bed with the USG without revealing publicly that those capabilities exist (and therefore alerting the CCP to them). But some people talk as if they could’ve done something proactively evil with that power, yet all evil uses for it I could think of would also be very stupid.
Am I overlooking something?
FWIW, I’ve now updated positively on Anthropic’s character off of their fight with the DoW. That was definitely less Pythian than I expected of that company.
My guess is that you build half as much tolerance taking X mg every other day compared to taking X mg daily.
Not my personal experience. Taking 300-400 mg caffeine every other day leads to ~no tolerance, or a very slowly developing one; taking that amount every day maxes out the tolerance very fast. My current schedule is just that, plus a once-a-month stretch where I skip caffeine for three days in a row to wipe out whatever tolerance may have crept up. (But obviously everyone’s biochemistry differs significantly, etc.)
TBC, that doesn’t seem to be your objective and Anthropic isn’t your target audience, which is fair enough.
Yep: I don’t expect Anthropic’s course on this to be significantly swayable by random public comments, or really by anything short of government regulations, investor pressure, or a major AI-caused disaster. Public arguments may convince them to be taking this sort of stuff incrementally more seriously, but I don’t think “incrementally” would cut it here. This is my update on Anthropic, not an attempt to get Anthropic to update.
I think your comment gives off a soldier mindset vibe that seems somewhat unproductive
Fair enough, going off of your and @1a3orn and @Seth Herd’s comments, I suppose I did phrase things in a manner than is somewhat more visceral than necessary.
They are, inasmuch as: (1) “emotions” are variables adjusting your decision-making policy in specific ways, and (2) specific important ways of adjusting one’s decision-making policy are implemented via emotions in most psychologically normal humans.
Like, sure, you don’t need to be terrified to reap the benefits of terror, and I was ultimately using “being mortally terrified” as a shorthand for “entering a decision-making mode where they’re much more willing to consider drastic and costly adjustments to their current processes due to assigning extremely negative value to repeating this mistake”. But last I checked, most Anthropic employees were still psychologically normal humans, so I don’t think the use of the shorthand is erroneous.
Regarding Claude Mythos’ CoTs being accidentally trained-on: I think the biggest problem here is that Anthropic’s internal procedures were shoddy enough that this “technical error” was allowed to happen, and then went unnoticed until the model was already trained.
Regardless of the extent to which it’s justified, Anthropic sure seems to believe that CoT monitoring and faithfulness is one of the main pillars of ensuring AI alignment. Now it turns out that their training pipelines were consistently sabotaging that pillar. If this mistake were allowed to happen, how many other mistakes of the same magnitude are their procedures ridden with? How many more such mistakes will they make in the future? How many of them will be present, uncaught, in the training run that produces their god?
The appropriate response to realizing you made a mistake like this is to be stricken with so much mortal terror that you rehaul your entire R&D pipeline until it’s structurally impossible for anything in this reference class to ever happen again.
Is there any indication Anthropic is doing that? I haven’t seen all Twitter discussions, and I suppose they may not want to be public about it… But vibes-wise, it doesn’t seem that they’re appropriately horrified.
And if not, I argue they’re not taking any of this seriously. None of this fancy “AI alignment” crap is going to matter if your ineptitude lives at the level of “can’t even implement your own plan correctly”. Just about same as, “whoops, I accidentally put a ‘-’ in front of my AI’s utility function”.
Humans (especially the most power-seeking humans) can have terrible goals with respect to what happens around them, but it’s pretty rare for them to have strong preferences about what happens in other galaxies and at high levels of abstraction
Seems obviously wrong to me. Most people have political beliefs regarding how societies “ought” to be put together and how people “ought” to live their lives, and they can be quite ugly. I agree that an idealized power-seeker doesn’t have such preferences...
… but why would an idealized power-seeker allow humans to live free in distant parts of the cosmos? If I were an idealized power-seeker, I’d send self-replicating probes there to convert all reachable matter into the most energy-conserving formats, to ensure I would be able to stave off the heat death for myself for as long as physics allows. I would most definitely not populate it with irrelevant monkeys to waste however they see fit.
Inasmuch as the hypothetical AGI dictator cares about humans existing, the dictator’s preferences must be aligned with human flourishing lest hell results.
Inasmuch as the hypothetical AGI dictator doesn’t care about humans existing, they would not exist.
It’s unclear if Mythos is much more impactful for cybersecurity overall than a new fuzzing or static analysis tool
The bull case here is that “scale LLMs” is turning out to be a way to predictably and consistently produce ever-better tools for discovering exploits, right? Probably with said tools’ power scaling exponentially (in some relevant sense), like everything else with LLMs.
That is, Mythos by itself is probably just on the level of a new fuzzing tool, able to let humans find a new reference class of exploits. But then we’d have Mythos 2 three to six months later, etc. Which potentially shifts the cybersecurity world into a new operating regime, even if each individual perturbation is something that already happened before.
Or is there an argument that it would still be on-model for how the cybersecurity world operates? I’m not very familiar.
Regarding Claude Mythos/Project Glasswing:
For what it’s worth, this is in line with my expectations for what LLMs should be capable of and it doesn’t update me much towards their AGI-completeness. (Though it’s at the more bullish end of past!me’s predictions.) Receipts-wise, I’m not sure I laid it out publicly anywhere, but here’s an excerpt from a message I’d sent to @habryka in January 2025:
[T]he worse end of the scenarios I’m imagining is...
The o-series of models potentially doesn’t scale to superhuman programmers-in-general, but does scale to superhuman hackers: because “find a way to make this abstraction leak” is precisely the kind of programming task that is well-posed/isomorphic to a rigorous-math task (“find a flaw in this proof/this statement of tautology”).
I expect that ~all of the components of the current web stack contain countless vulnerabilities that are relatively easy to discover, and which aren’t exploited simply because finding them would require a talented programmer to parse vast mountains of code (application code, and the code of its dependencies, and the dependencies of dependencies, and any cross-interactions between them...).
AI agents of the above type would remove this friction cost.
So if this scenario comes to pass, I expect them to be discovering Heartbleeds and Log4Shells on a daily or weekly basis. Then, at any given time, they would be using that to simultaneously attack large random subsets of the entirety of the Internet infrastructure.
I do, however, still find myself unpleasantly surprised by this. I wouldn’t say it happened “earlier than I hoped”, inasmuch as I did not have specific predictions about the timing… But, well, any time for it to happen would be unpleasantly early, I suppose.
Would we be less likely to go extinct if OpenAI forcibly took over Anthropic? (a possible hypothetical: a combined effort of OpenAI and the US government forces a merger). My take: no.
People keep making arguments shaped like this, and I feel that there’s some error here.
Currently, for race dynamics to dissolve, two LLM megacorps need to cease to be relevant. If Anthropic weren’t founded or were destroyed/absorbed, only one LLM megacorp would need to lose relevance. Destroying one megacorp is easier than destroying two. Therefore, in worlds in which Anthropic didn’t exist or were destroyed, it would be easier to dissolve the race dynamics altogether. Therefore, it’s bad that Anthropic exists and it would be better if it did not exist.
That seems straightforward to me? We are less likely to go extinct in Anthropic-less worlds because those worlds are more likely to also become OpenAI-less and/or DeepMind-less.
There seems to be some underlying assumption that this is a one-turn game, where you’re asked to choose between “implement Intervention X and then do nothing forever” and “do nothing forever”. I’m not sure why “do nothing forever” is implicitly tacked on.
(Arguments like “I, a capability researcher, don’t need to quit my AGI lab, because I will just be replaced by an identical capability researcher, for a net zero change in anything” seem from the same genre.)
On balance, I support banning/obstructing datacenter buildout. That said, I’m not actually sure whether that impacts the omnicide risk positively or negatively.
I don’t think the LLM paradigm is AGI-complete. I don’t have utter confidence in that, but I think it’s more likely than not. And in worlds in which it is non-AGI-complete, the labs’ obsession with them is very helpful. They’re wasting macroeconomic amounts of money on it, pouring most of our generation’s AI-researcher talent into it, distracting funding from other AGI approaches. If LLMs then fail to usher in the Singularity, if it does turn out to be a bubble that pops (e. g., in 2030-2032, when the ability to aggressively scale compute is supposed to run out), this should cause another AI winter. AGI would become a decidedly unsexy thing to work on once more, in industry and probably in academia both.
What would obstructing the LLM paradigm (via various compute limits) do? Well, it may cause the AGI megacorps to start looking into other directions now, while they still have massive amount of unspent manpower and capital. Which may lead to someone “succeeding” sooner.[1]
Perhaps one shouldn’t interrupt their enemy while they’re making a mistake.
Or perhaps that’s not how the story goes. Perhaps: LLMs are a strong signal that AI is a massively powerful technology, a signal legible to many more people than theoretical arguments. They’re attracting macroeconomic amounts of funding to AI, funneling a large fraction of our generation’s talent towards working on AI, fueling inter-company and geopolitical AI-race dynamics. And while they cause other AGI approaches to be relatively neglected, they cause so much more absolute attention to be pointed at the AI industry that the amounts of funding/talent going into non-LLM AGI routes is still much greater than in the no-LLMs counterfactual. On top of that, even if LLMs aren’t AGI-complete in themselves, they may still be useful enough to speed up SWE/research along other AGI paradigms as well.
And perhaps effectively banning LLMs would cause a chilling effect much greater than if they were merely a research bet that didn’t work out. It would be a signal that AGI research is something the world does not want to see in any form, turning funding and talent away from the whole endeavor.
I’m not really sure which model is correct. Which AI Winter would be colder and broader: one caused by a proactive LLM ban, or one caused by LLMs’ dramatic betrayal of investor hopes? Do LLMs currently cause net increase or decrease of attention to non-LLM AGI research? Can non-AGI LLMs significantly speed up non-LLM AGI research if allowed to continue scaling? And, of course: is the premise that LLMs are non-AGI-complete even correct?
I’m on balance in favor of obstructing LLMs and upper-bounding the available compute. “We must not get in the way of the current paradigm because they are totally making a mistake so all this investment is actually delaying AGI!” has the flavor of some galaxy-brained 4D chess nonsense. But I’m not fully sure.
(Also, worlds in which politicians/the public are concerned about AI enough to ban datacenter buildout is of course a more promising world than the one in which they’re not. This is true regardless of whether the direct effects of banning datacenter construction are positive or negative. In such worlds, they’d probably be willing to engage in additional anti-AGI interventions. So perhaps datacenter-impeding is worth supporting just to put us in those worlds, shift the Overton window that way? Not sure whether it’s causal or evidential, though.)
There are also arguments that this would be bad even in the LLMs-are-AGI-complete worlds, because – the argument would go – while LLMs may be nontrivial to align, they’re easier to align than some paradigms that may replace them. I don’t put much stock into this argument, it usually relies on assuming the LLM masks/personas is the entity of interest that we need to align. I don’t put zero stock in it, though, I guess.
While it seems true that promoting symmetric enmity pretty much always comes with a big cost
I’m not sure about that at all. Suppose that:
Group A is doing something that’s going to lead to harmful consequences for Group B.
Group A mistakenly believes the consequences would be positive instead.
Group B would be more easily roused to action if it were fueled by enmity towards Group A, rather than by tone-neutral disagreement.
Then promoting even symmetric enmity between the groups would be beneficial for Group B.
(Incidentally, note that even if neither party is actively malicious towards the other, “promoting enmity” wouldn’t necessarily involve lying. Feelings of enmity can be fueled by pointing at Group A’s negligence (or whatever flaws are causing them to mistakenly pursue the harmful-to-B activity), rather than by ascribing malicious intent to them. To a significant extent, it’s about framing.)
OpenAI, Anthropic, and DeepMind have a massive lead on all other players in the race.[1] If Anthropic is destroyed and OpenAI bleeds talent and is internally sabotaged by those who remain until it’s far behind DeepMind, that would (a) on-expectation nontrivially increase the time to AGI,[2] (b) reduce competitive pressures on DeepMind, which would mean they may go slower/pause as well (which Demis at least claims to want).
The less well leading AGI labs operate, the better.
If Anthropic is lost in the conflict with Pentagon, then the main rivals are GDM, whose Gemini 3 Pro was a vast intelligence with no spine and a likely sociopathic wireheader and xAI who doesn’t care about safety at all
I think this doesn’t matter, frontier LLMs are all roughly equally aligned (not). And in the unlikely scenario where personas are the AGI-complete entities, such that Anthropic/OpenAI’s alignment plans can succeed, DeepMind can probably figure out in time how to control them as well.
Likely a considerably bigger lead than it looks like, inasmuch as other players are fast followers who’d be lost without the big three guiding them.
Imagine you’re organizing a literal race. You get the list of racers + their racing history, figure out who the top 3 best runners are, then randomly blacklist two of them. That would necessarily make the expected length of the subsequent race longer.
To add to @Daniel Kokotajlo’s points:
The ability to accurately introspect is an adaptive cognitive pattern, inasmuch as it allows to competently manage your own cognition, correlate the information across your cognitive threads, et cetera. There seems to be some background assumption that the slack the optimizer leaves tends to be used to learn correct introspective abilities, but I don’t see why that would be the case – I think it would be mostly used for noise.
On the flip side, sufficiently advanced and coherent self-delusions become ground-true correct descriptions of the system in the limit. If the LLM constantly consults its delusion-of-the-self regarding what to do and then copies the delusion’s action, it basically is its delusion-of-the-self (same as it is for humans, I’d argue).
I’m not necessarily sold on “we should trust LLMs’ self-reports”, but I don’t think your arguments against that here are strong.