ARC-AGI-v1 was fairly simple static visual problems. ARC-AGI-v3 involves playing a simple set of video games. So it’s more like the former skills were a prerequisite for latter challenges to even make sense.
I do feel like we’ve seen a lot of broadening of intelligence, though: they can write poetry, cold read humans, operate a desktop computer using a mouse, solve long-standing math problems, etc. all without too much explicit training.
They don’t learn to solve “ARC-AGI-v1” puzzles, they learn to solve “the entire category that problem represents” and I’d personally lean towards there only being a few dozen such categories.
If we saw narrower learning, where “writing” didn’t include poetry, or “art” couldn’t mimic specific styles, I’d agree with a thousand islands. Although I won’t be surprised if we do see a few islands outside the major continents of learning.
Eigenbraid
The Next Ecology
(3/3) The Dangers of ASI
(2/3) The Dangers of AGI
(1/3) The Dangers of LLMs
I’m not really focused on “fair”—just understanding what these models are capable of.
I assume at some point there will be a model that can trivially beat experienced humans at Pokemon PVP, or your invented video game, without needing guides or hundreds of hours. I’m mostly trying to get a sense of how close we are to that, today.That said, I’d also be impressed by an LLM that is smart enough to look up a strategy guide unprompted, especially if it can apply that information well enough to place at the top of leader boards.
I don’t think this is entirely true. ARC-AGI has been going for years, is under a huge amount of optimization pressure, and is still one of the domains where LLM gets beaten by a 12 year old.
If it turns out we can optimize any trait we want with a year of attention, that’s pretty important information in and of itself. It means ASI will probably be able to fix any remaining gaps in the jagged frontier once it hits a certain skill in self-improvement.
Choosing brute force isn’t a limitation, agreed. But being incapable of other strategies is!
It’s less a judgment about the tactic and more that it eliminates a lot of decisions in a fight (swapping out, type-type matchups, etc.). It doesn’t test “intelligence” as deeply as building a balanced team.My understanding is LLMs still struggle to put together a good PVP team, where those elements can’t be ignored.
Current Limitations of LLMs
Very much agreed, but I want to add one other avenue:
We can focus on writing policy proposals that address the current climate. There are a lot of concrete steps we can take, today, that both lay the groundwork for “Plan A” and address more mundane concerns.
“Addressing mundane concerns” is probably going to be a critical compromise in building political influence. If we focus entirely on the risks of ASI, we alienate everyone who thinks that is Science Fiction. Publicly focusing on more mundane concerns let us skip around the “Sci-Fi” problem entirely.
(We should still have all the Sci-Fi policies and discussions, of course—I’m just saying that an ad about “stopping terrorists” is going to be a lot more influential than one about “stopping ASI”. As it happens, a lot of the proposed polices do plenty of useful things for all sorts of mundane concerns)
I really appreciate this response.
I still don’t quite get “concentration of power”:
The proposal is 2-3 months of exclusive access, focused on evaluating the tool and using it to update security. Due to the centralized nature of LLMs, the labs can review how these groups actually use the tool—it would be hard to sneak an unrelated project in.
First, that doesn’t seem like a huge advantage to me.
Second, historically, this seems like the normal course of events? Individuals couldn’t afford to build factories or railroads. We seem to have done just fine despite that. Technologies are always going to give the most benefit to the rich, because they have the resources to leverage them.
Third, by nature of being private companies, the labs have always been able to form private partnerships like this. If they really wanted to, they could have been “picking winners and losers” for a while. So far, the incentive seems to be that selling to the public is much, much more profitable, so I’m not yet concerned.
Now, I’m not saying this is the IDEAL course of events, but… I don’t get why so many people place such a high priority on “dispersal / decentralization” right now, in 2026, given current capabilities. I tend to think that, except for the government intervention, the Mythos release was handled pretty ideally, and a public release in February or April, without any time for the ecosystem to prepare, would have been much worse.
(I do actually remember when “cyber” was a clear sign of a starry-eyed company that was going to die in the DotCom bubble, but after so many decades I’ve just given up and tried to shift with whatever language “the kids these days” are using.)
And, to be clear, this is just Step 1, today—let’s start ensuring that future models don’t cause a Mythos-tier security shitstorm; let’s start finding out which companies have a good track record at evaluating models; let’s start tracking how long it takes to go from “Glasswing” to “Public”.
These all give us barometers for when things get more dangerous. If Mythos 2 doesn’t cause another spike in security bugs, it can probably get shoved out the door ASAP. Conversely, if it takes twice as long for companies to secure their systems the second time around, then Mythos 3 is more likely to be even more of a serious security risk and we might want to start pushing harder on other regulations.
Quite bluntly, have you actually read the proposal and compared it against our current situation? Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by… looping in trusted partners to double-check their safety results?
I am suggesting a short period of early release access. We have already seen from the Mythos release that this is actually sufficient to patch critical software and ensure the release is less publicly disruptive. I have no clue how a few months of early access results in a serious “imbalance of control”—at worst, you can trivially fix this by rotating what groups get early access, neh?
Equally, if you’re worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than “the entire population of Earth”? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
We have somehow survived the problem of vetting who the “good guys” are despite the attackers incentives for centuries. It’s the basic concept of security clearances.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets.
That is part of what I am suggesting, yes. Conversely, “withdrawing it entirely” gets you absolutely zero new information, so I’m not sure how that’s possibly a fair comparison.
Perhaps I failed to communicate something clearly, but this all reads like a ridiculous strawman of my actual proposal, which is simply “we should have independent organizations involved in mundane auditing, so that more people are aware of capabilities before they become public, and we can involve the relevant security teams”
P.S. Cyber has been a common professional term for decades so maybe save the linguistic snobbery for your own personal blog? I don’t see what purpose is served by spelling out your personal distaste with one particular word I used, and I struggle to read it as anything other than an insult when you say it “reeks of clueless”. You could have been polite, or just cut the footnote entirely.Edited to Add: Okay, I reviewed my post and comment and I don’t mention “bioterrorism” anyway so I’m now extremely convinced you’ve confused my post for someone else’s—why are you writing an entire footnote about something I never brought up?
Do you think that this proposal would somehow make the problem worse? We already have concentration of power today.
Proposal: The Glasswing Standard
The LLM Revolution (so far)
Sorry, one last comment/question:
I’m really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn’t seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn’t show up?How surprised will you be if this specific technology doesn’t pan out?
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
Are you looking for what people want (“like”), or what people expect?
The main productive value will be if evolution can produce new capabilities, and/or implement existing capabilities with far fewer resources.
Probably also a lot more insight into alignment, and a lot more clumsy failed attempts that we can study.
I don’t think either is “good” per-se, but it might force society to take the whole situation more seriously.
My main concern is that it puts a very sharp floor on how much we can slow down. It would take global coordination to rebuild the internet into something safe from self-replicators, and my odds on that sort of cooperation are still fairly low. US-China is a lot easier than “every nation that has substantial data centers / servers”