ARC-AGI-v1 was fairly simple static visual problems. ARC-AGI-v3 involves playing a simple set of video games. So it’s more like the former skills were a prerequisite for latter challenges to even make sense.
I do feel like we’ve seen a lot of broadening of intelligence, though: they can write poetry, cold read humans, operate a desktop computer using a mouse, solve long-standing math problems, etc. all without too much explicit training.
They don’t learn to solve “ARC-AGI-v1” puzzles, they learn to solve “the entire category that problem represents” and I’d personally lean towards there only being a few dozen such categories.
If we saw narrower learning, where “writing” didn’t include poetry, or “art” couldn’t mimic specific styles, I’d agree with a thousand islands. Although I won’t be surprised if we do see a few islands outside the major continents of learning.
Eigenbraid
I’m not really focused on “fair”—just understanding what these models are capable of.
I assume at some point there will be a model that can trivially beat experienced humans at Pokemon PVP, or your invented video game, without needing guides or hundreds of hours. I’m mostly trying to get a sense of how close we are to that, today.That said, I’d also be impressed by an LLM that is smart enough to look up a strategy guide unprompted, especially if it can apply that information well enough to place at the top of leader boards.
I don’t think this is entirely true. ARC-AGI has been going for years, is under a huge amount of optimization pressure, and is still one of the domains where LLM gets beaten by a 12 year old.
If it turns out we can optimize any trait we want with a year of attention, that’s pretty important information in and of itself. It means ASI will probably be able to fix any remaining gaps in the jagged frontier once it hits a certain skill in self-improvement.
Choosing brute force isn’t a limitation, agreed. But being incapable of other strategies is!
It’s less a judgment about the tactic and more that it eliminates a lot of decisions in a fight (swapping out, type-type matchups, etc.). It doesn’t test “intelligence” as deeply as building a balanced team.My understanding is LLMs still struggle to put together a good PVP team, where those elements can’t be ignored.
Very much agreed, but I want to add one other avenue:
We can focus on writing policy proposals that address the current climate. There are a lot of concrete steps we can take, today, that both lay the groundwork for “Plan A” and address more mundane concerns.
“Addressing mundane concerns” is probably going to be a critical compromise in building political influence. If we focus entirely on the risks of ASI, we alienate everyone who thinks that is Science Fiction. Publicly focusing on more mundane concerns let us skip around the “Sci-Fi” problem entirely.
(We should still have all the Sci-Fi policies and discussions, of course—I’m just saying that an ad about “stopping terrorists” is going to be a lot more influential than one about “stopping ASI”. As it happens, a lot of the proposed polices do plenty of useful things for all sorts of mundane concerns)
I really appreciate this response.
I still don’t quite get “concentration of power”:
The proposal is 2-3 months of exclusive access, focused on evaluating the tool and using it to update security. Due to the centralized nature of LLMs, the labs can review how these groups actually use the tool—it would be hard to sneak an unrelated project in.
First, that doesn’t seem like a huge advantage to me.
Second, historically, this seems like the normal course of events? Individuals couldn’t afford to build factories or railroads. We seem to have done just fine despite that. Technologies are always going to give the most benefit to the rich, because they have the resources to leverage them.
Third, by nature of being private companies, the labs have always been able to form private partnerships like this. If they really wanted to, they could have been “picking winners and losers” for a while. So far, the incentive seems to be that selling to the public is much, much more profitable, so I’m not yet concerned.
Now, I’m not saying this is the IDEAL course of events, but… I don’t get why so many people place such a high priority on “dispersal / decentralization” right now, in 2026, given current capabilities. I tend to think that, except for the government intervention, the Mythos release was handled pretty ideally, and a public release in February or April, without any time for the ecosystem to prepare, would have been much worse.
(I do actually remember when “cyber” was a clear sign of a starry-eyed company that was going to die in the DotCom bubble, but after so many decades I’ve just given up and tried to shift with whatever language “the kids these days” are using.)
And, to be clear, this is just Step 1, today—let’s start ensuring that future models don’t cause a Mythos-tier security shitstorm; let’s start finding out which companies have a good track record at evaluating models; let’s start tracking how long it takes to go from “Glasswing” to “Public”.
These all give us barometers for when things get more dangerous. If Mythos 2 doesn’t cause another spike in security bugs, it can probably get shoved out the door ASAP. Conversely, if it takes twice as long for companies to secure their systems the second time around, then Mythos 3 is more likely to be even more of a serious security risk and we might want to start pushing harder on other regulations.
Quite bluntly, have you actually read the proposal and compared it against our current situation? Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by… looping in trusted partners to double-check their safety results?
I am suggesting a short period of early release access. We have already seen from the Mythos release that this is actually sufficient to patch critical software and ensure the release is less publicly disruptive. I have no clue how a few months of early access results in a serious “imbalance of control”—at worst, you can trivially fix this by rotating what groups get early access, neh?
Equally, if you’re worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than “the entire population of Earth”? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
We have somehow survived the problem of vetting who the “good guys” are despite the attackers incentives for centuries. It’s the basic concept of security clearances.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets.
That is part of what I am suggesting, yes. Conversely, “withdrawing it entirely” gets you absolutely zero new information, so I’m not sure how that’s possibly a fair comparison.
Perhaps I failed to communicate something clearly, but this all reads like a ridiculous strawman of my actual proposal, which is simply “we should have independent organizations involved in mundane auditing, so that more people are aware of capabilities before they become public, and we can involve the relevant security teams”
P.S. Cyber has been a common professional term for decades so maybe save the linguistic snobbery for your own personal blog? I don’t see what purpose is served by spelling out your personal distaste with one particular word I used, and I struggle to read it as anything other than an insult when you say it “reeks of clueless”. You could have been polite, or just cut the footnote entirely.Edited to Add: Okay, I reviewed my post and comment and I don’t mention “bioterrorism” anyway so I’m now extremely convinced you’ve confused my post for someone else’s—why are you writing an entire footnote about something I never brought up?
Do you think that this proposal would somehow make the problem worse? We already have concentration of power today.
Sorry, one last comment/question:
I’m really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn’t seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn’t show up?How surprised will you be if this specific technology doesn’t pan out?
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
I don’t think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viabilityObjection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.
Hi, I’m Eigenbraid. I read the sequences back in the LessWrong 1.0 era. Now that LLMs are becoming a big thing, I want to get back into this community—I still think of it as the “OG” place for discussions of AI. I’m also on Twitter (x.com/eigenbraid).
I’m super-curious about these weird new minds we’ve created: I want to get to know them, and what they are right now. I’m also fascinated and alarmed by the potential for AGI/ASI sometime in my life time.
It feels like there’s a bunch of scattered communities that have each figured out part of the AI puzzle and I’m trying to pull together a complete picture for myself. The cyborgism community (x.com/repligate/ and such) takes a very different approach from LessWrong, for instance. I’d love to hear about other clusters out there on the cutting-edge of the discussion.
“The monthly general Open and Welcome thread”
This should probably link to https://www.lesswrong.com/w/open-threads?sortedBy=new
(currently it links to https://www.lesswrong.com/w/open-threads)
(Sorry if this sort of nit is too pedantic 😅)
Are you looking for what people want (“like”), or what people expect?
The main productive value will be if evolution can produce new capabilities, and/or implement existing capabilities with far fewer resources.
Probably also a lot more insight into alignment, and a lot more clumsy failed attempts that we can study.
I don’t think either is “good” per-se, but it might force society to take the whole situation more seriously.
My main concern is that it puts a very sharp floor on how much we can slow down. It would take global coordination to rebuild the internet into something safe from self-replicators, and my odds on that sort of cooperation are still fairly low. US-China is a lot easier than “every nation that has substantial data centers / servers”