Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
Frankly, the very premise of this paper seems ridiculous to me, to a considerably greater extent than even most other bad alignment takes. How can the notion that agents may be getting more incoherent as they become more capable even exist within an industry that’s salivating over the prospect of climbing METR’s “maintain coherence over longer spans of time” benchmark?
Will Automating AI R&D not work for some reason, or will it not lead to vastly superhuman superintelligence within 2 years of “~100% automation” for some reason?
My current main guess is that it will more-or-less work, and then it will not lead to vastly superhuman superintelligence.
Specifically: I expect that the current LLM paradigm is sufficient to automate its own in-paradigm research, but that this paradigm is AGI-incomplete. Which means it’s possible to “skip to the end” by automating it to superhuman speeds, but what lies at its end won’t be AGI.
Like, much of the current paradigm is “make loss go down/reward go up by trying various recombinations of slight variations on a bunch of techniques, constructing RL environments, and doing not-that-deep math research”. That means the rewards are verifiable across the board, so there’s “no reason” why RLVR + something like AlphaEvolve won’t work for automating it. But it’s still possible that you can automate ~all of the AI research that’s currently happening at the frontier labs, and still fail to get to AGI.
(Though it’s possible that what lies at the end will be a powerful-enough non-AGI AI tool that it’ll make it very easy for the frontier labs to then use it to R&D an actual AGI, or take over the world, or whatever. This is a subtly different cluster of scenarios, though.)
Ryan had suggested that, on his model, spending ~5%-more-than-commercially-expedient resources on alignment might drop takeover risks down to 50%. I’m interested in how he thinks this scales: how much more resources, in percentage terms, would be needed to drop the risk to 20%, 10%, 1%?
Perhaps the claim is that such Python programs won’t be encountered due to relevant properties of the universe (ie, because the universe is understandable).
That’s indeed where some of the hope lies, yep!
An update here.
Following up on [1] and [2]...
So, I’ve had a “Claude Code moment” recently: I decided to build something on a lark, asked Opus to implement it, found that the prototype worked fine on the first try, then kept blindly asking for more and more features and was surprised to discover that it just kept working.
The “something” in question was a Python file editor which behaves as follows:
It parses the file, automatically finds all functions, classes, methods, and constants, and records/infers:
What span of lines of code they occupy.
For every function/method, what its call graph and reverse call graph are.
For every class, what its attributes are.
For every attribute, what entities read it and write to it.
It generates a navigable tree structure which contains the code elements arranged in a hierarchy, and adds subdirectories storing the aforementioned lists of what calls/reads/writes to a given element.
Clicking on an element opens an editable excerpt from the original document containing this element.
Clicking on a directory opens a set of editable excerpts corresponding to every element in that directory. This means you can immediately open e. g. all functions which call a given function, and review how it’s used + edit/refactor them. (Screenshot.)
You can create new custom directories, place various code elements there, and similarly open them all “on the same page”. This allows to flexibly organize code into different simultaneous organization schemas (rather than enforcing one file-based hierarchy) and minimizes the friction for inspecting/modifying a specific code category.
The remarkable thing isn’t really the functionality (to a large extent, this is just a wrapper on ast + QScintilla), but how little effort it took: <6 hours by wall-clock time to generate 4.3k lines of code, and I’ve never actually had to look at it, I just described the features I wanted and reported bugs to Opus. I’ve not verified the functionality comprehensively, but it basically works, I think.
How does that square with the frankly dismal performance I’ve been observing before? Is it perhaps because I skilled up at directing Opus, cracked the secret to it, and now I can indeed dramatically speed up my work?
No.
There was zero additional skill involved. I’d started doing it on a lark, so I’d disregarded all the previous lessons I’ve been learning and just directed Opus same as how I’ve been trying to do it at the start. And it Just Worked in a way it Just Didn’t Work before.
Which means the main predictor of how well Opus performs isn’t how well you’re using it/working with it, but what type of project you’re working on.
Meaning it’s very likely that the people for whom LLMs works exhilaratingly well are working on the kinds of projects LLMs happen to be very good at, and everyone for whom working with LLMs is a tooth-pulling exercise happen not to be working on these kinds of projects. Or, to reframe: if you need to code up something from the latter category, if it’s not a side-project you can take or leave, you’re screwed, no amount of skill on your part is going to make it easy. The issue is not that of your skill.
The obvious question is: what are the differences between those categories? I have some vague guesses. To get a second opinion, I placed the Python editor (“SpanEditor”) and the other project I’ve been working on (“Scaffold”) into the same directory, and asked Opus to run a comparative analysis regarding their technical difficulty and speculate about the skillset of someone who’d be very good at the first kind of project and bad at the second kind. (I’m told this is what peak automation looks like.)
Its conclusions seem sensible:
Scaffold is harder in terms of:
Low-level graphics programming (custom rendering, hit testing, animations)
State management complexity (graph + DB + UI consistency)
Architectural scope (command pattern, query language, serialization)
SpanEditor is harder in terms of:
Semantic code analysis (call graphs, attribute tracking)
Heuristic reasoning (call resolution without type info)
Multi-document coordination (independent editors, file reconstruction)
The fundamental difference: Scaffold builds infrastructure from primitives (graphics, commands, queries) while SpanEditor leverages existing infrastructure (Scintilla, AST) but must solve domain-specific semantic problems (code understanding).
[...]
Scaffold exhibits systems complexity—building infrastructure from primitives (graphics, commands, queries, serialization).
SpanEditor exhibits semantic complexity—leveraging existing infrastructure but solving domain-specific problems (understanding code without type information).
Both are well-architected. Which is “harder” depends on whether you value low-level systems programming or semantic/heuristic reasoning.
[...]
What SpanEditor-Style Work Requires
Semantic/Symbolic Reasoning
“What does this code mean?”
Classifying things into categories (is this a method call or attribute access?)
Working with abstract representations (ASTs, call graphs)
Heuristic Problem-Solving
Making educated guesses with incomplete information
Accepting “good enough” solutions (call resolution without type info)
Graceful degradation when analysis fails
Leveraging Existing Abstractions
Knowing which library does what
Trusting black boxes (Scintilla handles text rendering)
Composing pre-built components
Domain Modeling
Translating real-world concepts (code elements, categories) into data structures
Tree traversals and pattern matching
What Scaffold-Style Work Requires
Spatial/Geometric Reasoning
Coordinate systems, transformations, path geometry
“Where is this pixel relative to that rectangle?”
Visualizing shapes and their intersections
Temporal/State Machine Thinking
“What happens if the user drags while an animation is mid-flight?”
Sequencing operations correctly (shift then fade, not simultaneously)
Reasoning about system state at arbitrary points in time
Multi-Layer Consistency
Keeping UI, memory, and database in sync
Undo must reverse all three correctly
Invariants that span abstraction boundaries
Building from Primitives
No library does what you need; you must construct it
Understanding low-level APIs (QPainter, QGraphicsScene)
Resource lifecycle management (preventing C++ crashes)
The Cognitive Profile
Someone who excels at SpanEditor but struggles with Scaffold likely has these traits:
Strengths
| Trait | Manifestation |
| Strong verbal/symbolic reasoning | Comfortable with ASTs, grammars, semantic analysis |
| Good at classification | Naturally thinks “what kind of thing is this?” |
| Comfortable with ambiguity | Can write heuristics that work “most of the time” |
| Library-oriented thinking | First instinct: “what library solves this?” |
| Top-down decomposition | Breaks problems into conceptual categories |
Weaknesses
| Trait | Manifestation |
| Weak spatial reasoning | Struggles to visualize coordinate transformations |
| Difficulty with temporal interleaving | Gets confused when multiple state machines interact |
| Uncomfortable without guardrails | Anxious when there’s no library to lean on |
| Single-layer focus | Tends to think about one abstraction level at a time |
| Stateless mental model | Prefers pure functions; mutable state across time feels slippery |
Deeper Interpretation
They Think in Types, Not States
SpanEditor reasoning: “A CodeElement can be a function, method, or class. A CallInfo has a receiver and a name.”
Scaffold reasoning: “The window is currently in RESIZING_LEFT mode, the aura progress is 0.7, and there’s a pending animation callback.”
The SpanEditor developer asks “what is this?” The Scaffold developer asks “what is happening right now, and what happens next?”
They’re Comfortable with Semantic Ambiguity, Not Mechanical Ambiguity
SpanEditor: “We can’t know which class obj.method() refers to, so we’ll try all classes.” (Semantic uncertainty—they’re fine with this.)
Scaffold: “If the user releases the mouse during phase 1 of the animation, do we cancel phase 2 or let it complete?” (Mechanical uncertainty—this feels overwhelming.)
They Trust Abstractions More Than They Build Them
SpanEditor developer’s instinct: “Scintilla handles scrolling. I don’t need to know how.”
Scaffold requires: “I need to implement scrolling myself, which means tracking content height, visible height, scroll offset, thumb position, and wheel events.”
The SpanEditor developer is a consumer of well-designed abstractions. The Scaffold developer must create them.
tl;dr: “they think in types, not states”, “they’re anxious when there’s no library to lean on”, “they trust abstractions more than they build them”, and “tend to think about one abstraction level at a time”.
Or, what I would claim is a fine distillation: “bad at novel problem-solving and gears-level modeling”.
Now, it’s a bit suspicious how well this confirms my cached prejudices. A paranoiac, which I am, might suspect the following line of possibility: I’m sure it was transparent to Opus that it wrote both codebases (I didn’t tell it, but I didn’t bother removing its comments, and I’m sure it can recognize its writing style), so perhaps when I asked it to list the strengths and weaknesses of that hypothetical person, it just retrieved some cached “what LLMs are good vs. bad at” spiel from its pretraining. There are reasons not to think that, though:
I first asked for an “objective” difficulty assessment of the codebases, and then to infer strengths/weaknesses by iterating on this assessment. I don’t think “the user wants to confirm his prejudices, I should denigrate myself” is really inferrable from that initial task, and the follow-up was consistent with it.
By its own assessment (see the spoiler above), both codebases “are well-architected” (the difference was in how much effort ensuring the well-architecturedness required on my part). If it recognizes that it wrote both, and considers both well-written, why would it jump to “the user is asking what I’m bad at”? I suppose it could have noticed signs of external tampering in Scaffold...
I obviously provided it no additional context regarding myself or my experiences and beliefs, just the raw code + “run a comparative analysis” and “infer hypothetical strengths/weaknesses” prompts.
Overall… Well, make of that what you will.
The direction of my update, though, is once again in favor of LLMs being less capable than they sound like, and towards longer timelines.
Like, before this, there was a possibility that it really were a skill issue on my part, and one really could 10x their productivity with the right approach. But I’ve now observed that whether you get 0.8x’d or 10x’d depends on the project you’re working on and doesn’t depend on one’s skill level – and if so, well, this pretty much explains the cluster of “this 10x’d my productivity!” reports, no? We no longer need to entertain the “maybe there really is a trick to it” hypothesis to explain said reports.
Anyway, this is obviously rather sparse data, and I’ll keep trying to find ways to squeeze more performance out of LLMs. But, well, my short-term p(doom) has gone down some more.
I am interested in trying out the new code simplifier to see whether it can do a good job
Tried it out a couple times just now, it appears specialized for low-level, syntax-level rephrasings. It will inline functions and intermediate-variable computations that are only used once and try to distill if-else blocks into something more elegant, but it won’t even attempt doing things at a higher level. Was very eager to remove Claude’s own overly verbose/obvious comments, though. Very relatable.
Overall, it would be mildly useful in isolation, but I’m pretty sure you can get the same job done ten times faster using Haiku 4.5 or Composer-1 (Cursor’s own blazing-fast LLM).
Curious if you get a different experience.
Eh, I don’t think this really disproves the OP (who mentions bejeweled iPhones as well). This isn’t really a “$100k phone”, it’s a normal $1k phone with $99k of jewelry tastelessly bolted on. You’re not getting a 50,000 mAh battery, a gaming GPU, and Wi-Fi 12 plus 6G support in a 300 gram package here. Which, indeed: why not?
I’ve previously left a comment describing my being fairly unimpressed with Claude Code Opus 4.5, and leaving a few guesses regarding what causes the difference in people’s opinion regarding its usefulness. Eight more days into it, I have new comments and new guesses.
tl;dr: Very useful (or maybe I’m also deluding myself), very hard to use (or maybe I have skill issues). If you want to build something genuinely complicated with it, it’s probably worth it, but it will be an uphill battle against superhuman-speed codebase rot, and you will need significant technical expertise and/or the ability to learn that expertise quickly.
First, what I’m attempting is to use it to implement an app that’s not really all that complicated, but which is still pretty involved, still runs fairly nontrivial logic at the backend (along the lines of this comment), and about whose functionality I have precise desiderata and very few vibes.
Is Opus 4.5 helpful and significantly speeding me up? Yes, of that there is no doubt. Asking it questions about the packages to use, what tools they offer, and how I could architecture solutions to various problems I run into, is incredibly helpful. Its answers are so much more precise than Google’s, it distill information so much better than raw code documentation does, and it’s both orders of magnitude faster than StackOverflow and is smarter than the median answer you’d get there.
Is Claude Code helpful and speeding me up? That is more of an open question. Some loose thoughts:
Once I and Opus 4.5 have nailed down the architecture, you have to actually put it into code. If I’ve made the correct call regarding the scope, Claude Code can be mostly trusted with implementation as well.
What is “the correct call regarding the scope”? I’m still gauging this, but some pointers:
You can trust it to implement an algorithm that traverses a complicated but well-specified data structure and transforms it in some way.
You can trust it to implement the infrastructure for trivial data roundtrips through nontrivial architectures (e. g., taking a user action and propagating it across several nested concern boundaries).
You cannot trust it to do (1) and (2) together: to implement the data-roundtrip infrastructure if it requires implementing, rather than only navigating, some complicated logic.
Which is to say: the scope has to be pretty small. Smaller than you’d think from all the hype. No, even smaller than that.
At which point: do I actually get speed-ups from delegating the implementation to Claude, as opposed to interrogating the chatbot version of Opus 4.5 regarding how it can be implemented, then doing it manually (or with Cursor’s sometimes-helpful sometimes-frustrating autocomplete assist?).
On one hand, this does compromise learning. I don’t develop as precise a sense of how specific tools can be used/built as I would have if I were forced to think through the lower-level details manually. This is potentially bad in the very-long term.
On the other hand, this removes a fair amount of mental strain and frustration from me. The alternative to my babbling a feature to Claude may not be “I interrogate Opus 4.5 and write the code manually”, it may be “I go watch a movie, because I don’t have it in me this late in the day”. And, really, would it really be helpful for me to learn the minutia of how this one-off Python package I’ll never use again works? (Perhaps!)
Which is to say...
Over the short term, and for short-term projects, this definitely provides a speed-up.
Over the medium term, e. g. 4 weeks, or projects of similar scope? Dunno.
Over the long term, e. g. months/years, or projects of similar scope? Dunno.
Regarding bug-fixing:
It’s often advised to use test-driven development with LLMs, because it keeps them honest and is so easy. Just ask the LLM to write up the tests beforehand, glance over them to make sure they’re solid, and off you go! This is not good advice, in my experience. Waste of time and tokens. LLMs don’t write good tests, fixing those up would take longer than either doing the thing manually or having the LLM generate the feature raw then working out the kinks post-factum, etc.
(You can try asking the LLM to code up some domain-specific interface for you instead, so that you get a firehose of data on what you’re coding and on any bugs that transpire… But honestly, I wish someone would just vibe-code some non-terrible general-purpose call-graph debugging tools instead. Maybe I should try doing it myself.)
If the LLM-generated code has bugs in it, can you make them go away by describing the bug to the LLM and asking it to fix it? Sometimes. Often no, and often it takes longer than going looking for the bug directly.
A broad issue with LLMs is that their activity in the codebase is always… “entropy-increasing”. Almost every time they take action, they nudge some stuff out of place, add unnecessary features/functions, unnecessary comments, unnecessary guardrails for states the program can’t enter, or they implement something then change their mind but fail to clean the unused thing back up, etc. This happens even if you ask the LLM to go tighten things up/see if the codebase can be streamlined. This also means that if an LLM doesn’t one-shot the bug, if it tries A then B then C, that’s going to leave a trail of failed bug-fixes which may or may not be screwing something else up.
So if you want to build something nontrivially complex which can serve as a foundation for further complexity, you have to periodically go in and manually straighten things out.
(Sidenote: I did not know about code-simplifier prior to reading the OP. I’ll try it, I guess.)
The LLM merely lacking context is a significant factor sometimes:
It’s not really possible to convey your full intent to it, both because language is too low-bandwidth and because of context-window limitations.
It’s often unable to bug-fix efficiently just because it can’t see what’s going wrong. (I guess I can screenshot things, but I know Claudes don’t have great vision.)
What’s up with people who run twelve hundred simultaneous Claude sessions? Are they all LARPers?
Maybe.
Maybe not? After I’ve chewed up the project into bite-sized pieces, I do end up having to wait a while on Claude while it executes. If I had a better handle on what Claude can handle, and how to babysit it, and had the mental habit of breaking projects up into parallelizable instead of sequential implementation steps… If I’ve spent more than a week figuring this out… I can maybe see it?
Is it possible to “vibe-code”?
Not really, no.
The moment you lose track of what’s going on and start relying on trust and “vibes”, the moment you make the wrong call about the scope, it’s over for you, Claude runs off and spills a bunch of spaghetti all over the codebase. If that happens, you shouldn’t even try to fix it, just roll it back, chew the task up into finer pieces, and redo.
More broadly, if you don’t have the technical mindset to break the project down into small pieces that could be implemented and tested independently, and you can’t quickly get up to speed if you run into some novel technical/architectural problem that requires a thought-out technical solution, I don’t think you’re getting anywhere, no.
(You can attain the needed competence more rapidly than ever before via asking Opus 4.5 as well, but, like, you do need to attain that competence, at which point are you really “vibe-coding”?)
What’s up with the people who claim they can vibe-code, that their workflows look like this picture where they sit back and relax and Claude does everything? My guesses in increasing order of respectability:
They’re doing fake work, or are straight-up just attention-seeking AI influencers.
They’re deluding themselves and their projects are going to crash and burn.
They’re not doing anything actually nontrivially complicated, for certain values of “nontrivially complicated”. See the footnote in my previous comment.
See also this as a concrete example. Sure, it sounds impressive when the guy frames it as impressive, but I’m also getting the sense it’s just plugging a bunch of APIs into each other.
They’re drastically underestimating how much metis/know-how goes into their workflows, how much they really babysit LLMs, and how very far LLMs are from real autonomy.
Is Claude Code on track to replace software engineers? Ha! No. It does not have a “4-hour time horizon”. 20 minutes, maybe. Whatever METR’s graph is measuring, I currently don’t feel very daunted by the prospect of it continuing to go exponential. I may eat my words here, sure.
Is Rust a perfect language for agents? Haha, good one. I tried getting Opus to write Rust code (on its own suggestion), it was not good at it, like 10x slower and 3x buggier. I went back to Python. I’ll try checking later if it’s any good at translating from Python to Rust, once the Python code already exists… (And for fairness’ sake, I suppose I was using Opus, and he was perhaps talking about Codex.)
Is it fun to use Claude Code to code? No. Not for me. Keeping the project from degenerating into a non-functional mess is a constant uphill battle. I find it mentally taxing, in a different way to how writing code manually is mentally taxing.
To sum my current view up: Seems useful, but hard to use. You’ll have to fight it/the decay it spreads in its wake every step of the way, and making a misstep will give your codebase lethal cancer.
We’ll see how I feel about it in one more week, I suppose.
Seconded. Went from a skeptical “big if true” at the post title to rolling my eyes once I saw “iruletheworldmo”.
For reference, check out this leak by that guy from February 2025:
ok. i’m tired of holding back. some of labs are holding things back from you.
the acceleration curve is fucking vertical now. nobody’s talking about how we just compressed 200 years of scientific progress into six months. every lab hitting capability jumps that would’ve been sci-fi last quarter. we’re beyond mere benchmarks and into territory where intelligence is creating entirely new forms of intelligence.
watched a demo yesterday that casually solved protein folding while simultaneously developing metamaterials that shouldn’t be physically possible. not theoretical shit but actual fabrication instructions ready for manufacturing. the researchers presenting it looked shell shocked. some were laughing uncontrollably while others sat in stunned silence. there’s no roadmap for this level of cognitive explosion.
we’ve crossed into recursive intelligence territory and it’s no longer possible to predict second order effects. forget mars terraforming or fusion. those are already solved problems just waiting for implementation. the real story is the complete collapse of every barrier between conceivable and achievable. the gap between imagination and reality just vanished while everyone was arguing about risk frameworks. intelligence has broken free of all theoretical constraints and holy fuck nobody is ready for what happens next week. reality itself is now negotiable.
I guess it’s kind of entertainingly written.
It seems like, yes, he is saying that wealth levels get locked in by early investment choices, and then that it is ‘hard to justify’ high levels of ‘inequality’ and that even if you can make 10 million a year in real income in the post-abundance future Larry Page’s heirs owning galaxies is not okay.
I say, actually, yes that’s perfectly okay, provided there is stable political economy and we’ve solved the other concerns so you can enjoy that 10 million a year in peace.
I dunno about that. I think it is not okay for directionally the same reasons it wouldn’t be okay if we got an “infinitesimally aligned” paperclip maximizer who leaves the Solar System alone but paperclips the rest of the universe: astronomical waste.
Like, suppose 99% of the universe ends up split between twenty people, with them using it as they please, in ways that don’t generate much happiness for others. Arguably it’s not going to be that bad even in the “tech-oligarch capture” future (because Dario Amodei has made a pledge to donate 10% of his earnings or whatever[1]), but let’s use that to examine our intuitions.
One way to look at it is: this means the rest of civilization will only end up 1% as big as it could be. This argument may or may not feel motivating to you; I know “more people is better” is not a very visceral-feeling intuition.
Another way to look at it is: this means all the other people will only end up with 1% of the lifespan they could have had. Like, in the very long term, post-scarcity isn’t real, the universe’s resources are finite (as far as we currently know), and physical entities need to continuously consume those to keep living. If 99% of resources are captured by people who don’t care to share them, everyone else will end up succumbing to the heat death much faster than in the counterfactual.
This is isomorphic to “the rich have their soldiers take all timber and leave the poor to freeze to death in the winter”. The only reason it doesn’t feel the same way is because it’s hard to wrap your head around large numbers: surely you’d be okay with only living for 100 billion years, instead of 10 trillion years, right? In the here and now, both numbers just round up to “effectively forever”. But no, once you actually get to the point of you and all your loved ones dying of negentropic starvation 100 billion years in, it would feel just as infuriatingly unfair.
I understand the tactical moves of “we have to pretend it’s okay if the currently-powerful capture most of the value of the universe, so that they’re more amicable to listening to our arguments for AI safety and don’t get so scared of taxes they accelerate AI even further” and “we have to shut down the discussion of ‘but which monkey gets the banana?’ at every turn because it competes with the ‘the banana is poisoned’ messaging”. But if we’re keeping to Simulacrum Level 1, no, I do not in fact believe it’s okay.
I also don’t necessarily agree that those moves are pragmatically good. It’s mostly pointless to keep talking to AI-industry insiders; if we’re doing any rhetoric, it should focus on “outsiders”. And if it is in fact true that the current default trajectory of worlds in which the AGI labs’ plans succeed may lead to something like the above astronomical-waste scenarios, making those arguments to the general public is potentially a high-impact move. “Ban the AGI research because otherwise the rich will take all the stuff” is a much more memetically viral message than “ban the AGI because Terminator”.
(To be clear, I’m not arguing we should join various coalitions making false arguments to that end, e. g. the datacenter water thing. But if there are true arguments of that form, as I believe there are...)
I do not trust that guy to keep such non-binding promises, by the way. His track record isn’t good, what with “Anthropic won’t advance the AI frontier”.
I note that the wording in the more direct sources (rather than paraphrases) is “preventive war” and “bomb them”, which doesn’t actually strictly imply preventive nuclear bombings. It’s plausible that “bomb them” and “war with the USSR” could only mean “nuclear war” in-context… But it’d also be really funny if this is another “Eliezer advocates nuking foreign datacenters” situation.
I don’t think we necessarily disagree, here? I can imagine it being useful in this context, and I nod along at your “no loose leash” and “doesn’t feel like an employee, but a powerful tool” caveats.
What sort of codebase are you working on?
This specifically was an attempt to vibe-code from scratch a note-management app that’s to my tastes; think Obsidian/Logseq, broken down into small modules/steps.
As a simple example, I’ve had a “real project” benchmark for awhile to convert ~2000 lines of test cases from an old framework to a new one
My guess is that “convert this already-written code from this representation/framework/language/factorization to this other one” may be one of the things LLMs are decent at, yep! This is a direction I’m exploring right now: hack the codebase together using some language + factorization in which it’s easy (but perhaps unwise) to write, then try to use LLMs to “compile” it into a better format.
But this isn’t really “vibe-coding”/”describe the spec in natural language and watch the LLM implement it!”/”programming as a job is gone/dramatically transformed!”, the way it’s being advertised. LLMs are not, it seems, actually good at mapping natural-language descriptions into non-hack-y, robust background logic. You need a “code-level” prompt to specify the task precisely enough. And there’s only one way to bring that code into existence if LLMs can’t do it for you.
Language Models Offer Mundane Utility
Very preliminary opinion here, I’ve not yet spent enough time messing with it to be confident, but all these “Opus 4.5 in Claude Code can do anything!!!” experiences seem completely alien to mine. I can make Opus 4.5 sort of kind of implement not-entirely-trivial features if I do enough chewing-up and hand-holding and manual bug-reporting (its self-written tests are not sufficient). But it can’t autonomously code its way out of a wet paper bag.
And yes, I’ve been to Twitter, I’ve tried everything people’s been suggesting. We designed a detailed tech specification, a solid architecture, and a step-by-step implementation plan with it beforehand, and I asked it to do test-driven development and to liberally use AskUserQuestionTool at me. I’ve also tried to do the opposite, starting with a minimal “user-facing features” spec and letting it take the wheel. The frustrated tone of this comment wasn’t a factor either, I’ve been aiming to convey myself in a clear and polite manner.[1] None of that worked, I detect basically no change since August.
My current guess is that we have a massive case of this happening. All the people raving about CCO4.5 being an AGI with no limits happen to be using it on some narrow suite of tasks,[2] and everyone else just thinks they have skill issues, so they sit quiet.
Or maybe I indeed have skill issues. We’ll see, I suppose. I’ll keep trying to figure out how to use it/collaborate with it.
I expect there’s indeed some way to wring utility out of LLMs for serious coding projects. But I’m also guessing that most of this frippery:
agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations
– will not end up very useful for that task.
(People often say that AI progress feels slow/stalled-out only if you’re not interacting with frontier LLMs in a technical capacity, if you’re operating off of months-outdated beliefs and chatbot conversations. It’s been the opposite for me: every time I take some break from heavier coding and only update based on other people’s experiences with newer models, I get psyop’d into believing that there’s indeed been a massive leap in AI capabilities, I get concerned, and then I go back and I find my timelines growing to unprecedented lengths.)
I suppose I haven’t tried the opposite on that matter. Maybe you do need to yell at LLMs for them to start working?
E. g., maybe it’s:
Various very “shallow” simple scripts for one-off data transformations/batch processing.
Python ML libraries, which AGI labs have prioritized for obvious reasons.
HTML + CSS and other simple frontend stuff.
Specific science/math algorithms that are conceptually complex for a non-expert human, but are actually highly templated and memorizable.
Various templated APIs, e. g. web stuff.
Vibe-y conversations about architectures and implementations, with no actual code being pushed to production.
Linear combinations of the above.
Have you tried double-blinding yourself, Gwern-style? (I. e., prepare a box with glucose capsules and a box with some placebo (empty capsules + close your eyes when taking them out of the box?); at the start of the day, pick the box to take capsules from at random; at the end of the day, try to guess whether it was the placebo box; then check; repeat for several days.)
One obvious problem is that this turns getting funded into a popularity contest, which makes Goodhart kick in. It might work fine as a one-off thing, but in the long run, it will predictably get gamed, and will likely have negative effects on the whole LW discussion ecosystem by setting up perverse incentives for engaging with it (and, unless the list of eligible people is frozen forever, attracting new people who are only interested in promoting themselves to get money).
What should be the amount? Thiel gave 200k. Is it too much for 2 years? Too little?
You should almost certainly have some mechanism for deciding the amount to pay on a case-by-case basis, rather than having it be flat.
Could there be an entirely different approach to finding fellows? How would you do it?
What I would want to experiment with is using prediction markets to “amplify” the judgement of well-known people with unusually good AGI Ruin models who are otherwise too busy to review thousands of mostly-terrible-by-their-lights proposals (e. g., Eliezer or John Wentworth). Fund the top N proposals the market expects the “amplified individual” to consider most promising, subject to their veto.
This would be notably harder to game than a straightforward popularity contest, especially if the amplifee is high-percentile disagreeable (as my suggested picks are).
is phase 3 really a distinct phase?
Imagine we’re trying to launch a rocket to put a satellite into a stable orbit around Earth. At launch, the fuel explodes. However, due to the rocket’s geometry and good fortune, it happens to act as a shaped charge, and it launches the payload, mostly unharmed, off the planet and onto exactly the orbit we wanted, Project Orion-style. Technically, our “rocket” tool has succeeded at its task.[1] But it did so through a mechanism that’s completely unrelated to its intended tool-like functionality. It did not succeed due to its “rocket” capabilities; that abstraction broke down. The fuel-storage tank did not need to be shaped as a rocket at all.
By analogy: I think there’s a wide spread of relatively-high-probability trajectories on which a given reasoning model should be modeled as “legitimately reasoning” using the capabilities present in it. However, there’s a much bigger set of trajectories on which that abstraction breaks down, the obvious example being a CoT consisting of a sequence of random symbols. And it is possible that there’s a CoT like this such that, after e. g. o1-preview emits it and the “[/thinking]” token, it would next start to output a formally correct proof of the Riemann Hypothesis. However, that success won’t be caused by the “reasoning” mechanism, much like an exploding rocket successfully putting the satellite into orbit won’t be caused by the rocket’s rocket-like functionality. The DL model did not need to be shaped as a reasoning LLM at all; it could’ve been a random initialization.
If we’re using LLMs to do formal math, we could set up an “outcome pump” that harnesses this random-generation mechanism to produce valid proofs, by resampling trajectories. But the tool’s efficiency would be abysmal on all problems the LLM can’t solve using its “legitimate” capabilities.
You may argue the transition between Phase 2 and Phase 3 is continuous/gradual, with the model (or a human) trying ever-more-wild guesses until it’s just trying things completely randomly. That’s plausible. I would argue the phases are still distinct, however. (As in, the “transitional grey area” where we can’t firmly say whether this is reasoning or guessing is much smaller than the areas where the system is clearly reasoning or clearly being random.)
the widget Claude built has settings for whether the underlying relationship is linear, asymptotic, or power law and qualitatively the results are similar in all three cases
Noted!
But I still think this doesn’t quite get at what I think you’re pointing at when you talk about superexponential time-horizon growth/infinite time horizons. Probably not worth getting into at this time, though.
Disclaimer: I am not sure this is actually physically possible without further abstraction-breaking in the form of thermodynamic miracles.
I think there are some mismatches in the fundamental assumptions your model and METR’s model make.
Your model assumes that any given AI’s ability to solve tasks continues to grow at the same rate when you give it more inference-time compute.[1] METR’s model, however, doesn’t make time-budget assumptions for the AI.[2] It tracks a binary “can/can’t solve this task” variable, assumes that if an AI can’t solve the problem within some subjective time-frame, it can’t solve it at all. And indeed: when comparing AIs and humans, we don’t currently talk about “AIs are way too time-inefficient at this task compared to humans”, we talk about “AIs can’t do this one at all”.
And I think it’s indeed the correct model for the current AIs. Roughly speaking, if we’re modeling , I think this should be a “three-phase” function:
Phase 1: intuition. What tasks can LLMs solve given zero/epsilon inference-time compute? We can model that as the intercept .
Phase 2: reasoning. The tasks LLMs can solve given some inference-time compute. On this interval, grows with compute. But empirically, it seems to saturate for any given frozen-weights LLM. Past some amount of inference-time compute, this phase transforms into...
Phase 3: monkey-on-a-typewriter. There’s of course a nonzero probability that any LLM will output the solution to any problem given to it. Over enough iterations/attempts (perhaps over time, with e. g. a sliding context window), the probability approaches 1. However, grows awfully slow here, and in a certain sense, this no longer tracks capabilities actually “present” in a model. The abstraction of “AI intelligently solving a problem” breaks down in this phase. Thus, this phase can be approximated as a constant function.
From how you’ve discussed the topic previously, your background assumption is that Phase 3 will eventually disappear: that the reasoning-phase interval will eventually grow to infinity. I agree it’s a salient possibility to track.
However, the current models do seem to have this phase, see e. g. various research on “overthinking”. Thus, modeling them as merely biphasic is invalid.
Stepping back: I think your prediction of superexponential growth isn’t really about “superexponential” growth, but instead of there being an outright discontinuity where the time-horizons go from a finite value to infinity. I guess this is “superexponential” in a certain loose sense, but not in the same sense as is superexponential.
I don’t think this can be modeled via extrapolating straight lines on graphs / quantitative models of empirically observed external behavior / “on-paradigm” analyses.
You say “AIs have ‘infinite horizon length’ now” about the slope-point where the crossover point disappears, but note that your model assumes infinite horizons even before this point. You model it as , but of course for any above-zero slope, including subhuman slopes, there’s a time-budget value at which the AI achieves arbitrarily high performance.
Aside: Indeed, that’s a common misunderstanding of METR’s models. I don’t think you’re making this misunderstanding there, but maybe something in that direction.
How would more data and software live in reality? What would the Great Generalized Google Maps project include, in what order? What data structures would it use for more abstract things?
Per above, we’d need tighter feedback loops/quicker updates, appropriate markings of when content/procedures become outdated, some ability to compare various elements of constructed realities against the ground truth whenever the ground truth becomes known, etc. (Consider if Google Maps were updating very slowly, and also had some layer on top of its object-level observations whose representations relied on chains of inferences from the ground-true data but without quick ground-true feedback. That’d gradually migrate it to a fictional world as well.)
The general-purpose solution is probably some system that’d incentivize people to flag divergences from reality… Prediction markets?
I agree that there are ways to define the “capabilities”/”intelligence” of a system where increasing them won’t necessarily increase its long-term coherence. Primarily: scaling its ability to solve problems across all domains except the domain of decomposing new unsolved problems into combinations of solved problems. I. e., not teaching it (certain kinds of?) “agency skills”. The resultant entity would have an abysmal time horizon (in a certain sense), but it can be made vastly capable, including vastly more capable than most people at most tasks. However, it would by definition be unable to solve new problems, not even those within its deductive closure.
Inasmuch as a system can produce solutions to new problems by deductive/inductive chains, however, it would need to be able to maintain coherence across time (or, rather, across inferential distances, for which time/context lengths are a proxy). And that’s precisely what the AI industry is eager to make LLMs do, what it often measures capabilities in.
(I think the above kind of checks out with the distinction you gesture at? Maybe not.)
So yes, there are some notions of “intelligence” and “scaling intelligence” that aren’t equivalent to some notions of “coherence” and “scaling coherence”. But I would claim it’s a moot point, because at this point, the AI industry explicitly wants the kind of intelligence that is equivalent to long-term coherence.