For my own future reference, here are some “benchmarks” (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical. (The “serious” version would probably start with the Evals section of technicalities’ 2025 shallow review of technical AIS, or SemiAnalysis’ in-house view.)
the AI village and blog, not really a “benchmark” per se but my richest source of intuitions about current frontier models’ capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other “benchmark-equiv” frontier models
altruism-related, e.g. the AI village’s raise as much money for charity as you can. Confoundingly, the much more capable agents a year after raised only 25% of the amount earlier-gen models raised ($510 from 17 human donors for Doctors Without Borders vs $1,984 in year 1 for Helen Keller International and Malaria Consortium)
update: the AI Digest folks concluded it’s because “humans are less interested”
Chats on read.haus with AI simulations of prominent authors become preferable to reading the latter’s real content. Scott Alexander, Sarah Constantin, Spencer Greenberg, Byrne Hobart, Tyler Cowen, Dwarkesh Patel, Andy Matuschak etc are all on there but they never come across quite right to me
Starburst, fictional theoretical physics. I don’t really get their leaderboard though
a small set of work-related spreadsheet modelling problems I keep thinking current agents should easily do but they keep failing in very irritating ways, Claude Code included. I’m waiting for agents that will finally speed me up not slow me down on these. Possibly skill issue on my part
FWIW, Anthropic’s members of technical staff estimates of productivity boost: currently 1.15-1.4x with Sonnet 4.5 for most, except that one person at 2x as “their workflow was now mainly focused on managing multiple agents”, wonder if it’s the same person Sholto Douglas mentioned worked with 9 agents at the same time
update: section 7.3.4 of the Opus 4.5 system card says 2-3x productivity boost vs Sonnet 4.5′s 1.15-1.4x above: “Nine of 18 participants reported ≥100% productivity improvements, with a median estimate of 100% and a mean estimate of 220%”
update: section 2.3.6 of the Mythos system card says ~4x productivity boost geomean with wide distribution, but: “productivity uplift on individual tasks does not translate one-for-one into acceleration of research progress. Compute is also a key ingredient, as promising ideas need to be de-risked at scale. Our best estimates of the elasticity of progress to researcher output, combined with the observed uplift, yield an overall progress multiplier below 2×. We estimate that reaching 2× on overall progress via this channel would require uplift roughly an order of magnitude larger than what we observe”
their slope on the chart below exceeds that of humans (I’m not a fan of the notion of task horizon length, it bakes in perf plateauing that doesn’t happen to humans thinking longer, hence slope)
great summary of reasons time horizon is overrated/misinterpreted by Thomas Kwa
math-related:
FrontierMath Tier 4 because I like math x AI, plus commentary like Kevin Buzzard’s “I was amused this week to have been sent data on what happens if you ask lots of agents to try and solve these problems and you mark the question as being solved if at least one agent gets the answer correct at least once”
more generally in math x AI: Gavin Leech’s examples thread of “crucially useful AI in research maths”
the “Erdos problems benchmark”: the proportion of currently outstanding Erdos problems amenable to current AI tools operated with minimal human intervention (inspired by Terry Tao, more below)
vibe-proving math theorems in Lean except it doesn’t take a week and isn’t “extremely annoying” (despite Adam Mastroianni’s argument that what a dream job really feels like is to be perpetually annoyed). The main issue is in verifying that the human proof-to-Lean code translation is faithful, which doesn’t seem automatable
Epoch’s Capabilities Index because it’s general (composite metric of most of the high-profile benchmarks out there) stitched together using a methodology that seems intuitively correct (item response theory), although admittedly as someone who started out believing anything is measurable if you try hard enough I’ve gradually grown disillusioned enough to down-weight even ostensibly good composite benchmarks like ECI a fair bit. Also CAIS’s definition of AGI
Scale’s Remote Labor Index because I work remotely. 230 projects from Upwork freelancers “excluding projects requiring physical labor, long-term evaluation, or direct client interaction”, mean and median human completion time 29 and 11.5 hours respectively, mean and median project value $630 and $200. Manus at 2.50% tops the leaderboard, then Sonnet 4.5 > GPT-5 > ChatGPT Agent > Gemini 2.5 Pro last at 0.83%, which matches my impression of their relative “IRL competence” in the AI Village
As of mid-March 2026, Claude Opus 4.6 (Cowork) got it up to 4.17%
Visakan Veerasamy’s threadthulhu gets tamed: “No mention of threading culture is complete without a hat-tip to Visakan Veeraswamy, (@visakanv) of course. Visa took the basic linear threading idea pioneered by Marc and turned it into a dizzying artform, turning his account into a tangled, densely interlinked, quote-linked, promiscuously forking Lovecraftian monstrosity of a twitter hyperobject. I came up with a term for it: threadthulhu (my main contribution to culture through the twitter years was coming up with names for things). My own threadthulu was only middling crazy. Orderly enough that I was able to index all my good threads in a meta-thread over the years, and slaughter it relatively cleanly to create the raw material for this book. I doubt Visa’s insane threadthulu can be killed at all, let alone properly butchered into a book-like echo like this one. I vibecoded the pipeline that created this book, but it will probably take AGI to similarly tame Visa’s threadthulu”
Pokemon FireRed: Fable beat it in just over 50 hours with a vision-only harness. Julian: “That’s an hour faster than a heavily-harnessed GPT 5.5 beat FireRed, and >6x faster than the 325 hours it took a lightly-harnessed Opus 4.7 to beat Pokémon Red. (For comparison, an average human would take about 30 hours for FireRed and 26 hours for Red.)” Started from how well is Claude playing Pokemon? in March 2025
“Their blind spot is cards that seem OK at a glance, but won’t make sense when reviewed months later. When I write cards, I simulate seeing them in future sessions, based on years of SRS review experience. LLMs don’t have that taste, and we couldn’t figure out how to infuse it”
A guy I worked with a while back, much smarter than me, once spent an entire afternoon refusing to merge a PR that, on paper, was correct. The change worked. The tests passed. CI was green. He couldn’t articulate what was wrong. He kept saying “I just don’t believe this code.”
Eventually he asked the author to walk him through the reasoning, line by line, out loud. Maybe forty minutes in, the author said something offhand: “well, this assumes the queue is FIFO, but I think that’s safe.” It wasn’t safe. The queue was FIFO in development and best-effort-FIFO in production, and the difference was buried in a runbook nobody had looked at in two years. My colleague had smelled it from the diff. He couldn’t explain why up front, but luckily his persistence (and frankly his good reputation as someone one should listen to when it comes to software engineering) made him too hard to dismiss. He just “didn’t believe the code”.
That afternoon cost a few hours of two engineers’ time. It probably saved a months-long incident.
That kind of work is impossible for an AI. It’s also impossible to prompt an AI into doing, because the input my colleague used — a constellation of subtle features, a year-long history of similar bugs, a half-conscious memory of “where pain has come from before” — isn’t anywhere it can be fed in. It lived in him. He’d built it the slow way, over a decade.
This is what the AI-coding pitch never confronts. Yes, generate code faster, sure. But the catching — the eye that says no — comes from somewhere AI cannot reach.
I’m not sure how to operationalise a bet against Ekrem’s claim that “this kind of work is impossible for an AI”, but I’d like to; this smells like Flying Machines Which Do Not Fly, the NYT editorial predicting
[It] might be assumed that the flying machine which will really fly might be evolved by the combined and continuous efforts of mathematicians and mechanicians in from one million to ten million years...
69 days before the Wright brothers did their thing at Kitty Hawk. And I say this as someone who thinks tacit knowledge is very important.
Coding, math, whatever. Can LLMs predict the outcomes of physical experiments?
Suppose I pour 8 oz (226.8 g) of boiling water into a ceramic coffee mug that weighs 1.25 lb (0.57 kg). The ambient air is still and 20 degrees Celsius. The cup starts at room temperature. Give me an equation for the temperature of the water in Celsius over time. The only free variable in the equation should be the number of seconds t since the water was poured. Focus on accuracy during the first 5 minutes.
Does that seem hard? I think it’s hard. The relevant physical phenomena include at least:
Conduction of heat between the water, the mug, the air, and the table.
Conduction of heat inside each of those things.
Convection (fluid movement) inside the water and the air.
Evaporation cooling as water molecules become vapor.
Movement of water vapor in the air.
Radiation. (Like all matter, the mug and water emit temperature-dependent infrared radiation.)
Surface tension, thermal expansion/contraction, re-absorption of air into the water as it cools, probably more.
And many details aren’t specified in the prompt. Is the mug made of porcelain or stoneware? What is the mug’s shape? What is the table made of? How humid is the air? How am I reducing the spatially varying water temperature to a single number?
So this isn’t a problem where you can sit around and think and find
with a “correct” answer that you can find by thinking. Reality is too complicated. Instead, answering question requires “taste”—guessing which factors are most important, making assumptions about missing details, etc.
I tasked 16 agents with writing a Rust-based C compiler, from scratch, capable of compiling the Linux kernel. Over nearly 2,000 Claude Code sessions and $20,000 in API costs, the agent team produced a 100,000-line compiler that can build Linux 6.9 on x86, ARM, and RISC-V.
Bit more commentary on the capabilities benchmarking angle:
This project was designed as a capability benchmark. I am interested in stress-testing the limits of what LLMs can just barely achieve today in order to help us prepare for what models will reliably achieve in the future.
I’ve been using the C Compiler project as a benchmark across the entire Claude 4 model series. As I did with prior projects, I started by drafting what I wanted: a from-scratch optimizing compiler with no dependencies, GCC-compatible, able to compile the Linux kernel, and designed to support multiple backends. While I specified some aspects of the design (e.g., that it should have an SSA IR to enable multiple optimization passes) I did not go into any detail on how to do so.
Previous Opus 4 models were barely capable of producing a functional compiler. Opus 4.5 was the first to cross a threshold that allowed it to produce a functional compiler which could pass large test suites, but it was still incapable of compiling any real large projects. My goal with Opus 4.6 was to again test the limits.
Over nearly 2,000 Claude Code sessions across two weeks, Opus 4.6 consumed 2 billion input tokens and generated 140 million output tokens, a total cost just under $20,000. Compared to even the most expensive Claude Max plans, this was an extremely expensive project. But that total is a fraction of what it would cost me to produce this myself—let alone an entire team.
This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build a bootable Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis, and has a 99% pass rate on most compiler test suites including the GCC torture test suite. It also passes the developer’s ultimate litmus test: it can compile and run Doom.
By 2026, more code gets written in a week than the world wrote in 2020. Open source projects fork themselves into an endless orgy of abundance. Some high school students build functionally near-identical versions of Windows and Google Drive (and every video game in existence) from scratch in a month, because they can and they wanted one new feature on top of it. Everyone and their dog has a software product line. Big Tech unleashes a torrent of lawsuits against people cloning their products, echoing the Oracle v Google lawsuit about Java, but those lawsuits will take years to complete, and months feel like decades on the ground.
Back to Carlini on where Opus 4.6 fell short:
The compiler, however, is not without limitations. These include:
It lacks the 16-bit x86 compiler that is necessary to boot Linux out of real mode. For this, it calls out to GCC (the x86_32 and x86_64 compilers are its own).
It does not have its own assembler and linker; these are the very last bits that Claude started automating and are still somewhat buggy. The demo video was produced with a GCC assembler and linker.
The compiler successfully builds many projects, but not all. It’s not yet a drop-in replacement for a real compiler.
The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled.
The Rust code quality is reasonable, but is nowhere near the quality of what an expert Rust programmer might produce.
The resulting compiler has nearly reached the limits of Opus’s abilities. I tried (hard!) to fix several of the above limitations but wasn’t fully successful. New features and bugfixes frequently broke existing functionality.
As one particularly challenging example, Opus was unable to implement a 16-bit x86 code generator needed to boot into 16-bit real mode. While the compiler can output correct 16-bit x86 via the 66⁄67 opcode prefixes, the resulting compiled output is over 60kb, far exceeding the 32k code limit enforced by Linux. Instead, Claude simply cheats here and calls out to GCC for this phase (This is only the case for x86. For ARM or RISC-V, Claude’s compiler can compile completely by itself.)
Broadly speaking, we now see an empirical tradeoff between the level of AI involvement in the solution, and the difficulty or novelty of that solution. In particular, the recent solutions have spanned a spectrum roughly describable as follows:
1. Completely autonomous AI solutions to Erdos problems that are short and largely follow a standard technique. (In many, but not all, of these cases, some existing literature was found that proved a very similar result by a similar method.)
2. AI-powered modifications of existing solutions (which could be either human-generated or AI-generated) that managed to improve or modify these solutions in various ways, for instance by upgrading a partial solution to a full solution, or optimizing the parameters of the proof.
3. Complex interactions between humans and AI tools in which the AI tools provided crucial calculations, or proofs of key steps, allowing the collaboration to achieve moderately complicated and novel solutions to open problems.
4. Difficult research-level papers solving one or more Erdos problems by mostly traditional human means, but for which AI tools were useful for secondary tasks such as generation of code, numerics, references, or pictures.
The seeming negative correlation between the amount of AI involvement and the depth of result is somewhat reminiscent of statistical paradoxes such as Berkson’s paradox https://en.wikipedia.org/wiki/Berkson%27s_paradox or Simpson’s paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox . One key confounding factor is that highly autonomous AI workflows are much more scaleable than human-intensive workflows, and are thus better suited for being systematically applied to the “long tail” of obscure Erdos problems, many of which actually have straightforward solutions. As such, many of these easier Erdos problems are now more likely to be solved by purely AI-based methods than by human or hybrid means.
Given the level of recent publicity given to these problems, I expect that over the next few weeks, pretty much all of the outstanding Erdos problems will be quietly attempted by various people using their preferred AI tool. Most of the time, these tools will not lead to any noteworthy result, but such failures are unlikely to be reported on any public site. It will be interesting to see what (verified) successes do emerge from this, which should soon give a reasonably accurate picture of what proportion of currently outstanding Erdos problems are simple enough to be amenable to current AI tools operated with minimal human intervention. (My guess is that this proportion is on the order of 1-2%.) Assessing the viability of more hybridized human-AI approaches will take significantly longer though, as human expert attention will remain a significant bottleneck.
So I’ll whimsically define the “Erdos problems benchmark” to be “the proportion of currently outstanding Erdos problems amenable to current AI tools operated with minimal human intervention”, and the current “SOTA” to be Tao’s guess of 1-2% as of Jan 2026. My guess is it won’t be saturated in ~2 years like every other benchmark because open math problems can be unboundedly hard, but who knows?
For my own future reference, here are some “benchmarks” (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical. (The “serious” version would probably start with the Evals section of technicalities’ 2025 shallow review of technical AIS, or SemiAnalysis’ in-house view.)
the AI village and blog, not really a “benchmark” per se but my richest source of intuitions about current frontier models’ capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other “benchmark-equiv” frontier models
certain folks’ domain-specific opinions, e.g.
lc on cybersec (like this take on AISLE),
Kevin Buzzard and Terry Tao on math,
Adam Karvonen on physical tasks relevant to manufacturing,
Sarah Constantin on lit reviews,
nostalgebraist on blog posts worth reading,
Linch on writing stories which they’re “substantially worse at than blogging”,
Gwern and Jennifer Chen on writing diversity & creativity (but not Sam Altman),
Cole Wyeth on novel ideas (update: Cole thinks ChatGPT 5.2 met his bar for autonomously having an original insight by solving an open COLT problem with no assistance) and on agency (Anthropic’s Project Vend, Pokemon, time horizons on realistic SWE tasks passing 8-16 hours)
Thane Ruthenis (although Thane’s milestones are on a totally different capability tier),
Steven Byrnes on AGI etc many others
“AGI is here” takes, e.g. Logan Zoellner—Jan ’23, Abram Demski—Mar ‘24, Tyler Cowen—Apr ‘25, JenniferRM—Dec ‘25, Gordon Worley—Feb ’26
altruism-related, e.g. the AI village’s raise as much money for charity as you can. Confoundingly, the much more capable agents a year after raised only 25% of the amount earlier-gen models raised ($510 from 17 human donors for Doctors Without Borders vs $1,984 in year 1 for Helen Keller International and Malaria Consortium)
update: the AI Digest folks concluded it’s because “humans are less interested”
writing-related:
the winners of the Un-slop Prize
Pilish poem length and interestingness
Chats on read.haus with AI simulations of prominent authors become preferable to reading the latter’s real content. Scott Alexander, Sarah Constantin, Spencer Greenberg, Byrne Hobart, Tyler Cowen, Dwarkesh Patel, Andy Matuschak etc are all on there but they never come across quite right to me
AI “starts outputting pitch-perfect blog posts that sound like Adam Mastroianni”, or changes the opinions of Jasmine Sun, Erik Hoel, Sam Kriss, nostalgebraist above, etc
Eliezer likes their story plot idea for “a new Bruce Kent story”:
Starburst, fictional theoretical physics. I don’t really get their leaderboard though
a small set of work-related spreadsheet modelling problems I keep thinking current agents should easily do but they keep failing in very irritating ways, Claude Code included. I’m waiting for agents that will finally speed me up not slow me down on these. Possibly skill issue on my part
programming-related: ProgramBench (supposedly impossible)
FWIW, Anthropic’s members of technical staff estimates of productivity boost: currently 1.15-1.4x with Sonnet 4.5 for most, except that one person at 2x as “their workflow was now mainly focused on managing multiple agents”, wonder if it’s the same person Sholto Douglas mentioned worked with 9 agents at the same time
update: section 7.3.4 of the Opus 4.5 system card says 2-3x productivity boost vs Sonnet 4.5′s 1.15-1.4x above: “Nine of 18 participants reported ≥100% productivity improvements, with a median estimate of 100% and a mean estimate of 220%”
update: section 2.3.6 of the Mythos system card says ~4x productivity boost geomean with wide distribution, but: “productivity uplift on individual tasks does not translate one-for-one into acceleration of research progress. Compute is also a key ingredient, as promising ideas need to be de-risked at scale. Our best estimates of the elasticity of progress to researcher output, combined with the observed uplift, yield an overall progress multiplier below 2×. We estimate that reaching 2× on overall progress via this channel would require uplift roughly an order of magnitude larger than what we observe”
how blind models see the earth, plot the Mandelbrot set, etc
avoiding spiralling into spiritual bliss attractors (maybe this is just Claude being a hippie)
just for fun, Gary Marcus’ 5 challenges by 2029 (from 2022)
their slope on the chart below exceeds that of humans (I’m not a fan of the notion of task horizon length, it bakes in perf plateauing that doesn’t happen to humans thinking longer, hence slope)
great summary of reasons time horizon is overrated/misinterpreted by Thomas Kwa
math-related:
FrontierMath Tier 4 because I like math x AI, plus commentary like Kevin Buzzard’s “I was amused this week to have been sent data on what happens if you ask lots of agents to try and solve these problems and you mark the question as being solved if at least one agent gets the answer correct at least once”
more generally in math x AI: Gavin Leech’s examples thread of “crucially useful AI in research maths”
the “Erdos problems benchmark”: the proportion of currently outstanding Erdos problems amenable to current AI tools operated with minimal human intervention (inspired by Terry Tao, more below)
vibe-proving math theorems in Lean except it doesn’t take a week and isn’t “extremely annoying” (despite Adam Mastroianni’s argument that what a dream job really feels like is to be perpetually annoyed). The main issue is in verifying that the human proof-to-Lean code translation is faithful, which doesn’t seem automatable
Epoch’s Capabilities Index because it’s general (composite metric of most of the high-profile benchmarks out there) stitched together using a methodology that seems intuitively correct (item response theory), although admittedly as someone who started out believing anything is measurable if you try hard enough I’ve gradually grown disillusioned enough to down-weight even ostensibly good composite benchmarks like ECI a fair bit. Also CAIS’s definition of AGI
Scale’s Remote Labor Index because I work remotely. 230 projects from Upwork freelancers “excluding projects requiring physical labor, long-term evaluation, or direct client interaction”, mean and median human completion time 29 and 11.5 hours respectively, mean and median project value $630 and $200. Manus at 2.50% tops the leaderboard, then Sonnet 4.5 > GPT-5 > ChatGPT Agent > Gemini 2.5 Pro last at 0.83%, which matches my impression of their relative “IRL competence” in the AI Village
As of mid-March 2026, Claude Opus 4.6 (Cowork) got it up to 4.17%
Nicholas Carlini’s “build a C compiler” (more below)
Visakan Veerasamy’s threadthulhu gets tamed: “No mention of threading culture is complete without a hat-tip to Visakan Veeraswamy, (@visakanv) of course. Visa took the basic linear threading idea pioneered by Marc and turned it into a dizzying artform, turning his account into a tangled, densely interlinked, quote-linked, promiscuously forking Lovecraftian monstrosity of a twitter hyperobject. I came up with a term for it: threadthulhu (my main contribution to culture through the twitter years was coming up with names for things). My own threadthulu was only middling crazy. Orderly enough that I was able to index all my good threads in a meta-thread over the years, and slaughter it relatively cleanly to create the raw material for this book. I doubt Visa’s insane threadthulu can be killed at all, let alone properly butchered into a book-like echo like this one. I vibecoded the pipeline that created this book, but it will probably take AGI to similarly tame Visa’s threadthulu”
ARC-AGI-3, GPT-5.4 high being at 0.3% as of March 2026
games-related:
Pokemon FireRed: Fable beat it in just over 50 hours with a vision-only harness. Julian: “That’s an hour faster than a heavily-harnessed GPT 5.5 beat FireRed, and >6x faster than the 325 hours it took a lightly-harnessed Opus 4.7 to beat Pokémon Red. (For comparison, an average human would take about 30 hours for FireRed and 26 hours for Red.)” Started from how well is Claude playing Pokemon? in March 2025
A puzzle game called SPL-T, where someone got Claude Code to break Josh Holder’s world record
auto-creating memory flashcards Andy Matuschak thinks are good (>90%?), they’ve actually worsened over time oddly. This is a taste problem (more at Memory Machines):
“Their blind spot is cards that seem OK at a glance, but won’t make sense when reviewed months later. When I write cards, I simulate seeing them in future sessions, based on years of SRS review experience. LLMs don’t have that taste, and we couldn’t figure out how to infuse it”
A programming-related one, from Christian Ekrem’s The Tacit Dimension: Why Your Best Engineers Can’t Tell You What They Know:
I’m not sure how to operationalise a bet against Ekrem’s claim that “this kind of work is impossible for an AI”, but I’d like to; this smells like Flying Machines Which Do Not Fly, the NYT editorial predicting
69 days before the Wright brothers did their thing at Kitty Hawk. And I say this as someone who thinks tacit knowledge is very important.
I liked dynomight’s “temperature over time of boiling water poured into a ceramic coffee mug” as a low-budget DIY test of research taste, so it goes into the list above. Opus 4.6 did best and cost $0.61:
More detail:
From Nicholas Carlini’s Anthropic blog post:
Bit more commentary on the capabilities benchmarking angle:
This reminds me of a passage from L Rudolf L’s history of the future:
Back to Carlini on where Opus 4.6 fell short:
Another whimsical “benchmark”: Terry Tao wrote on Mathstodon that
So I’ll whimsically define the “Erdos problems benchmark” to be “the proportion of currently outstanding Erdos problems amenable to current AI tools operated with minimal human intervention”, and the current “SOTA” to be Tao’s guess of 1-2% as of Jan 2026. My guess is it won’t be saturated in ~2 years like every other benchmark because open math problems can be unboundedly hard, but who knows?