An observation about how amazing large economies are: the inflation adjusted GDP of the entire United States in 1810 (~8million people) is roughly the same amount that is spent on PhD stipends per year in the US in 2026 (around 0.05% of 2026 GDP).
I think most of the uncertainty in answering this question comes from the uncertainty around the actual hardness of the Riemann hypothesis. The difficulty of open mathematical questions seems hard to pinpoint and has an enormous range which is most likely relevant even on a scale of varying strong artificial intelligences. To focus more on the relation of GDP to model capabilities for solving “extremely hard but humanly possible to solve problems”, one could substitute the Riemann hypothesis with any Millenium problem. Then I think the answer is probably “not much more than today, maybe 2-8%” (Gemini tells me we’re at 1% now).
METR should test for a 99.9% task completion rate (in addition to the current 80% and 50%). A key missing ingredient holding back LLM economic impact is that they’re just not robust enough. This can be viewed analogously to the problem of self-driving. Every individual component of self-driving is ~solved, but stringing them together results in a non-robust final product. I believe that automating research/engineering completely will require nines of reliability that we just don’t have. And testing for nines of reliability could be done by giving the model many very short time horizon tasks and seeing how it performs.
This can be further motivated by considering what happens if we string together tasks with a non-99.99...% completion rate. Say we take the GPT 5.1 codex max result. METR claims this model has a 50% time horizon of 2 hours and 40 minutes. Say we tell the model to do task A which is 2 hours and 40 minutes. P(A) = 0.5. Now if the model decides it needs to do task B to further it’s research, we have P(B) = 0.5. P(A, B) = P(A)P(B) = 0.25 (These events are not independent, but I express them as such for illustrative effect). We can then consider task C, D, E, etc. This holds even for higher completion rates of 80%. Once we get up to 99.9%, we have P(A) = 0.999, P(B) = 0.999, P(A, B) = P(A)P(B) = ~0.998… This is where we can really start seeing autonomous research imo.
It would be interesting to benchmark humans at 99.9% task completion rate and see what their task length is.
(Disclaimer: I am not completely sure of METR’s methodology for determining task length)
Unfortunately the available benchmark tasks do not allow for 99%+ reliability measurements. Because we don’t have 1,000 different one-minute tasks the best we could do would be something like whether GPT5.1 can do all 40 tasks 25 times each with perfect reliability. Most likely it will succeed at all of them because we just don’t have a task that happens to trip it up.
As for humans’ 99.9%, at a granular enough level it would be 0.2 seconds (typing one keystroke) because few people have higher than 99.9% accuracy. But in the context of a larger task, we can correct our typos, so it isn’t super relevant.
A key missing ingredient holding back LLM economic impact is that they’re just not robust enough.
I disagree with this in this particular context. We are looking at AI companies trying to automate AI automation via AIs. Most tasks in AI R&D don’t require much reliability, I don’t know the distribution of outcomes in ML experiments but I reckon a lot of them are basically failures/have null results, but the distribution of the impact of such experiments has a long tail[1]. Also ML experiments don’t have many irreversible parts, AI R&D researchers aren’t like surgeons where mistakes have huge costs: Any ML experiment can be sandboxed, given a bounded amount of resources, shut down when it takes up too much. You need high reliability when the cost of failure is necessarily very high, but when running ML experiments that’s not the case.
Edit: Claude 4.5 Sonnet gives feedback on my text above, says that the search strategy matters if we’re looking at ML engineering. If it’s breadth-first & innovations don’t require a deep tree to go down, then low reliability is fine. But if we need to combine ≥4 innovations in a depth-first search then reliability matters more.
I don’t think this is a crux for me but learning that it’s a thin-tailed distribution would make me at least think about this problem a bit more. Claude claims hyperparameter tunes have lognormal returns (shifted so that the mean is slightly below baseline).
Claude’s rebuttal is exactly my claim. If major AI research breakthroughs could be done in 5 hours, then imo robustness wouldn’t matter as much. You could run a bunch of models in parallel and see what happens (this is part of why models are so good at olympiads), but an implicit part of my argument/crux is that AI research is necessarily deep (meaning you need to string some number of successfully completed tasks together such that you get an interesting final result). And if the model messes up one part, your chain breaks. Not only does this give you weird results, but it breaks your chain of causality[1], which is essential for AI research.
I’ve also tried doing “vibe AI researching” (no human in the loop) with current models and I find it just fails right away. If robustness doesn’t matter, why don’t we see current models consistently making AI research breakthroughs at their current 80% task completion rate?
A counterargument to this is that if METR’s graph trend keeps up, and task length gets to some threshold, I’ll call it a week for example, then you don’t really care about P(A)P(B)P(C)..., you can just do the tasks in parallel and see which one works. (However, if my logic holds, I would guess that METR’s task benchmark hits a plateau at some point before doing full-on research at least with current model robustness)
By chain of causality, I mean that I did task A. If I am extremely confident that task A is correct I can then do a search from task A. Say I stumble on some task B, then C. If I get an interesting result from task C, then I can keep searching from there so long as I am confident in my results. I can also mentally update my causal chain by some kind of ~backprop. “Oh using a CNN in task A, then setting my learning rate to be this in task B, made me discover this new thing in task C so now I can draw a generalized intuition to approach task D. Ok this approach to D failed, let me try this other approach”.
Am I crazy that I think that coding agents dramatically slow me down? Everyone around me (including some of the most technically gifted people I know) raves about them and how it’s made them 2x/5x/10x/100x more productive. I haven’t felt this at all except for the case of vibe coding something for fun where I don’t really care about the quality/implementation of the code. For running experiments where I care about the code, I’ll try and have Claude code implement something, it will spit out a bunch of code. I’ll read the code, dislike parts of it and try to get Claude to iterate on it. Claude won’t really get to the target of what I want so at some point I’ll give up and write the entire thing myself. Also, every time I do vibe code something, I understand it less well and thus iterate worse on subsequent experiments/code I want to write. If I am unique in this, a reason could be because I’ve spent probably 1000x more time writing code as I have spent reading code and I suck at reading code.
I also can see a reason for this being that the delta between code that’s 98% well done for an experiment and code that’s 100% well done for the same experiment is dramatically different. And agents help to speed up the initial writing dramatically while not being able to iron off the last few small percentage points. In other words, the value I get out of perfect code, and almost perfect code is dramatically different. Am I crazy?
One thing I’ve noticed is that AI agents are much better at writing code than changing code. One approach that sometimes works for me is to write a spec and then have AI build it, and then iterate by updating the spec to clarify everything that wasn’t what I wanted, deleting all of the code, and having a new AI agent try again. Or if you’re happy with some files and not others, delete everything you’re not happy with and keep the pieces you are happy with.
You can save some loop time by making your prompts much more detailed, and using plan mode to check if your prompts have the amount of context you think they do. If I actually care about the output of a coding session, my prompts get huge, covering what I’m trying to do, why, what I’m not trying to do, hard constraints, relevant code to read, how I want particularly tricky pieces done, high-level design, everything I can possibly think of that might go wrong, and so on. Spending 10 minutes just brain dumping can save you a lot of time later.
My CLAUDE.md also has a bunch of standard things (always use strict typing, which libraries and linters I prefer, DRY advice that agents sometimes follow, etc.).
Even then, you probably won’t get 100% perfect code, but it should at least be in a state where improving it is easier than writing it from scratch (or for me, easier than finding the motivation to write code from scratch).
Why don’t EA/rationalist firms use prediction markets for deciding who to hire (futarchy style)? Your application to a company would be something like ticking a box that says, “I give you permission to create a prediction market on me that I won’t trade in.” The statement the market would be trading could be P(we won’t regret hiring X | we hire X) or P(X will meet the following KPIs | we hire X), where X is the candidate. Effectively, this would outsource the work of hiring to people who want to profit on the market.
You could also make candidates do some set of tests or competitions and show that information to traders (as well as resume and other standard info). Then you pick the top trading candidates you want (or several above some threshold). This would also give more opportunity to some candidate X to publicly do things they believe will convince traders to increase their market’s price. It would be really cool if the mechanism actually deciding hiring decisions had the capacity to look at everything a candidate was willing to do/show to get hired, vs the current model of very restricted time commitment per candidate.
I suspect the blocker to this is that the trading volume would just be too low to give meaningful information.
Mostly because it doesn’t work? Because the analogies you’re assuming between big stocks and small decisions don’t apply? Big stocks have billions of dollars traded and reporting/auditing requirements backed by courts and banks and governments. Try drawing a line from how Dow stocks behave to how penny stocks behave, and then extrapolate way past there.
For most people the answer is “no, there is no process for this”.
For long-time collaborators or close allies we sometimes make it work, and it’s fine to ask if you know us (e.g. an Inkhavener recently asked me if they could stay for a few weeks as they were in the process of moving to the Bay, and I looked into it but it didn’t work out).
I’ve heard this critique lobbed around a fair bit in AI safety circles: “{some org} is bad at philosophy”. What does this mean? I’ve heard this both from collaborators in person and on LW. A decent number of times, this critique has been directed at Anthropic. I’ll apologize for how vague (and unclear) this post is in advance but does anyone have any idea what people are getting at when they say this? It is usually an unmotivated statement too (at least in my judgement) and a statement meant to critique an organization’s approach to AI safety.
You might take a look at Wei Dai’s writing on metaphilosophy. He has a specific view that isn’t shared by everyone on this site. But a core part of his view is that “a powerful AI (or human-AI civilization) guided by wrong philosophical ideas would likely cause astronomical (or beyond astronomical) waste.”
An observation about how amazing large economies are: the inflation adjusted GDP of the entire United States in 1810 (~8million people) is roughly the same amount that is spent on PhD stipends per year in the US in 2026 (around 0.05% of 2026 GDP).
Any predictions for what percentage of world GDP AI will be when an AI solves the Riemann hypothesis? (If this happens)
The real part will be 1⁄2, not sure about the imaginary part.
I think most of the uncertainty in answering this question comes from the uncertainty around the actual hardness of the Riemann hypothesis. The difficulty of open mathematical questions seems hard to pinpoint and has an enormous range which is most likely relevant even on a scale of varying strong artificial intelligences. To focus more on the relation of GDP to model capabilities for solving “extremely hard but humanly possible to solve problems”, one could substitute the Riemann hypothesis with any Millenium problem. Then I think the answer is probably “not much more than today, maybe 2-8%” (Gemini tells me we’re at 1% now).
METR should test for a 99.9% task completion rate (in addition to the current 80% and 50%). A key missing ingredient holding back LLM economic impact is that they’re just not robust enough. This can be viewed analogously to the problem of self-driving. Every individual component of self-driving is ~solved, but stringing them together results in a non-robust final product. I believe that automating research/engineering completely will require nines of reliability that we just don’t have. And testing for nines of reliability could be done by giving the model many very short time horizon tasks and seeing how it performs.
This can be further motivated by considering what happens if we string together tasks with a non-99.99...% completion rate. Say we take the GPT 5.1 codex max result. METR claims this model has a 50% time horizon of 2 hours and 40 minutes. Say we tell the model to do task A which is 2 hours and 40 minutes. P(A) = 0.5. Now if the model decides it needs to do task B to further it’s research, we have P(B) = 0.5. P(A, B) = P(A)P(B) = 0.25 (These events are not independent, but I express them as such for illustrative effect). We can then consider task C, D, E, etc. This holds even for higher completion rates of 80%. Once we get up to 99.9%, we have P(A) = 0.999, P(B) = 0.999, P(A, B) = P(A)P(B) = ~0.998… This is where we can really start seeing autonomous research imo.
It would be interesting to benchmark humans at 99.9% task completion rate and see what their task length is.
(Disclaimer: I am not completely sure of METR’s methodology for determining task length)
Unfortunately the available benchmark tasks do not allow for 99%+ reliability measurements. Because we don’t have 1,000 different one-minute tasks the best we could do would be something like whether GPT5.1 can do all 40 tasks 25 times each with perfect reliability. Most likely it will succeed at all of them because we just don’t have a task that happens to trip it up.
As for humans’ 99.9%, at a granular enough level it would be 0.2 seconds (typing one keystroke) because few people have higher than 99.9% accuracy. But in the context of a larger task, we can correct our typos, so it isn’t super relevant.
Is 80% the highest success rate you can practically test?
UPD Thomas essentially answered elsewhere:
I disagree with this in this particular context. We are looking at AI companies trying to automate AI automation via AIs. Most tasks in AI R&D don’t require much reliability, I don’t know the distribution of outcomes in ML experiments but I reckon a lot of them are basically failures/have null results, but the distribution of the impact of such experiments has a long tail [1] . Also ML experiments don’t have many irreversible parts, AI R&D researchers aren’t like surgeons where mistakes have huge costs: Any ML experiment can be sandboxed, given a bounded amount of resources, shut down when it takes up too much. You need high reliability when the cost of failure is necessarily very high, but when running ML experiments that’s not the case.
Edit: Claude 4.5 Sonnet gives feedback on my text above, says that the search strategy matters if we’re looking at ML engineering. If it’s breadth-first & innovations don’t require a deep tree to go down, then low reliability is fine. But if we need to combine ≥4 innovations in a depth-first search then reliability matters more.
I don’t think this is a crux for me but learning that it’s a thin-tailed distribution would make me at least think about this problem a bit more. Claude claims hyperparameter tunes have lognormal returns (shifted so that the mean is slightly below baseline).
Claude’s rebuttal is exactly my claim. If major AI research breakthroughs could be done in 5 hours, then imo robustness wouldn’t matter as much. You could run a bunch of models in parallel and see what happens (this is part of why models are so good at olympiads), but an implicit part of my argument/crux is that AI research is necessarily deep (meaning you need to string some number of successfully completed tasks together such that you get an interesting final result). And if the model messes up one part, your chain breaks. Not only does this give you weird results, but it breaks your chain of causality[1], which is essential for AI research.
I’ve also tried doing “vibe AI researching” (no human in the loop) with current models and I find it just fails right away. If robustness doesn’t matter, why don’t we see current models consistently making AI research breakthroughs at their current 80% task completion rate?
A counterargument to this is that if METR’s graph trend keeps up, and task length gets to some threshold, I’ll call it a week for example, then you don’t really care about P(A)P(B)P(C)..., you can just do the tasks in parallel and see which one works. (However, if my logic holds, I would guess that METR’s task benchmark hits a plateau at some point before doing full-on research at least with current model robustness)
By chain of causality, I mean that I did task A. If I am extremely confident that task A is correct I can then do a search from task A. Say I stumble on some task B, then C. If I get an interesting result from task C, then I can keep searching from there so long as I am confident in my results. I can also mentally update my causal chain by some kind of ~backprop. “Oh using a CNN in task A, then setting my learning rate to be this in task B, made me discover this new thing in task C so now I can draw a generalized intuition to approach task D. Ok this approach to D failed, let me try this other approach”.
Am I crazy that I think that coding agents dramatically slow me down? Everyone around me (including some of the most technically gifted people I know) raves about them and how it’s made them 2x/5x/10x/100x more productive. I haven’t felt this at all except for the case of vibe coding something for fun where I don’t really care about the quality/implementation of the code. For running experiments where I care about the code, I’ll try and have Claude code implement something, it will spit out a bunch of code. I’ll read the code, dislike parts of it and try to get Claude to iterate on it. Claude won’t really get to the target of what I want so at some point I’ll give up and write the entire thing myself. Also, every time I do vibe code something, I understand it less well and thus iterate worse on subsequent experiments/code I want to write. If I am unique in this, a reason could be because I’ve spent probably 1000x more time writing code as I have spent reading code and I suck at reading code.
I also can see a reason for this being that the delta between code that’s 98% well done for an experiment and code that’s 100% well done for the same experiment is dramatically different. And agents help to speed up the initial writing dramatically while not being able to iron off the last few small percentage points. In other words, the value I get out of perfect code, and almost perfect code is dramatically different. Am I crazy?
One thing I’ve noticed is that AI agents are much better at writing code than changing code. One approach that sometimes works for me is to write a spec and then have AI build it, and then iterate by updating the spec to clarify everything that wasn’t what I wanted, deleting all of the code, and having a new AI agent try again. Or if you’re happy with some files and not others, delete everything you’re not happy with and keep the pieces you are happy with.
You can save some loop time by making your prompts much more detailed, and using plan mode to check if your prompts have the amount of context you think they do. If I actually care about the output of a coding session, my prompts get huge, covering what I’m trying to do, why, what I’m not trying to do, hard constraints, relevant code to read, how I want particularly tricky pieces done, high-level design, everything I can possibly think of that might go wrong, and so on. Spending 10 minutes just brain dumping can save you a lot of time later.
My CLAUDE.md also has a bunch of standard things (always use strict typing, which libraries and linters I prefer, DRY advice that agents sometimes follow, etc.).
Even then, you probably won’t get 100% perfect code, but it should at least be in a state where improving it is easier than writing it from scratch (or for me, easier than finding the motivation to write code from scratch).
Why don’t EA/rationalist firms use prediction markets for deciding who to hire (futarchy style)? Your application to a company would be something like ticking a box that says, “I give you permission to create a prediction market on me that I won’t trade in.” The statement the market would be trading could be P(we won’t regret hiring X | we hire X) or P(X will meet the following KPIs | we hire X), where X is the candidate. Effectively, this would outsource the work of hiring to people who want to profit on the market.
You could also make candidates do some set of tests or competitions and show that information to traders (as well as resume and other standard info). Then you pick the top trading candidates you want (or several above some threshold). This would also give more opportunity to some candidate X to publicly do things they believe will convince traders to increase their market’s price. It would be really cool if the mechanism actually deciding hiring decisions had the capacity to look at everything a candidate was willing to do/show to get hired, vs the current model of very restricted time commitment per candidate.
I suspect the blocker to this is that the trading volume would just be too low to give meaningful information.
Mostly because it doesn’t work? Because the analogies you’re assuming between big stocks and small decisions don’t apply? Big stocks have billions of dollars traded and reporting/auditing requirements backed by courts and banks and governments. Try drawing a line from how Dow stocks behave to how penny stocks behave, and then extrapolate way past there.
Is it possible to stay at Lighthaven even if not attending an event there?
Yes, I’ve asked them to rent a room for a few nights. They may need to trust you.
For most people the answer is “no, there is no process for this”.
For long-time collaborators or close allies we sometimes make it work, and it’s fine to ask if you know us (e.g. an Inkhavener recently asked me if they could stay for a few weeks as they were in the process of moving to the Bay, and I looked into it but it didn’t work out).
I’ve heard this critique lobbed around a fair bit in AI safety circles: “{some org} is bad at philosophy”. What does this mean? I’ve heard this both from collaborators in person and on LW. A decent number of times, this critique has been directed at Anthropic. I’ll apologize for how vague (and unclear) this post is in advance but does anyone have any idea what people are getting at when they say this? It is usually an unmotivated statement too (at least in my judgement) and a statement meant to critique an organization’s approach to AI safety.
You might take a look at Wei Dai’s writing on metaphilosophy. He has a specific view that isn’t shared by everyone on this site. But a core part of his view is that “a powerful AI (or human-AI civilization) guided by wrong philosophical ideas would likely cause astronomical (or beyond astronomical) waste.”