Eli was saying something like “90%-horizons of 100 years sound about right for Superhuman Coder level performance”
To be clear, this is on (a theoretical extrapolated version of) METR HCAST, not the real world distribution of software engineering projects.
Also to remind others of the definition of superhuman coder, it’s a pretty high bar:
Superhuman coder (SC): An AI system for which the company could run with 5% of their compute budget 30x as many agents as they have human research engineers, each of which is on average accomplishing coding tasks involved in AI research (e.g. experiment implementation but not ideation/prioritization) at 30x the speed (i.e. the tasks take them 30x less time, not necessarily that they write or “think” at 30x the speed of humans) of the company’s best engineer. This includes being able to accomplish tasks that are in any human researchers’ area of expertise.
To be clear, this is on (a theoretical extrapolated version of) METR HCAST, not the real world distribution of software engineering projects.
Also to remind others of the definition of superhuman coder, it’s a pretty high bar: