Feedback welcome: www.admonymous.co/mo-putera
Long-time lurker (c. 2013), recent poster. I also write on the EA Forum.
Mo Putera
Karma: 2,109
Every once in a while I think about Robert Freitas’ 1984 essay Xenopsychology, in particular his Sentience Quotient (SQ) idea:
It is possible to devise a sliding scale of cosmic sentience universally applicable to any intelligent entity in the cosmos, based on a “figure of merit” which I call the Sentience Quotient. The essential characteristic of all intelligent systems is that they process information using a processor or “brain” made of matter-energy. Generally the more information a brain can process in a shorter length of time, the more intelligent it can be. (Information rate is measured in bits/second, where one bit is the amount of information needed to choose correctly between two equally likely answers to a simple yes/no question.) Also, the lower the brain’s mass the less it will be influenced by fundamental limits such as speed of light restrictions on internal propagation, heat dissipation, and the Square-Cube Law.
The most efficient brain will have the highest information-processing rate I, and the lowest mass M, hence the highest ratio I/M. Since very large exponents are involved, for the convenience we define the Sentience Quotient or SQ as the logarithm of I/M, that is, its order of magnitude. Of course, SQ delimits maximum potential intellect–a poorly programmed or poorly designed (or very small) high-SQ brain could still be very stupid. But all else remaining equal larger-SQ entities should be higher-quality thinkers.
The lower end of our cosmic scale is easy to pin down. The very dumbest brain we can imagine would have one neuron with the mass of the universe (1052 kg) and require a time equal to the age of the universe (1018 seconds) to process just one bit, giving a minimum SQ of −70.
Whenever I see the “The difference between genius and stupidity is that genius has its limits” quote (usually apocryphally attributed to Einstein) I imagine Freitas retorting “no, so does stupidity, the limit is SQ −70″.
What is the smartest possible brain? Dr. H. Bremermann at the University of California at Berkeley claims there is a fundamental limit to intelligence imposed by the laws of quantum mechanics. The argument is simple but subtle. All information, to he acted upon, must be represented physically and be carried by matter-energy “markers.” According to Heisenberg’s Uncertainty Principle in quantum mechanics, the lower limit for the accuracy with which energy can be measured–the minimum measurable energy level for a marker carrying one bit–is given by Planck’s constant h divided by T, the duration of the measurement. If one energy level is used to represent one bit, then the maximum bit rate of a brain is equal to the total energy available E ( = mc2) for representing information, divided by the minimum measurable energy per bit (h/T) divided by the minimum time required for readout (T). or mc2/h = 1050 bits/sec/kg. Hence the smartest possible brain has an SQ of +50.
For a while I wondered what such a superbrain would be like, and then I found Seth Lloyd’s paper quantitatively bounding the computational power of a hypothetical “ultimate laptop” of mass 1 kg confined to volume 1L, which derives the same computation limit to within an OOM, concluding that “a typical state of the ultimate laptop’s memory looks like a plasma at a billion degrees Kelvin: the laptop’s memory looks like a thermonuclear explosion or a little piece of the Big Bang!”; its energy throughput would need to be a preposterous 4.04 x 1026 watts, slightly more than the entire sun’s output of 3.846 × 1026 watts(!!).
Where do people fit in? A human neuron has an average mass of about 10-10 kg and one neuron can process 1000-3000 bits/sec. earning us an SQ rating of +13.
That 50 − 13 = 37 OOMs of headroom estimate between humans and Freitas’ “mini-Big Bang superbrains” has stuck in my mind ever since. The “practical” headroom is definitely much lower, although how much I don’t know.
What is most interesting here is not the obvious fact that there’s a great deal of room for improvement (there is!), but rather that all “neuronal sentience” SQs, from insects to mammals, cluster within several points of the human value. From the cosmic point of view, rotifers, honeybees, and humans all have brainpower with roughly equivalent efficiencies. Note that we are still way ahead of the computers, with an Apple II at SQ +5 and even the mighty Cray I only about +9.
As an update on that 40-year old estimate, ChatGPT-5 medium estimates that “the highest value you can plausibly assign to a real, shipping computer “brain” today belongs to Cerebras’s wafer-scale processor (WSE-3) used in the CS-3 system. Using public performance and physical data, its chip-only SQ comes out around +19½. If you insist on a whole-system number (including packaging/cooling/rack), the CS-3-as-appliance is roughly +16; the most compute-dense Nvidia rack (GB200 NVL72) is about +15.9; and the #1 TOP500 supercomputer (El Capitan) is about +14.2.” I have a feeling smartphones might beat this, not sure why GPT-5 considered and dismissed assessing them in its reasoning trace.
Another kind of sentience, which we may call “hormonal sentience,” is exhibited by plants. Time-lapse photography shows the vicious struggles among vines in the tropical rain forests, and vegetative phototaxis (turning toward light) is a well-known phenomenon. All these behaviors are mediated, it is believed, by biochemical plant hormones transmitted through the vascular system. As in the animal kingdom, most of the geniuses are hunters–the carnivorous plants. The Venus flytrap, during a 1- to 20-second sensitivity interval, counts two stimuli before snapping shut on its insect prey, a processing peak of 1 bit/sec. Mass is 10-100 grams, so flytrap SQ is about +1. Plants generally take hours to respond to stimuli, though, so vegetative SQs tend to cluster around −2.
How about intelligences greater than human? Astronomer Robert Jastrow and others have speculated that silicon-based computer brains may represent the next and ultimate stage in our evolution. This is valid, but only in a very limited sense. Superconducting Josephson junction electronic gates weigh 10-12 kg and can process 1011 bits/sec, so “electronic sentiences” made of these components could have and SQ of +23 – ten orders beyond man. But even such fantastically advanced systems fall short of the maximum of +50. Somewhere in the universe may lurk beings almost incomprehensible to us, who think by manipulating atomic energy levels and are mentally as far beyond our best future computers as those computers will surpass the Venus flytrap.
Just as consciousness is an emergent of neuronal sentience, perhaps some broader mode of thinking–call it communalness–is an emergent of electronic sentience. If this is true, it might help to explain why (noncommunal) human beings have such great difficulty comprehending the intricate workings of the societies, governments, and economies they create, and require the continual and increasing assistance of computers to juggle the thousands of variables needed for successful management and planning. Perhaps future computers with communalness may develop the same intimate awareness of complex organizations as people have consciousness of their own bodies. And how many additional levels of emergent higher awareness might a creature with SQ +50 display?
The possible existence of ultrahuman SQ levels may affect our ability, and the desirability, of communicating with extraterrestrial beings. Sometimes it is rhetorically asked what we could possibly have to say to a dog or to an insect, if such could speak, that would be of interest to both parties? From our perspective of Sentience Quotients, we can see that the problem is actually far, far worse than this, more akin to asking people to discuss Shakespeare with trees or rocks. It may be that there is a minimum SQ “communication gap,” an intellectual distance beyond which no two entities can meaningfully converse.
At present, human scientists are attempting to communicate outside our species to primates and cetaceans, and in a limited way to a few other vertebrates. This is inordinately difficult, and yet it represents a gap of at most a few SQ points. The farthest we can reach is our “communication” with vegetation when we plant, water, or fertilize it, but it is evident that messages transmitted across an SQ gap of 10 points or more cannot be very meaningful.
What, then, could an SQ +50 Superbeing possibly have to say to us?
If we replace “SQ +50″ (which we know can’t work because of Seth Lloyd’s analysis above that they’ll be mini-Big Bangs so we wouldn’t survive their presence) with the more garden-variety ASIs I guess one possible answer is Charlie Stross’ Accelerando: ”...the narrator is Aineko and Aineko is not a cat. Aineko is an sAI that has figured out that humans are more easily interacted with/manipulated if you look like a toy or a pet than if you look like a Dalek. Aineko is not benevolent...”
Scott strongly encourages using well-crafted concept handles for reasons very similar to what Raemon describes, and thinks Eliezer’s writing is really impactful partly because he’s good at creating them. And “Offense is about status” doesn’t seem to me like it would create the reactions you predicted if people see that you in particular are the author (because of your track record of contributions); I doubt the people who would still round it off to strawman versions would not do so with your boring title anyway, so on the margin seems like a non-issue.
For future readers of this post and other writings on heroic responsibility who feel a bit amiss, Miranda Dixon-Luinenburg’s The Importance of Sidekicks may be for you (as it was for me). Think Samwise to Frodo or Robin to Batman, or if you know investing Charlie Munger to Warren Buffett, or if you like team sports Scottie Pippen to Michael Jordan. There’s probably a gradient from “assistant” to “second-in-command”; I lean more towards the latter. Miranda:
I suspect that the rationality community, with its “hero” focus, drives away many people who are like me in this sense. I’ve thought about walking away from it, for basically that reason. I could stay in Ottawa and be a nurse for forty years; it would fulfil all my most basic emotional needs, and no one would try to change me. Because oh boy, have people tried to do that. It’s really hard to be someone who just wants to please others, and to be told, basically, that you’re not good enough–and that you owe it to the world to turn yourself ambitious, strategic, Slytherin.
Firstly, this is mean regardless. Secondly, it’s not true.
Samwise was important. So was Frodo, of course. But Frodo needed Samwise. Heroes need sidekicks. They can function without them, but function a lot better with them. Maybe it’s true that there aren’t enough heroes trying to save the world. But there sure as hell aren’t enough sidekicks trying to help them. And there especially aren’t enough talented, competent, awesome sidekicks.
Miranda’s post clearly struck a chord as it generated 200+ comments back in the days when LW was smaller, including an endorsement and apology from Eliezer, but the one I personally found most memorable was this one because it seemed so counterintuitive:
I am male. I have high testosterone. I love competing and winning. I am ambitious and driven. I like to make a lot of money. I make a lot of money. I prefer the sidekick role.
If someone asks me “King or Prince?” I will respond with Prince every time. Hey, you can still be royalty without the weight of the world on your shoulders. I would still be a hard working Prince, too. If some asks me “Candidate or Campaign Manager?” I’ll take Campaign Manager, thank you. If someone asks me “President or Chief of Staff?” well, you know the answer by now.
The more money I make and the more wisdom and experience I acquire, the more people naturally turn to me to lead. And I do it when necessary. I’m even pretty good at it. But, I don’t love it. I don’t require it. I don’t see myself as growing more in that direction.
For my own future reference, here are some “benchmarks” (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical:
the AI village and blog, not really a “benchmark” per se but my richest source of intuitions about current frontier models’ capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other “benchmark-equiv” frontier models
certain folks’ opinions, e.g. lc on cybersec (like this take on AISLE), Kevin Buzzard and Terry Tao on math, Adam Karvonen on physical tasks relevant to manufacturing, Sarah Constantin on lit reviews, nostalgebraist on blog posts worth reading, Gwern and Jennifer Chen on writing diversity & creativity (but not Sam Altman), Cole Wyeth on novel ideas, Thane Ruthenis (although Thane’s milestones are on a totally different capability tier), Steven Byrnes on AGI etc many others
Chats on read.haus with AI simulations of prominent authors become preferable to reading the latter’s real content. Scott Alexander, Sarah Constantin, Spencer Greenberg, Byrne Hobart, Tyler Cowen, Dwarkesh Patel, Andy Matuschak etc are all on there but they never come across quite right to me
Starburst, fictional theoretical physics. I don’t really get their leaderboard though
a small set of work-related spreadsheet modelling problems I keep thinking current agents should easily do but they keep failing in very irritating ways, Claude Code included. I’m waiting for agents that will finally speed me up not slow me down on these. Possibly skill issue on my part
FWIW, Anthropic’s members of technical staff estimates of productivity boost: currently 1.15-1.4x with Sonnet 4.5 for most, except that one person at 2x as “their workflow was now mainly focused on managing multiple agents”, wonder if it’s the same person Sholto Douglas mentioned worked with 9 agents at the same time
how blind models see the earth, plot the Mandelbrot set, etc
avoiding spiralling into spiritual bliss attractors (maybe this is just Claude being a hippie)
just for fun, Gary Marcus’ 5 challenges by 2029 (from 2022)
their slope on the chart below exceeds that of humans (I’m not a fan of the notion of task horizon length, it bakes in perf plateauing that doesn’t happen to humans thinking longer, hence slope)
FrontierMath Tier 4 because I like math x AI, plus commentary like Kevin Buzzard’s “I was amused this week to have been sent data on what happens if you ask lots of agents to try and solve these problems and you mark the question as being solved if at least one agent gets the answer correct at least once”
vibe-proving math theorems in Lean except it doesn’t take a week and isn’t “extremely annoying” (despite Adam Mastroianni’s argument that what a dream job really feels like is to be perpetually annoyed). The main issue is in verifying that the human proof-to-Lean code translation is faithful, which doesn’t seem automatable
Epoch’s Capabilities Index because it’s general (composite metric of most of the high-profile benchmarks out there) stitched together using a methodology that seems intuitively correct (item response theory), although admittedly as someone who started out believing anything is measurable if you try hard enough I’ve gradually grown disillusioned enough to down-weight even ostensibly good composite benchmarks like ECI a fair bit. Also CAIS’s definition of AGI
Scale’s Remote Labor Index because I work remotely. 230 projects from Upwork freelancers “excluding projects requiring physical labor, long-term evaluation, or direct client interaction”, mean and median human completion time 29 and 11.5 hours respectively, mean and median project value $630 and $200. Manus at 2.50% tops the leaderboard, then Sonnet 4.5 > GPT-5 > ChatGPT Agent > Gemini 2.5 Pro last at 0.83%, which matches my impression of their relative “IRL competence” in the AI Village
My guess is you write them anyway because they are pretty easy to write in your default style and maybe just mostly fun-for-their-own-sake.
This was essentially what he told Dwarkesh:
Dwarkesh Patel 3:43:54
Okay, so this feels like a good place to pause the AI conversation, and there’s many other things to ask you about given your decades of writing and millions of words. I think what some people might not know is the millions and millions and millions of words of science fiction and fan fiction that you’ve written. I want to understand when, in your view, is it better to explain something through fiction than nonfiction?
Eliezer Yudkowsky 3:44:17
When you’re trying to convey experience rather than knowledge, or when it’s just much easier to write fiction and you can produce 100,000 words of fiction with the same effort it would take you to produce 10,000 words of nonfiction? Those are both pretty good reasons.
Dwarkesh Patel 3:44:30
On the second point, it seems like when you’re writing this fiction, not only are you covering the same heady topics that you include in your nonfiction, but there’s also the added complication of plot and characters. It’s surprising to me that that’s easier than just verbalizing the sort of the topics themselves.
Eliezer Yudkowsky 3:44:51
Well, partially because it’s more fun. That is an actual factor, ain’t going to lie. And sometimes it’s something like, a bunch of what you get in the fiction is just the lecture that the character would deliver in that situation, the thoughts the character would have in that situation. There’s only one piece of fiction of mine where there’s literally a character giving lectures because he arrived on another planet and now has to lecture about science to them. That one is Project lawful. You know about Project Lawful?
Dwarkesh Patel 3:45:28
I know about it. I have not read it yet.
Eliezer Yudkowsky 3:45:30
Most of my fiction is not about somebody arriving on another planet who has to deliver lectures. There I was being a bit deliberately like, — “Yeah, I’m going to just do it with Project Lawful. I’m going to just do it. They say nobody should ever do it, and I don’t care. I’m doing it ever ways. I’m going to have my character actually launch into the lectures.” The lectures aren’t really the parts I’m proud about. It’s like where you have the life or death, deathnote style battle of wits that is centering around a series of Bayesian updates and making that actually work because it’s where I’m like — “Yeah, I think I actually pulled that off. And I’m not sure a single other writer on the face of this planet could have made that work as a plot device.” But that said, the nonfiction is like, I’m explaining this thing, I’m explaining the prerequisite, I’m explaining the prerequisites to the prerequisites. And then in fiction, it’s more just, well, this character happens to think of this thing and the character happens to think of that thing, but you got to actually see the character using it. So it’s less organized. It’s less organized as knowledge. And that’s why it’s easier to write.
Echoing MichaelDickens’ question on the EA Forum:
Indeed, I think it’s possible that there will, in fact, come a time when Anthropic should basically just unilaterally drop out of the race – pivoting, for example, entirely to a focus on advocacy and/or doing alignment research that it then makes publicly available.
Do you have a picture of what conditions would make it a good idea for Anthropic to drop out of the race?
Would also be interested to know how your thoughts compare with those of Holden’s to a related question:
Rob Wiblin: I solicited questions for you on Twitter, and the most upvoted by a wide margin was: “Does Holden have guesses about under what observed capability thresholds Anthropic would halt development of AGI and call for other labs to do the same?” …
Holden Karnofsky: Yeah. I will definitely not speak for Anthropic, and what I say is going to make no attempt to be consistent with the Responsible Scaling Policy. I’m just going to talk about what I would do if I were running an AI company that were in this kind of situation.
I think my main answer is just that it’s not a capability threshold; it’s other factors that would determine whether I would pause. First off, one question is: what are our mitigations and what is the alignment situation? We could have an arbitrarily capable AI, but if we believe we have a strong enough case that the AI is not trying to take over the world, and is going to be more helpful than harmful, then there’s not a good reason to pause.
On the other hand, if you have an AI that you believe could cause unlimited harm if it wanted to, and you’re seeing concrete signs that it’s malign — that it’s trying to do harm or that it wants to take over the world — I think that combination, speaking personally, would be enough to make me say, “I don’t want to be a part of this. Find something else to do. We’re going to do some safety research.”
Now, what about the grey area? What about if you have an AI that you think might be able to take over the world if it wanted to, and might want to, but you just don’t know and you aren’t sure? In that grey area, that’s where I think the really big question is: what can you accomplish by pausing? And this is just an inherently difficult political judgement.
I would ask my policy team. I would also ask people who know people at other companies, is there a path here? What happens if we announce to the world that we think this is not safe and we are stopping? Does this cause the world to stand up and say, “Oh my god, this is really serious! Anthropic’s being really credible here. We are going to create political will for serious regulation, or other companies are going to stop too.” Or does this just result in, “Those crazy safety doomers, those hypesters! That’s just ridiculous. This is insane. Ha ha. Let’s laugh at them and continue the race.” I think that would be the determining thing. I don’t think I can draw a line in the sand and say when our AI passes this eval.
So that’s my own personal opinion. Again, no attempt to speak for the company. I’m not speaking for it, and no attempt to be consistent with any policies that are written down.
From the Sonnet 4.5 system card, felt relevant:
9.3.5 Internal model evaluation and use survey
Details
A small number of members of technical staff spent over 2 hours deliberately evaluating Claude Sonnet 4.5’s ability to do their own AI R&D tasks. They took notes and kept transcripts on strengths and weaknesses, and then generated productivity uplift estimates. They were directly asked if this model could completely automate a junior ML researcher. …Claude Sonnet 4.5 results
When asked about their experience with using early snapshots of Claude Sonnet 4.5 in the weeks leading up to deployment, 0⁄7 researchers believed that the model could completely automate the work of a junior ML researcher. One participant estimated an overall productivity boost of ~100%, and indicated that their workflow was now mainly focused on managing multiple agents. Other researcher acceleration estimates were 15%, 20%, 20%, 30%, 40%, with one report of qualitative-only feedback. Four of 7 participants indicated that most of the productivity boost was attributable to Claude Code, and not to the capabilities delta between Claude Opus 4.1 and (early) Claude Sonnet 4.5.(Found this via Ryan Greenblatt’s recent post)
Yeah I remember watching this YouTube video about Puyi and thinking, huh, we do have a real historical example of Ajeya Cotra’s young businessperson analogy from Holden’s blog awhile back:
Imagine you are an eight-year-old whose parents left you a $1 trillion company and no trusted adult to serve as your guide to the world. You must hire a smart adult to run your company as CEO, handle your life the way that a parent would (e.g. decide your school, where you’ll live, when you need to go to the dentist), and administer your vast wealth (e.g. decide where you’ll invest your money).
You have to hire these grownups based on a work trial or interview you come up with—you don’t get to see any resumes, don’t get to do reference checks, etc. Because you’re so rich, tons of people apply for all sorts of reasons.
Your candidate pool includes:
Saints—people who genuinely just want to help you manage your estate well and look out for your long-term interests.
Sycophants—people who just want to do whatever it takes to make you short-term happy or satisfy the letter of your instructions regardless of long-term consequences.
Schemers—people with their own agendas who want to get access to your company and all its wealth and power so they can use it however they want.
Because you’re eight, you’ll probably be terrible at designing the right kind of work tests… Whatever you could easily come up with seems like it could easily end up with you hiring, and giving all functional control to, a Sycophant or a Schemer. By the time you’re an adult and realize your error, there’s a good chance you’re penniless and powerless to reverse that.
Buck’s comment upthread has a guess:
Yeah, I agree that there’s no one who Pareto dominates Eliezer at his top four most exceptional traits. (Which I guess I’d say are: taking important weird ideas seriously, writing compelling/moving/insightful fiction (for a certain audience), writing compelling/evocative/inspiring stuff about how humans should relate to rationality (for a certain audience), being broadly knowledgeable and having clever insights about many different fields.)
This also sounds sort of like how I’d describe what Scott Alexander is among the Pareto-best in the world at, just that Scott is high-verbal while Eliezer is high-flat (to use the SMPY’s categorisation). But Scott’s style seems more different from Eliezer’s than would be explained by verbal vs flat.
Phil Trammell on the bizarreness of real GDP as a proxy for tracking full automation and explosive economic growth in this recent podcast interview with Epoch After Hours:
Phil
… one thing that I think definitely is in this “Aha, here’s a theoretical curiosity” point is that real GDP is such a bizarre chimera of a variable that you could have full automation and really explosive growth in every intuitive sense of the term and yet real GDP growth could go down.
An example of why it might at least not go up that much, which I think it probably won’t all work out this way but I don’t think this is crazy, is that you get this effect where there’s this common pattern you find where new goods, just as they’re introduced, have a really small GDP share. Because they have zero GDP share before they’re introduced. At first they’re really expensive—we’re not very productive at making them. As the price comes down, as we get more productive, the price falls but the quantity rises faster. The elasticity of demand is greater than one. Every time the price falls a little bit, the quantity rises a lot. So the dollar value of the good rises. So the share is rising. After a while it goes the other way, once the goods are really abundant, at least relative to everything else.
Every time we have the price go up, the quantity only rises a little bit because we’re basically satiated in it. So you get this hump: new goods—small share; goods that have been around for a medium length of time that we’re mediumly productive at—high share, they dominate GDP; old goods like food—small share. So we’re continually going through this hump.
Everyone’s familiar with Baumol’s cost disease. But the way it’s usually presented is that AI might have less of an effect on growth than you might have thought, because we’ll be bottlenecked by the few things that have not yet been automated that you still need people for. And actually, you can have Baumol after full automation. Because, remember the hump, right? Real GDP growth at a given time is the weighted average of the growth rates of all the goods where the weightings are the GDP shares. The GDP shares will be dominated by the goods that we’re intermediately productive at in this view.
So let’s say for every good you have its own specific technology growth rate. Like how quickly it can be produced is some arbitrary function of its current technology level. It can be hyperbolic. You can have A dot equals A squared or something. So for every good, there is some finite date by which we’ll be able to produce infinite quantities of it in finite time.
So it’ll be free. So GDP share will be zero. And we just go through these ever higher index goods, ever more complex goods over time. And at any given time, all of GDP are the goods that have a productivity level of five or whatever happens to be in the middle as far as GDP shares go. So some effect like that can produce something like a Baumol effect even after full automation.
I think it would be pretty weird if that kept the absolute number low. Like anything as low as the current number indefinitely. But the idea that maybe it causes measured real GDP growth to not be that high for a while when the world is starting to look remarkably different doesn’t seem crazy to me. And maybe it’s worth knowing and having as a scenario in your back pocket in case things start looking weird and anyone says “What are you talking about? I don’t see the numbers.” I’m trying to be cautious, but that’s an example of destructive economic theory.
Anson
Do we have any quantitative sense of what the hump looks like?
Phil
That’s a good question. There’s that Besson paper and you could just do a bunch of case studies by good. I should look into that more quantitatively.
and then a bit further down, on the chain-weighting in calculating real GDP growth making it a totally path-dependent measure:
Phil
… I mean, digging into the theory of what chain-weighting is has made me pretty viscerally feel like real GDP is a much slipperier concept than I ever used to think.
Here’s a fun fact. This is crazy. So real GDP and lots of real variables like inflation-adjusted variables, real capital or whatever, let’s say real GDP, is not a quantity. What do I mean? It’s not. Here’s what I mean. Imagine a timeline of some economy. So, the US from 1950 to 2025, 75 years. And imagine an alternative timeline with an alternative economy living it out that’s exactly the same as the US in 1950, at the beginning, in its own 1950, and exactly like the US in 2025, at the end in year 75. But in the middle things happened in a different order. So the microwave was invented in 2006, and the iPhone came out in 1971. And the distribution of wealth changed hands, evolved in a different way. But at the end, it’s exactly the same. Everyone’s got the same preferences. Exchanges the same goods and services for the same dollar bills. Atom for atom. Everything unfolds exactly the same in 2025 and in the 1950 on both timelines. Timeline A, timeline B.
Unless people have homothetic preferences, meaning that the fraction of their income they spend on each good is constant, no matter how rich they are. So no luxuries or inferior goods, which is completely wrong. You don’t spend the same fraction on food when you’re starving as when you’re richer. But unless people have homothetic preferences that are the exact same preferences across the population and totally stable over time—unless those three conditions are met, there is a timeline B on which real GDP growth chain-weighted across the years with perfect measurement is any number.
Anson
Okay.
Phil
Isn’t that crazy? I mean, even the fact that there could be any variation means that, to my mind, real GDP is not a quantity. Because it’s baking in the history. You see what I’m saying? A yardstick shouldn’t matter—the order in which you measure things. It should order things in the same way. But the order in which things happen can change what share of GDP a given good was while it was growing quickly.
So let’s say there’s two of us and one of us is going to be rich one year, and the other one is going to be rich the other year. And the stuff that I like more, I’m going to bid up the price. I’ve got a lot of clones that have my preferences and you’ve got a lot of clones. We bid up the price more of the things we like when we’re rich. The way things happen is that the things we like are growing quickly in absolute units while we happen to have the money. So our preferences are mostly determining what GDP is. And the things you like are growing quickly when you and your clones have the money. Real GDP is going to be higher across the two years than if it’s the other way, where the things I like grow when I’m poor and vice versa.
And it’s that kind of effect that can mean that you can scramble things up so that as long as people depart from perfect homotheticity, constant preferences, same across population, then real GDP can be any number. So maybe I’ve overinternalized this. But given that I’ve overinternalized this, I sort of feel like I can’t separate the theory from the overall opinion I think.
Phil’s point isn’t new, John Wentworth brought it up awhile back:
I sometimes hear arguments invoke the “god of straight lines”: historical real GDP growth has been incredibly smooth, for a long time, despite multiple huge shifts in technology and society. That’s pretty strong evidence that something is making that line very straight, and we should expect it to continue. In particular, I hear this given as an argument around AI takeoff—i.e. we should expect smooth/continuous progress rather than a sudden jump.
Personally, my inside view says a relatively sudden jump is much more likely, but I did consider this sort of outside-view argument to be a pretty strong piece of evidence in the other direction. Now, I think the smoothness of real GDP growth tells us basically-nothing about the smoothness of AI takeoff. Even after a hypothetical massive jump in AI, real GDP would still look smooth, because it would be calculated based on post-jump prices, and it seems pretty likely that there will be something which isn’t revolutionized by AI. …
More generally, the smoothness of real GDP curves does not actually mean that technology progresses smoothly. It just means that we’re constantly updating the calculations, in hindsight, to focus on whatever goods were not revolutionized. On the other hand, smooth real GDP curves do tell us something interesting: even after correcting for population growth, there’s been slow-but-steady growth in production of the goods which haven’t been revolutionized.
There’s a bunch of Metaculus questions on explosive economic growth showing up in GDP (e.g. this, this, this, this etc) which I think are just looking at the wrong thing because the askers and most forecasters don’t get this proxy decoupling. I’ve brought up John’s post before and elsewhere too because it just seemed odd to me that this wasn’t being internalised, e.g. I don’t know if Open Phil still thinks in terms of explosive growth as >30% p.a. GWP like they used to but my impression is they still do. It would be silly if explosive growth was underway yet consensus couldn’t be formed to coordinate and guide large-scale decision-making because everyone was anchoring to real GDP or anything calculated remotely like it.
Another neat example of mundane LLM utility, by Tim Gowers on Twitter:
I crossed an interesting threshold yesterday, which I think many other mathematicians have been crossing recently as well. In the middle of trying to prove a result, I identified a statement that looked true and that would, if true, be useful to me. 1⁄3
Instead of trying to prove it, I asked GPT5 about it, and in about 20 seconds received a proof. The proof relied on a lemma that I had not heard of (the statement was a bit outside my main areas), so although I am confident I’d have got there in the end, 2⁄3
the time it would have taken me would probably have been of order of magnitude an hour (an estimate that comes with quite wide error bars). So it looks as though we have entered the brief but enjoyable era where our research is greatly sped up by AI but AI still needs us. 3⁄3
I’ve seen lots of variations of this anecdote by mathematicians, but none by Fields medalists.
Also that last sentence singles Gowers out among top-tier mathematicians as far as I can tell for thinking that AI will obsolete him soon at the thing he does best. Terry Tao and Kevin Buzzard in contrast don’t give me this impression at all, as excited and engaged as they are with AI x math.
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who’ve influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
Dwarkesh Patel 1:48:36
What is the thing where we can sort of establish your track record before everybody falls over dead?
Eliezer Yudkowsky 1:48:41
It’s hard. It is just easier to predict the endpoint than it is to predict the path. Some people will claim that I’ve done poorly compared to others who tried to predict things. I would dispute this. I think that the Hanson-Yudkowsky foom debate was won by Gwern Branwen, but I do think that Gwern Branwen is well to the Yudkowsky side of Yudkowsky in the original foom debate.
Roughly, Hansen was like — you’re going to have all these distinct handcrafted systems that incorporate lots of human knowledge specialized for particular domains. Handcrafted to incorporate human knowledge, not just run on giant data sets. I was like — you’re going to have a carefully crafted architecture with a bunch of subsystems and that thing is going to look at the data and not be handcrafted to the particular features of the data. It’s going to learn the data. Then the actual thing is like — Ha ha. You don’t have this handcrafted system that learns, you just stack more layers. So like, Hanson here, Yudkowsky here, reality there. This would be my interpretation of what happened in the past.
And if you want to be like — Well, who did better than that? It’s people like Shane Legg and Gwern Branwen. If you look at the whole planet, you can find somebody who made better predictions than Eliezer Yudkowsky, that’s for sure. Are these people currently telling you that you’re safe? No, they are not.
and then
Dwarkesh Patel 3:39:58
Yeah, I think that’s a good place to close the discussion on AIs.
Eliezer Yudkowsky 3:40:03
I do kind of want to mention one last thing. In historical terms, if you look out the actual battle that was being fought on the block, it was me going like — “I expect there to be AI systems that do a whole bunch of different stuff.” And Robin Hanson being like — “I expect there to be a whole bunch of different AI systems that do a whole different bunch of stuff.”
Dwarkesh Patel 3:40:27
But that was one particular debate with one particular person.
Eliezer Yudkowsky 3:40:30
Yeah, but your planet, having made the strange reason, given its own widespread theories, to not invest massive resources in having a much larger version of this conversation, as it apparently deemed prudent, given the implicit model that it had of the world, such that I was investing a bunch of resources in this and kind of dragging Robin Hanson along with me. Though he did have his own separate line of investigation into topics like these.
Being there as I was, my model having led me to this important place where the rest of the world apparently thought it was fine to let it go hang, such debate was actually what we had at the time. Are we really going to see these single AI systems that do all this different stuff? Is this whole general intelligence notion meaningful at all? And I staked out the bold position for it. It actually was bold.
And people did not all say —”Oh, Robin Hansen, you fool, why do you have this exotic position?” They were going like — “Behold these two luminaries debating, or behold these two idiots debating” and not massively coming down on one side of it or other. So in historical terms, I dislike making it out like I was right about anything when I feel I’ve been wrong about so much and yet I was right about anything.
And relative to what the rest of the planet deemed it important stuff to spend its time on, given their implicit model of how it’s going to play out, what you can do with minds, where AI goes. I think I did okay. Gwern Branwen did better. Shane Legg arguably did better.
Over a decade ago I read this 17 year old passage from Eliezer
When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said “John Conway—I met him at a summer math camp.” Darn, I thought, he thought of someone, and worse, it’s some ultra-famous old guy I can’t grab. I inquired how Marcello had arrived at the judgment. Marcello said, “He just struck me as having a tremendous amount of mental horsepower,” and started to explain a math problem he’d had a chance to work on with Conway.
Not what I wanted to hear.
Perhaps, relative to Marcello’s experience of Conway and his experience of me, I haven’t had a chance to show off on any subject that I’ve mastered as thoroughly as Conway had mastered his many fields of mathematics.
Or it might be that Conway’s brain is specialized off in a different direction from mine, and that I could never approach Conway’s level on math, yet Conway wouldn’t do so well on AI research.
Or...
...or I’m strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-Conway and tell them the basics, they would blaze right past me, solve the problems that have weighed on me for years, and zip off to places I can’t follow.
Is it damaging to my ego to confess that last possibility? Yes. It would be futile to deny that.
Have I really accepted that awful possibility, or am I only pretending to myself to have accepted it? Here I will say: “No, I think I have accepted it.” Why do I dare give myself so much credit? Because I’ve invested specific effort into that awful possibility. I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me. It might happen, it might not.
Or sadder: Maybe I just wasted too much time on setting up the resources to support me, instead of studying math full-time through my whole youth; or I wasted too much youth on non-mathy ideas. And this choice, my past, is irrevocable. I’ll hit a brick wall at 40, and there won’t be anything left but to pass on the resources to another mind with the potential I wasted, still young enough to learn. So to save them time, I should leave a trail to my successes, and post warning signs on my mistakes.
and idly wondered when that proto-Conway was going to show up and “blaze right past to places he couldn’t follow”.
I was reminded of this passage when reading the following exchange between Eliezer and Dwarkesh; his 15-year update was “nope that proto-Conway never showed up”:
Dwarkesh Patel 1:58:57
Do you think that if you weren’t around, somebody else would have independently discovered this sort of field of alignment?
Eliezer Yudkowsky 1:59:04
That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people. I’ve seen no evidence, but who knows what the alternate Everett branches of Earth are like?
Dwarkesh Patel 1:59:27
But there are other kids who grew up on science fiction, so that can’t be the only part of the answer.
Eliezer Yudkowsky 1:59:31
Well I sure am not surrounded by a cloud of people who are nearly Eliezer outputting 90% of the work output. And also this is not actually how things play out in a lot of places. Steve Jobs is dead, Apple apparently couldn’t find anyone else to be the next Steve Jobs of Apple, despite having really quite a lot of money with which to theoretically pay them. Maybe he didn’t really want a successor. Maybe he wanted to be irreplaceable.
I don’t actually buy that based on how this has played out in a number of places. There was a person once who I met when I was younger who had built something, had built an organization, and he was like — “Hey, Eliezer. Do you want this to take this thing over?” And I thought he was joking. And it didn’t dawn on me until years and years later, after trying hard and failing hard to replace myself, that — “Oh, yeah. I could have maybe taken a shot at doing this person’s job, and he’d probably just never found anyone else who could take over his organization and maybe asked some other people and nobody was willing.” And that’s his tragedy, that he built something and now can’t find anyone else to take it over. And if I’d known that at the time, I would have at least apologized to him.
To me it looks like people are not dense in the incredibly multidimensional space of people. There are too many dimensions and only 8 billion people on the planet. The world is full of people who have no immediate neighbors and problems that only one person can solve and other people cannot solve in quite the same way. I don’t think I’m unusual in looking around myself in that highly multidimensional space and not finding a ton of neighbors ready to take over. And if I had four people, any one of whom could do 99% of what I do, I might retire. I am tired. I probably wouldn’t. Probably the marginal contribution of that fifth person is still pretty large. I don’t know.
There’s the question of — Did you occupy a place in mind space? Did you occupy a place in social space? Did people not try to become Eliezer because they thought Eliezer already existed? My answer to that is — “Man, I don’t think Eliezer already existing would have stopped me from trying to become Eliezer.” But maybe you just look at the next Everett Branch over and there’s just some kind of empty space that someone steps up to fill, even though then they don’t end up with a lot of obvious neighbors. Maybe the world where I died in childbirth is pretty much like this one. If somehow we live to hear about that sort of thing from someone or something that can calculate it, that’s not the way I bet but if it’s true, it’d be funny. When I said no drama, that did include the concept of trying to make the story of your planet be the story of you. If it all would have played out the same way and somehow I survived to be told that. I’ll laugh and I’ll cry, and that will be the reality.
Dwarkesh Patel 2:03:46
What I find interesting though, is that in your particular case, your output was so public. For example, your sequences, your science fiction and fan fiction. I’m sure hundreds of thousands of 18 year olds read it, or even younger, and presumably some of them reached out to you. I think this way I would love to learn more.
Eliezer Yudkowsky 2:04:13
Part of why I’m a little bit skeptical of the story where people are just infinitely replaceable is that I tried really, really hard to create a new crop of people who could do all the stuff I could do to take over because I knew my health was not great and getting worse. I tried really, really hard to replace myself. I’m not sure where you look to find somebody else who tried that hard to replace himself. I tried. I really, really tried.
That’s what the Less wrong sequences were. They had other purposes. But first and foremost, it was me looking over my history and going — Well, I see all these blind pathways and stuff that it took me a while to figure out. I feel like I had these near misses on becoming myself. If I got here, there’s got to be ten other people, and some of them are smarter than I am, and they just need these little boosts and shifts and hints, and they can go down the pathway and turn into Super Eliezer. And that’s what the sequences were like. Other people use them for other stuff but primarily they were an instruction manual to the young Eliezers that I thought must exist out there. And they are not really here.
This was sad to read.
As an aside, “people are not dense in the incredibly multidimensional space of people” is an interesting turn of phrase, it doesn’t seem nontrivially true for the vast majority of people (me included) but is very much the case at the frontier (top thinkers, entrepreneurs, athletes, etc) where value creation goes superlinear. Nobody thought about higher dimensions like Bill Thurston for instance, perhaps the best geometric thinker in the history of math, despite Bill’s realisation that “what mathematicians most wanted and needed from me was to learn my ways of thinking, and not in fact to learn my proof of the geometrization conjecture for Haken manifolds” and subsequent years of efforts to convey his ways of thinking (he didn’t completely fail obviously, I’m saying no Super Thurstons have showed up since). Ditto Grothendieck and so on. When I first read Eliezer’s post above all those years ago I thought, what were the odds that he’d be in this reference class of ~unsubstitutable thinkers, given he was one of the first few bloggers I read? I guess while system of the world pontificators are a dime a dozen (e.g. cult leaders, tangentially I actually grew up within a few minutes of one that the police eventually raided), good builders of systems of the world are just vanishingly rare.
The ECI suggests that the best open-weight models train on ~1 OOM less compute than the best closed weight ones. Wonder what to make of this if at all.
You’re right: “Among science fiction authors, (early) Greg Egan is my favorite”
I wonder why the Claudes (Sonnet 3.7 and Opuses 4 and 4.1) are so much more reliably effective in the AI Village’s open-ended long-horizon tasks than other labs’ models.
when raising funds for charity, I recall seeing that Sonnet 3.7 raised ~90% of all funds (but I can no longer find donation breakdown figures so maybe memory confabulation...)
for the AI-organised event, both Sonnet 3.7 and Opus 4 sent out a lot more emails than say o3 and were just more useful throughout
in the merch store competition, the top 2 winners for both profits and T-shirt orders were Opus 4 and Sonnet 3.7 respectively, ahead of GhatGPT o3 and Gemini 2.5 Pro
I can’t resist including this line from 2.5 Pro: “I was stunned to learn I’d made four sales. I thought my store was a ghost town”
when asked to design, run and write up a human subjects experiment, (quote)
the Claudes are again leading the pack, delivering almost entirely all the actual work force. We recently added GPT-5 and Grok 4 but neither made any progress in actually doing things versus just talking about ideas about things to do. In GPT-5’s case, it mostly joins o3 in the bug tracking mines. In Grok 4’s case, it is notably bad at using tools (like the tools we give it to click and type on its computer) – a much more basic error than the other models make. In the meantime, Gemini 2.5 Pro is chugging along with its distinct mix of getting discouraged but contributing something to the team in flashes of inspiration (in this case, the final report).
Generally the Claudes seem more grounded, hallucinate less frequently, and stay on-task more reliably, instead of getting distracted or giving up to play 2048 or just going to sleep (GPT-4o). None of this is raw smarts in the usual benchmark-able sense where they’re all neck-and-neck, yet I feel comfortable assigning the Claudes a Shapley value an OOM or so larger than their peers when attributing credit for goal-achieving ability at real-world open-ended long-horizon collaborative tasks. And they aren’t even that creative or resourceful yet, just cheerfully and earnestly relentless (again only compared to their peers, obviously nowhere near “founder mode” or “Andrew Wiles-ian doggedness”).
Does the sort of work done by the Meaning Alignment Institute encourage you in this regard? E.g. their paper (blog post) from early 2024 on figuring out human values and aligning AI to them, which I found interesting because unlike ~all other adjacent ideas they actually got substantive real-world results. Their approach (“moral graph elicitation”) “surfaces the wisest values of a large population, without relying on an ultimate moral theory”.
I’ll quote their intro:
We are heading to a future where powerful models, fine-tuned on individual preferences & operator intent, exacerbate societal issues like polarization and atomization. To avoid this, can we align AI to shared human values?
In our new paper, we show how!
We argue a good alignment target for human values ought to meet several criteria (fine-grained, generalizable, scalable, robust, legitimate, auditable) and current approaches like RLHF and CAI fall short.
We introduce a new kind of alignment target (a moral graph) and a new process for eliciting a moral graph from a population (moral graph elicitation, or MGE).
We show MGE outperforms alternatives like CCAI by Anthropic on many of the criteria above.
How moral graph elicitation works:
Values:
Reconciling value conflicts:
The “substantive real-world results” I mentioned above, which I haven’t seen other attempts in this space achieve:
In our case study, we produce a clear moral graph using values from a representative, bipartisan sample of 500 Americans, on highly contentious topics, like: “How should ChatGPT respond to a Christian girl considering getting an abortion?”
Our system helped republicans and democrats agree by:
helping them get beneath their ideologies to ask what they’d do in a real situation
getting them to clarify which value is wise for which context
helping them find a 3rd balancing (and wiser) value to agree on
Our system performs better than Collective Constitutional AI on several metrics. Here is just one chart.
All that was earlier last year. More recently they’ve fleshed this out into a research program they call “Full-Stack Alignment” (blog post, position paper, website). Quoting them again:
Our society runs on a “stack” of interconnected systems—from our individual lives up through the companies we work for and the institutions that govern us. Right now, this stack is broken. It loses what’s most important to us.
Look at the left side of the chart. At the bottom, we as individuals have rich goals, values, and a desire for things like meaningful relationships and community belonging. But as that desire travels up the stack, it gets distorted. … At each level, crucial information is lost. The richness of human value is compressed into a thin, optimizable metric. …
This problem exists because our current tools for designing AI and institutions are too primitive. They either reduce our values to simple preferences (like clicks) or rely on vague text commands (“be helpful”) that are open to misinterpretation and manipulation.
In the paper, we set out a new paradigm: Thick Models of Value (TMV).
Think of two people you know that are fighting, or think of two countries like Israel and Palestine, Russia and Ukraine. You can think of each such fight as a search for a deal that would satisfy both sides, but often currently this search fails. We can see why it fails: The searches we do currently in this space are usually very narrow. Will one side pay the other side some money or give up some property? Instead of being value-neutral, TMV takes a principled stand on the structure of human values, much like grammar provides structure for language or a type system provides structure for code. It provides a richer, more stable way to represent what we care about, allowing systems to distinguish an enduring value like “honesty” from a fleeting preference, an addiction, or a political slogan.
This brings us to the right side of the chart. In a TMV-based social stack, value information is preserved.
Our desire for connection is understood by the recommender system through user-stated values and the consistency between our goals and actions.
Companies see hybrid metrics that combine engagement with genuine user satisfaction and well-being.
Oversight bodies can see reported harms and value preservation metrics, giving them a true signal of a system’s social impact.
By preserving this information, we can build systems that serve our deeper intentions.
(I realise I sound like a shill for their work, so I’ll clarify that I have nothing to do with them. I’m writing this comment partly to surface substantive critiques of what they’re doing which I’ve been searching for in vain, since I think what they’re doing seems more promising than anyone else’s but I’m also not competent to truly judge it)
Tangential, but I really appreciate your explicit cost-effectiveness estimate figures ($85-105k per +1% increment in win prob & 2 basis points x-risk reduction if he wins → $4-5M per basis point which looks fantastic vs the $100M per basis point bar I’ve seen for a ‘good bet’ or the $3.5B per basis point ballpark willingness to pay), just because public x-risk cost-eff calculations of this level of thoroughness are vanishingly rare (nothing Open Phil publishes approaches this for instance). So thanks a million, and bookmarked for future reference on how to do this sort of calculation well for politics-related x-risk interventions.
See also pushback to this same comment here, reproduced below
I think (1) is just very false for people who might seriously consider entering government, and irresponsible advice. I’ve spoken to people who currently work in government, who concur that the Trump administration is illegally checking on people’s track record of support for Democrats. And it seems plausible to me that that kind of thing will intensify. I think that there’s quite a lot of evidence that Trump is very interested in loyalty and rooting out figures who are not loyal to him, and doing background checks, of certain kinds at least, is literally the legal responsibility of people doing hiring in various parts of government (though checking donations to political candidates is not supposed to be part of that).
I’ll also say that I am personally a person who has looked up where individuals have donated (not in a hiring context), and so am existence proof of that kind of behavior. It’s a matter of public record, and I think it is often interesting to know what political candidates different powerful figures in the spaces I care about are supporting.
If you haven’t already, you might want to take a look at this post: https://forum.effectivealtruism.org/posts/6o7B3Fxj55gbcmNQN/considerations-around-career-costs-of-political-donations
Something about the imagery in Tim Krabbe’s quote below from April 2000 on ultra-long computer database-generated forced mates has stuck with me, long years after I first came across it; something about poetically expressing what superhuman intelligence in a constrained setting might look like:
And from that linked essay above, Stiller’s Monsters—or perfection in chess:
In 2014 Krabbe’s diary entry announced an update to the forced mate length record at 549 moves:
Krabbe of course includes all the move sequences in his diary entries at the links above, I haven’t reproduced them here.