Long-time lurker (c. 2013), recent poster. I also write on the EA Forum.
Mo Putera
Karma: 1,722
… moving away from “AI” and “AGI” as terms to talk about. I feel they are so old and overloaded with contradictory meanings that it would be better to start over fresh.
I interpret Holden Karnofsky’s PASTA from 2021 in the same vein (emphasis his):
This piece is going to focus on exploring a particular kind of AI I believe could be transformative: AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement. I will call this sort of technology Process for Automating Scientific and Technological Advancement, or PASTA.3 (I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation.)
(Note how Holden doesn’t care that the AI system be singular, unlike say the Metaculus AGI definition.) He continued (again emphasis his):
PASTA could resolve the same sort of bottleneck discussed in The Duplicator and This Can’t Go On—the scarcity of human minds (or something that plays the same role in innovation).
PASTA could therefore lead to explosive science, culminating in technologies as impactful as digital people. And depending on the details, PASTA systems could have objectives of their own, which could be dangerous for humanity and could matter a great deal for what sort of civilization ends up expanding through the galaxy.
By talking about PASTA, I’m partly trying to get rid of some unnecessary baggage in the debate over “artificial general intelligence.” I don’t think we need artificial general intelligence in order for this century to be the most important in history. Something narrower—as PASTA might be—would be plenty for that.
When I read that last paragraph, I thought, yeah this seems like the right first-draft operational definition of “transformative AI”, and I anticipated it to gradually disseminate into the broader conversation and be further refined, also because the person proposing this definition was Holden instead of some random alignment researcher or whatever. Instead it seems(?) mostly ignored, at least outside of Open Phil, which I still find confusing.
I’m not sure how you’re thinking about OSIs, would you say they’re roughly in line with what Holden meant above?
Separately, I do however think that the right operationalisation of AGI-in-particular isn’t necessarily Holden’s, but Steven Byrnes’. I like that entire subsection, so let me share it here in full:
A frequent point of confusion is the word “General” in “Artificial General Intelligence”:
The word “General” DOES mean “not specific”, as in “In general, Boston is a nice place to live.”
The word “General” DOES NOT mean “universal”, as in “I have a general proof of the math theorem.”
An AGI is not “general” in the latter sense. It is not a thing that can instantly find every pattern and solve every problem. Humans can’t do that either! In fact, no algorithm can, because that’s fundamentally impossible. Instead, an AGI is a thing that, when faced with a difficult problem, might be able to solve the problem easily, but if not, maybe it can build a tool to solve the problem, or it can find a clever way to avoid the problem altogether, etc.
Consider: Humans wanted to go to the moon, and then they figured out how to do so, by inventing extraordinarily complicated science and engineering and infrastructure and machines. Humans don’t have a specific evolved capacity to go to the moon, akin to birds’ specific evolved capacity to build nests. But they got it done anyway, using their “general” ability to figure things out and get things done.
So for our purposes here, think of AGI as an algorithm which can “figure things out” and “understand what’s going on” and “get things done”, including using language and science and technology, in a way that’s reminiscent of how most adult humans (and groups and societies of humans) can do those things, but toddlers and chimpanzees and today’s large language models (LLMs) can’t. Of course, AGI algorithms may well be subhuman in some respects and superhuman in other respects.
This image is poking fun at Yann LeCun’s frequent talking point that “there is no such thing as Artificial General Intelligence”. (Image sources: 1,2) Anyway, this series is about brain-like algorithms. These algorithms are by definition capable of doing absolutely every intelligent behavior that humans (and groups and societies of humans) can do, and potentially much more. So they can definitely reach AGI. Whereas today’s AI algorithms are not AGI. So somewhere in between here and there, there’s a fuzzy line that separates “AGI” from “not AGI”. Where exactly is that line? My answer: I don’t know, and I don’t care. Drawing that line has never come up for me as a useful thing to do.
It seems entirely possible for a collection of AI systems to be a civilisation-changing PASTA without being at all Byrnes-general, and also possible for a Byrnes-general algorithm to be below average human intelligence let alone be a PASTA.
subscribe to receive new posts!
(I didn’t see a link, I suppose you mean your Substack)
I’m reminded of Scott’s parable below from his 2016 book review of Hanson’s Age of Em, which replaces the business executives, the investors & board members, and even the consumers in your sources of economic motivation / ownership with economic efficiency-improving algorithms and robots and such. I guess I’m wondering why you think your Autofac scenario is more plausible than Scott’s dystopian rendering of Land’s vision.
There are a lot of similarities between Hanson’s futurology and (my possibly erroneous interpretation of) the futurology of Nick Land. I see Land as saying, like Hanson, that the future will be one of quickly accelerating economic activity that comes to dominate a bigger and bigger portion of our descendents’ lives. But whereas Hanson’s framing focuses on the participants in such economic activity, playing up their resemblances with modern humans, Land takes a bigger picture. He talks about the economy itself acquiring a sort of self-awareness or agency, so that the destiny of civilization is consumed by the imperative of economic growth.
Imagine a company that manufactures batteries for electric cars. The inventor of the batteries might be a scientist who really believes in the power of technology to improve the human race. The workers who help build the batteries might just be trying to earn money to support their families. The CEO might be running the business because he wants to buy a really big yacht. And the whole thing is there to eventually, somewhere down the line, let a suburban mom buy a car to take her kid to soccer practice. Like most companies the battery-making company is primarily a profit-making operation, but the profit-making-ness draws on a lot of not-purely-economic actors and their not-purely-economic subgoals.
Now imagine the company fires all its employees and replaces them with robots. It fires the inventor and replaces him with a genetic algorithm that optimizes battery design. It fires the CEO and replaces him with a superintelligent business-running algorithm. All of these are good decisions, from a profitability perspective. We can absolutely imagine a profit-driven shareholder-value-maximizing company doing all these things. But it reduces the company’s non-masturbatory participation in an economy that points outside itself, limits it to just a tenuous connection with soccer moms and maybe some shareholders who want yachts of their own.
Now take it further. Imagine there are no human shareholders who want yachts, just banks who lend the company money in order to increase their own value. And imagine there are no soccer moms anymore; the company makes batteries for the trucks that ship raw materials from place to place. Every non-economic goal has been stripped away from the company; it’s just an appendage of Global Development.
Now take it even further, and imagine this is what’s happened everywhere. There are no humans left; it isn’t economically efficient to continue having humans. Algorithm-run banks lend money to algorithm-run companies that produce goods for other algorithm-run companies and so on ad infinitum. Such a masturbatory economy would have all the signs of economic growth we have today. It could build itself new mines to create raw materials, construct new roads and railways to transport them, build huge factories to manufacture them into robots, then sell the robots to whatever companies need more robot workers. It might even eventually invent space travel to reach new worlds full of raw materials. Maybe it would develop powerful militaries to conquer alien worlds and steal their technological secrets that could increase efficiency. It would be vast, incredibly efficient, and utterly pointless. The real-life incarnation of those strategy games where you mine Resources to build new Weapons to conquer new Territories from which you mine more Resources and so on forever.
But this seems to me the natural end of the economic system. Right now it needs humans only as laborers, investors, and consumers. But robot laborers are potentially more efficient, companies based around algorithmic trading are already pushing out human investors, and most consumers already aren’t individuals – they’re companies and governments and organizations. At each step you can gain efficiency by eliminating humans, until finally humans aren’t involved anywhere.
I at first wondered whether this would count as an answer to nostalgebraist’s when will LLMs become human-level bloggers? which he asked back in March, but then upon rereading I’m less sure. I kind of buy DaemonicSigil’s top-karma response that “writing a worthwhile blog post is not only a writing task, but also an original seeing task… So the obstacle is not necessarily reasoning… but a lack of things to say”, and in this case you were clearly the one with the things to say, not Opus 4.1
What changed your mind? Any rabbit holes in particular I can go down?
(Helen, not Steven Byrnes)
(I haven’t read IABIED.) I saw your take right after reading Buck’s, so it’s interesting how his reaction was diametrically opposite yours: “I think the first two parts of the book are the best available explanation of the basic case for AI misalignment risk for a general audience. I thought the last part was pretty bad, and probably recommend skipping it.”
This reminds me of L Rudolf L’s A history of the future scenario-forecasting how math might get solved first, published all the way back in February (which now feels like an eternity ago):
A compressed version of what happened to programming in 2023-26 happens in maths in 2025-2026. The biggest news story is that GDM solves a Millennium Prize problem in an almost-entirely-AI way, with a huge amount of compute for searching through proof trees, some clever uses of foundation models for heuristics, and a few very domain-specific tricks specific to that area of maths. However, this has little immediate impact beyond maths PhDs having even more existential crises than usual.
The more general thing happening is that COT RL and good scaffolding actually is a big maths breakthrough, especially as there is no data quality bottleneck here because there’s an easy ground truth to evaluate against—you can just check the proof. AIs trivially win gold in the International Mathematical Olympiad. More general AI systems (including increasingly just the basic versions of Claude 4 or o5) generally have a somewhat-spotty version of excellent-STEM-postgrad-level performance at grinding through self-contained maths, physics, or engineering problems. Some undergrad/postgrad students who pay for the expensive models from OpenAI report having had o3 or o5 entirely or almost entirely do sensible (but basic) “research” projects for them in 2025.
Mostly by 2026 and almost entirely by 2027, the mathematical or theoretical part of almost any science project is now something you hand over to the AI, even in specialised or niche fields. …
(there’s more, but I don’t want to quote everything. Also IMO gold already happened, so nice one Rudolf)
Famously, a hundred years ago Keynes predicted that by now people would be working just fifteen hours a week. That hasn’t quite happened
This tangentially reminded me of Jason Crawford’s essay on this, which pointed to this interesting paper:
Nicholas Crafts came up with these estimates for expected lifetime hours of work for men aged 20:
Year Work hours Other hours 1881 114,491 (49%) 119,269 (51%) 1951 94,343 (33%) 191,429 (67%) 2011 70,612 (20%) 276,522 (80%) A reduction from 49% of an adult life spent working to 20% is almost as great as a reduction from forty hours a week to fifteen.
This was due to a combination of factors: working hours per week dropped by nearly half, child labor waned, and retirement was invented plus life expectancy increased.
how we reason about coherence when our reference frame shifts from the anthropocentric to the planetary, and what that implies for systems that optimize, adapt, or persist across scales.
This sounds interesting, do you have any further reading (by yourself or others) to point folks like myself to?
I am genuinely interested in reading about effective and clearly effective protests led by anyone currently doing protests, or within the last 10 years. Even if on a small scale.
Caveat that I don’t know much more than this, but I’m reminded of James Ozden’s lit reviews, e.g. How effective are protests? Some research and some nuance. Ostensibly relevant bits:
“Protest movements could be more effective than the best charities”—SSIR
About two weeks ago, I published an article in Stanford Social Innovation Review (SSIR), a magazine for those interested in philanthropy, social science and non-profits. … Although my article is reasonably brief (and I obviously recommend reading it in full!) here’s a quick summary of what I spoke about, plus some nuances I forgot or wasn’t able to add:
...
There is a reasonable amount of evidence that shows that protest movement can have significant impacts, across a variety of outcomes from policy, public opinion, public discourse, voting behaviour, and corporate behaviour. I’ll leave this point to be explained in greater detail in our summary of our literature review on protest outcomes!
...
3. A summary of Social Change Lab’s literature reviews, who we are, and our next steps
We’ve recently conducted two literature reviews, looking over 60+ academic studies across political science, sociology and economics, to tackle some key questions around protest movements. Specifically, we had two main questions:
What are the outcomes of protest and protest movements? - Literature Review
What factors make some protest movements more likely to succeed relative to others? - Literature Review
I like your posts, so it’s great that retatrutide seems to be reducing activation energy to publish them. It does however seem like it’s making you lose muscle, going by how you’ve lost 10%(!) of your bodyweight yet pushups and biking feel about the same; when I was 10% heavier than than I am now those activities felt considerably harder.
Tangentially, I was thin all my life (around 135 lbs or 62 kg) until I went to the US for undergrad, at the end of which I had ballooned to 175 lbs or nearly 80 kg. After I left the pounds rapidly melted away. None of this entailed any effort in either direction. I still find this striking: I wasn’t sedentary in college (in fact I was more physically active than I’ve been before or since), and I didn’t do late-night snacking or eat fast food except for the very occasional In-N-Out. I do remember being flabbergasted upon first encountering the serving sizes of familiar dishes at restaurants…
I just reread Scott’s review of John von Neumann’s bio The Man From The Future by Ananyo Bhattacharya, and it made me realise something else that felt off to me about the OP, which was that the OP’s insecurity seems to be primarily social status-related(?), whereas John’s seemed to be primarily existential. (This probably influenced his extreme views, like advocating for nuking Russia ASAP.) Some quotes:
[von Neumann] attributed his generation’s success to “a coincidence of some cultural factors” that produced “a feeling of extreme insecurity in the individuals, and the necessity to produce the unusual or face extinction. In other words, [the Jews’] recognition that the tolerant climate of Hungary might change overnight propelled some to preternatural efforts to succeed.
(FWIW Scott doesn’t buy this as a differentiating factor, but that’s not what I’m pointing at)
Throughout all this excellence, Bhattacharya keeps coming back to the theme of precariousness. Max von Neumann didn’t teach his kids five languages just because he wanted them to be sophisticated. He was preparing for them to have to flee Hungary in a hurry. This proved prescient; when John was fifteen, Communists took over Hungary, targeting rich families like the von Neumanns. A few months later, counterrevolutionaries defeated the Communists—then massacred thousands of Jews, who they suspected of collaborating. The von Neumanns survived by fleeing the country at opportune times, and maybe by being too rich to be credibly suspected of communist sympathies. But John’s “feeling of extreme insecurity…and…necessity to produce the unusual or face extinction” certainly wasn’t without basis. This was, perhaps, an education of a different sort.
Scott’s review also touches on the thing about advocating for nuking Russia ASAP, quoting his daughter Marina on his hatred of totalitarianism:
Throughout much of his career, he led a double life: as an intellectual leader in the ivory tower of pure mathematics and as a man of action, in constant demand as an advisor, consultant and decision-maker to what is sometimes called the military-industrial complex of the United States. My own belief is that these two aspects of his double life, his wide-ranging activities as well as his strictly intellectual pursuits, were motivated by two profound convictions. The first was the overriding responsibility that each of us has to make full use of whatever intellectual capabilities we were endowed with. He had the scientist’s passion for learning and discovery for its own sake and the genius’s ego-driven concern for the significance and durability of his own contributions. The second was the critical importance of an environment of political freedom for the pursuit of the first, and for the welfare of mankind in general.
I’m convinced, in fact, that all his involvements with the halls of power were driven by his sense of the fragility of that freedom. By the beginning of the 1930s, if not even earlier, he became convinced that the lights of civilization would be snuffed out all over Europe by the spread of totalitarianism from the right: Nazism and Fascism. So he made an unequivocal commitment to his home in the new world and to fight to preserve and reestablish freedom from that new beachhead.
In the 1940s and 1950s, he was equally convinced that the threat to civilization now came from totalitarianism on the left, that is, Soviet Communism, and his commitment was just as unequivocal to fighting it with whatever weapons lay at hand, scientific and economic as well as military. It was a matter of utter indifference to him, I believe, whether the threat came from the right or from the left. What motivated both his intense involvement in the issues of the day and his uncompromisingly hardline attitude was his belief in the overriding importance of political freedom, his strong sense of its continuing fragility, and his conviction that it was in the United States, and the passionate defense of the United States, that its best hope lay.
The Wigner quote seems a bit misleading to me because you left out the second half to make a point it didn’t support (“JvN was smarter than Einstein, and yet...”). The full quote is
I have known a great many intelligent people in my life. I knew Max Planck, Max von Laue, and Werner Heisenberg. Paul Dirac was my brother-in-Iaw; Leo Szilard and Edward Teller have been among my closest friends; and Albert Einstein was a good friend, too. And I have known many of the brightest younger scientists. But none of them had a mind as quick and acute as Jancsi von Neumann. I have often remarked this in the presence of those men, and no one ever disputed me. [...]
But Einstein’s understanding was deeper than even Jancsi von Neumann’s. His mind was both more penetrating and more original than von Neumann’s. And that is a very remarkable statement. Einstein took an extraordinary pleasure in invention. Two of his greatest inventions are the Special and General Theories of Relativity; and for all of Jancsi’s brilliance, he never produced anything so original.
I also always interpreted Wigner’s “I have often remarked this in the presence of those men, and no one ever disputed me” as referring to Planck, von Laue, Heisenberg, Dirac, Szilard, Teller, and Einstein, but not von Neumann, so your “Von Neumannn was pronounced, by a peer, to be smarter than Albert Einstein to his face and got no objection” interpretation feels off to me (but I may be wrong of course).
But that’s just nitpicking. Perhaps more substantively, I felt sad reading this section
I also work with and lead many other brilliant people who also never will be intellectual pillars of humanity and constantly feel bad about themselves for not being more brilliant. I’ve been surprised how much of my time at big companies is spent pulling people out of the pit of inadequacy and self disgust…
I’ve done a little bit of this too, although my go-to advice is to read Scott’s Parable of the Talents, in particular this passage (long quote, emphasis mine):
Every so often an overly kind commenter here praises my intelligence and says they feel intellectually inadequate compared to me, that they wish they could be at my level. But at my level, I spend my time feeling intellectually inadequate compared to Scott Aaronson. Scott Aaronson describes feeling “in awe” of Terence Tao and frequently struggling to understand him. Terence Tao – well, I don’t know if he’s religious, but maybe he feels intellectually inadequate compared to God. And God feels intellectually inadequate compared to John von Neumann.
So there’s not much point in me feeling inadequate compared to my brother, because even if I was as good at music as my brother, I’d probably just feel inadequate for not being Mozart.
And asking “Well what if you just worked harder?” can elide small distinctions, but not bigger ones. If my only goal is short-term preservation of my self-esteem, I can imagine that if only things had gone a little differently I could have practiced more and ended up as talented as my brother. It’s a lot harder for me to imagine the course of events where I do something different and become Mozart. Only one in a billion people reach a Mozart level of achievement; why would it be me?
If I loved music for its own sake and wanted to be a talented musician so I could express the melodies dancing within my heart, then none of this matters. But insofar as I want to be good at music because I feel bad that other people are better than me at music, that’s a road without an end.
This is also how I feel of when some people on this blog complain they feel dumb for not being as smart as some of the other commenters on this blog.
I happen to have all of your IQ scores in a spreadsheet right here (remember that survey you took?). Not a single person is below the population average. The first percentile for IQ here – the one such that 1% of respondents are lower and 99% of respondents are higher – is – corresponds to the 85th percentile of the general population. So even if you’re in the first percentile here, you’re still pretty high up in the broader scheme of things.
At that point we’re back on the road without end. I am pretty sure we can raise your IQ as much as you want and you will still feel like pond scum. If we raise it twenty points, you’ll try reading Quantum Computing since Democritus and feel like pond scum. If we raise it forty, you’ll just go to Terence Tao’s blog and feel like pond scum there. Maybe if you were literally the highest-IQ person in the entire world you would feel good about yourself, but any system where only one person in the world is allowed to feel good about themselves at a time is a bad system.
People say we should stop talking about ability differences so that stupid people don’t feel bad. I say that there’s more than enough room for everybody to feel bad, smart and stupid alike, and not talking about it won’t help. What will help is fundamentally uncoupling perception of intelligence from perception of self-worth.
(GPT-5 seems right here, this is Kleiber’s Law.)
re: Axler’s textbook above, also check out the paper it’s based on which is just 18 pages, Down with determinants! (I know you know this, just for others’ edification). Abstract:
This paper shows how linear algebra can be done better without determinants. The standard proof that a square matrix of complex numbers has an eigenvalue uses determinants. The simpler and clearer proof presented here provides more insight and avoids determinants. Without using determinants, this allows us to define the multiplicity of an eigenvalue and to prove that the number of eigenvalues, counting multiplicities, equals the dimension of the underlying space. Without using determinants, we can define the characteristic and minimal polynomials and then prove that they behave as expected. This leads to an easy proof that every matrix is similar to a nice upper-triangular one. Turning to inner product spaces, and still without mentioning determinants, this paper gives a simple proof of the finite-dimensional spectral theorem.
Link: The Napkin Project
Idea: high-quality explanations motivating undergrad++ math to bright high schoolers
Creator: Evan Chen (US IMO coach)
Reason: I’ll quote Evan himself:
The philosophy is stated in the preamble:
I’ll be eating a quick lunch with some friends of mine who are still in high school. They’ll ask me what I’ve been up to the last few weeks, and I’ll tell them that I’ve been learning category theory. They’ll ask me what category theory is about. I tell them it’s about abstracting things by looking at just the structure-preserving morphisms between them, rather than the objects themselves. I’ll try to give them the standard example Gp, but then I’ll realize that they don’t know what a homomorphism is. So then I’ll start trying to explain what a homomorphism is, but then I’ll remember that they haven’t learned what a group is. So then I’ll start trying to explain what a group is, but by the time I finish writing the group axioms on my napkin, they’ve already forgotten why I was talking about groups in the first place. And then it’s 1PM, people need to go places, and I can’t help but think:
Man, if I had forty hours instead of forty minutes, I bet I could actually have explained this all.
This book is my attempt at those forty hours.
This project has evolved to more than just forty hours.
Caveat that I didn’t actually do a math degree so I’d be curious to see takes from math folks saying disliking it, but I’ve enjoyed dipping in and out of its 1,048 pages over the years.
I think the way modern physics is taught probably gives people a overly clean/neat understanding of how most of the world works, and how to figure out problems in the world, but this might be ameliorated by studying the history of physics and how people come to certain conclusions.
Yeah and I think if done well it’s well-received here, e.g. AdamShimi’s My Number 1 Epistemology Book Recommendation: Inventing Temperature or Ben Pace’s 12 interesting things I learned studying the discovery of nature’s laws. (It’s hard to do well though it seems, I’m certainly dissatisfied with my own writeup attempts.)
Will you be writing elsewhere? I’ve benefited a lot from some of your comments, and would be bummed to see you leave.
Full Scott Aaronson quote in case anyone else is interested:
(couldn’t resist including that last sentence)