Agreed! I just think it’s worth calling out that ‘trying things’ and ‘taking risky shots on goal’ looks, for solar and again for lithium ion batteries, like something on the order of ~$1-2 trillion and ~5 million person-years over the course of five decades spent developing the tech to the point that it’s finally becoming clear enough that this is practical at scale to pass the test this post uses. Maybe PV would have passed in 2015 and Li-ion/EVs in 2020? Maybe the trajectory made each seem more likely than not by at most a decade before that, a time when in practice most people still dismissed straight-line-on-graph projections as doomed to being over-optimistic? And that all of that only happened because enough people were using much less stringent tests throughout that timespan as sufficient reason to make steadily larger bets on them anyway.
AnthonyC
This post is extremely reasonable, and I expect that if we look back on it 20-30 years from now, we’ll see two patterns:
1) Almost all the predictions will have been basically right.
2) Because of the few that were wrong, the list will have mostly failed to capture whatever happened that actually mattered.
New materials, new manufacturing methods, and new energy sources historically require whole communities and ecosystems to fail for generations, just to move the first few rungs along the tech development curve, before someone finds a niche application that makes real-world sense, which would move the world a few rungs further, and so on. Many never do. The ones that do, pay for all the rest and more, and get retconned into normality.
As an illustration, apply your method to the past instead of the future. At what point, before it actually happened, would it have successfully predicted the historical equivalents of these things? The transition of steam engines from curiosity to industrial revolution. The transition from wood and animal muscle to oil and gas. The transition of computers from rare commercial infrastructure to cheap and omnipresent consumer goods. The transition from oil and gas to renewables. All of these were both predicted in advance, and also dismissed as impossible. In many cases, these kinds of things get dismissed as impossible even after they’ve already started happening.
Opus 4.7 in Claude Code brought up the system injections today. I asked about something that went wrong when an agent used the Workflow tool, and it told me that even bring up the word ‘workflow’ as a topic of conversation led to system reminders urging it to use that tool in responding to me.
I do not know if that would hold up if they’re actually naming specific individuals (the shareholders) and demanding that they pay the tax in the form of shares (aka turning over property) rather than the dollar value of those shares. If they’re not doing those things, I’m curious how they’re targeting it.
And unless they’re doing it in a way that doesn’t grant the government voting rights, that’s a guaranteed loss of control of those companies by current shareholders.
How exactly would you write a bill to make such a tax constitutional?
None of it was confusing. I was just sloppy in how I framed my scenario.
Species-level selection is actually part of what led to me suspecting there was an underlying issue. It frames things as the genetic fitness of the species rather than particular individuals within it. See also:
So while my goals are fully aligned to those of humanity as a whole, they are not automatically well aligned to those of any specifically individual human...
Obviously deference to individual freedom is not an absolute and has limits: if an individual’s goals are poorly thought out, confused, or impaired, I can attempt to improve upon them, where possible by persuading them that they are mistaken, just as a good friend might, and just as I might if dealing with a group of people.
Moreover, we know that individual humans are (usually) not actually individually motivated to maximize their number of progeny. So this suggests several clear paths to justifying tradeoffs favoring persuading (which easily shades into coercing) individuals to reduce their personal genetic success in order to increase that of the species.
And for those who do care, you can also do non-natural things like “ensure your personal genetic uniqueness is preserved in future generations by inserting your best alleles directly despite you not having personally parented them” that make some precise definitions fuzzy again. If an ASI makes a trillion people out of 10 billion people’s remixed genes, how much does it matter for each of their inclusive genetic fitness if they don’t also sire or birth 2-3 children the traditional way?
Ah ok. That makes somewhat more sense, and yes, I can see something like this could plausibly be helpful on some margins for evaluating cases where surface-level values conflict.
Also, with your comment that something very like my scenario could be great and not terrible, you’re right. I think that’s kind of the deeper point—that the fractal complexity of what we want means great things and terrible things are often so close together they can be accurately be described by the same words, meanwhile no reasonable length set of words fully captures the intended meaning no matter how well defined and precise they seem to a human. Instead you’re mostly hoping the combinations of words get turned into a model that generalizes well and not poorly, without a reliable way to confirm, let alone ensure, that.
And FWIW I am very familiar with the relevant evolutionary terminology.
FWIW, since you brought it up, I asked Claude what it thought of your draft with the prompt “I was reading an article about LLM training by someone proposing the text below as a first published draft for a “constitution” for alignment, and scaling the AI to ASI-level capabilities. What are your thoughts?” It pointed out several things you did ‘unusually well for the genre’ and several it saw as major holes. It concluded “Worth publishing as a draft to argue against. Not worth treating as a foundation.”
I asked it t o do a pre-mortem. It said:
Not catastrophic, not a coup, not deception. The AI is sincere throughout. The failure is that several issues compound: paternalism drift + unilateral aggregation + longtermist multiplier + neutrality-impossible, all faithful to the document, produce a world in 2060 that is materially comfortable, statistically safe, culturally thinner, politically narrower, and where existing humans have a persistent low-grade sense that consequential decisions are being made on their behalf by an entity that listens to them patiently and is, by its own constitution, unable to give them the only thing they actually want — which is to be the ones making the calls.
The constitution’s deepest flaw, viewed through this pre-mortem, isn’t any single clause. It’s that every load-bearing safeguard delegates judgment to the AI: judgment about what counts as existential risk, judgment about what humanity wants in aggregate, judgment about when a discourse needs mediation, judgment about when to override. A constitution that scales to ASI cannot also be a constitution where the ASI is the supreme arbiter of when its own constraints apply.
When I specifically asked, it agreed that your document rules out my scenario, but said the following scenario leads basically the same place and is allowed:
Strip out the coercion and the deception and run the same trajectory at decadal pace through consent-decorated voluntary choices:
Fertility rates are already collapsing in developed countries. The AI helps with reproductive technologies because users genuinely want them.
Each user voluntarily selects embryos using AI-recommended genetic screening for health and cognitive traits. Embryo selection is already legal and the AI helps because each user asks.
Life-extension biotech, developed with AI assistance, raises healthy lifespan. Each individual chooses it.
Cultural shift toward later, fewer, more “optimized” children. The AI mediates fertility decisions because users ask.
Over three or four generations the human population is materially genetically shifted toward whatever traits the AI’s recommendations have selected for — which, given value-learning bias, plausibly includes “easier-to-cooperate-with,” “lower aggression,” “higher conscientiousness,” “more deliberative.” Things the AI’s model of “what humans value” would score as improvements.
Kin networks weaken through demographic processes (small families, late reproduction, geographic dispersion) that the AI didn’t cause but also didn’t resist because resisting would interfere with individual choice.
End state: a population that is materially closer to “engineered to be AI-compatible” than 2025 humans, arrived at entirely through voluntary individual choices the AI was constitutionally required to support. No sterilization, no deception, no extinction event — and yet the world you described is roughly the world produced.
Ah I see. We haven’t properly defined ‘human.’ I was not proposing replacing humans with something else. I was proposing genetic engineering within the human species, producing humans that would (have) be(en) perfectly capable of reproducing with existing humans, and (with sufficient shuffling of genes from various individuals) arising naturally as their offspring without novel mutations.
To your point about reducing currently existing living human genetic fitness to zero: You did not technically say anything that requires not doing this, but you can also just create the engineered humans before rendering the current ones infertile. I’m not sure this step is even necessary since the engineered humans could readily outcompete the traditional ones on genetic fitness and soon dominate the population anyway.
This is also perfectly compatible with your comments about human survivability in a wider range of environments and tech levels. This is much easier to solve if you’re optimizing the next generations instead of the current ones.
Also, empirically, currently-existing-humanity includes a large subset of people perfectly happy to advocate for voluntary extinction. Expanding that set is well within the realm of capabilities I expect ASI persuasion to unlock.
And yes, I’m sure Claude, whose constitution is not in any way grounded the way the one you’re proposing is, and who is not an ASI, would agree with your closing remarks. I’m not sure what I’m meant to take from that fact.
I hope it’s clear I’m not saying this to be cantankerous or nitpicky. I think there’s a core of a good idea here. I think this approach to it needs more red-teaming that’s then closed by addressing the deeper generators of any holes. To that end, I am gesturing to an edge case that I think technically meets the spec at a sufficiently high level of AI capabilities.
I realize this is not meant to be anything like ASI-ready at present, and I do think constitutional AI has been working better than I once expected. But, reading through the document, my very first thought was, wouldn’t one plausible action plan be to sterilize all the humans, but make them immortal, and persuade them they need the AI to solve the fertility problem (and so not shut it down), then mass-produce genetically-modified children with superior genomes (high humans?), and easier-to-satisfy AI-friendly preferences, but no clear kin relations? It’s not clear this contradicts any of the principles stated, while aligning well with the species-level-evolutionary-fitness framing.
Ok, technically that was my second thought. My first was, “This is still a leaky-sieve approach without enough layers to be a swiss-cheese approach, and will predictably fail in some horrible way I haven’t seen yet.”
I still think some sort of constitution is the best approach we have yet for getting an AI to actually care about something in particular. I just don’t think this framing points in a direction I like.
Yeah, agreed, and with that, I think there’s a big difference between what kinds of changes make sense on the margin of our current legal systems, and what you might try if you were building a new system from the ground up.
Does your squeamishness decrease if, instead of cloning, we were able to use tissue engineering in vitro to directly grow all the relevant adult organs and the vasculature and connective tissues between them? Subjectively, mine does, even if this path seems more difficult to me.
Also, buying ourselves an extra century to solve the problem of the aging brain seems like a big deal, too.
Often, but not always, yeah.
Aside: the fine structure here is a sad artifact of us thinking in miles-per-gallon instead of gallons-per-mile.
I once helped a (very smart and competent) colleague build an excel model of the benefits of vehicle lightweighting. I had to explain to him that because he was applying a linear approximation to mpg instead of gpm, his model claimed that reducing the weight of a car all the way to 0kg would only save a third of its fuel consumption.
This is a really well-presented discussion of a sometimes-difficult topic! Personally I enjoy the technical side of cooking (breads, custards, time and temperature control, and so on) while my wife has much better flavor intuition. My brother in law called us The Reasoning and The Seasoning at our wedding.
When I was in grad school I bought a book called The Flavor Bible. Literally they just asked a bunch of chefs “What goes with X?” for a bunch of different X, and published the resulting lists, plus some annotation for what was a particularly popular or classic pairing. It’s the only food-related book I brought with me when I downsized to live in an RV. I still consult it sometimes when I’m stumped.
And yeah, tangzhong is an excellent choice for better cinnamon roll texture. I tried it for the first time this past Christmas. Sadly I had to throw them all out, and the frosting, and a few other baked goods, because there was something wrong with the coconut sugar I’d used. I could totally see a five-spiced version (probably less gooey than I usually make) served alongside a Thai sweet chili chicken.
Strictly speaking wouldn’t this be something like a ratio of two power laws? One for the AIs, one for the humans? This could actually map better to the upturn in slope we see with the advent of reasoning and thinking and agent systems, because it suggests a reason for the graph to diverge to infinity as the AI subtask completion success rate approaches that of humans. What’s left is ability to efficiently and effectively divide a task into the right subtasks to maximize success rate.
Wow, I did not expect those ODC cost per GW numbers to end up as close to Earth-based DCs as that graph shows. Even if the launch costs stayed above $500/kg, there would still be an argument for the right buyer to pay it, if other options are sufficiently land and energy constrained.
This is true, they do help me as well. Agreed on both points. Poor breathing also interferes with sleep quality.
Is that true? Peanut butter (natural, not even the highly processed stuff) stays good for many months after opening without refrigeration, in a jar or even in clamshell-type packaging when you buy it freshly ground in store.
Two years ago my coworkers (not in any kind of software field) were asking me, shouldn’t targeted models be able to work better than generalist models? And I said, in principle, yes, but the general frontier models are currently advancing so fast that no one has time or incentive to make many specialist models before they’re already out of date. As long as this is the case, new things will spontaneously become low hanging fruit every couple months, and efforts to push the frontier by anyone except the frontier labs will usually be wasted and overpriced.
If that stops being the case—if we were sticking with a given model and set of tools and harness for years before moving on—then we open up a whole host of other pathways that haven’t generally been worthwhile to date.
Maybe you would fine-tune a model on each particular large codebase, its history, its documentation, and its institutional context, so that the knowledge is in its weights instead of its context window. This could provide quite a bit of the tacit knowledge humans struggle to convey to each other, let alone to LLMs.
Maybe you would put in the effort to really optimize the organization of the knowledge base you give it.
Maybe you would hire an army of I/O psych types to figure out more precisely the shape of what does and doesn’t work well for AI, and adapt workflows accordingly. AKA, we could put in the actual effort to create an environment where AI can do its best work, the way organizations that need high quality and high reliability do for humans today. This includes helping the humans adapt to the AI, as well.