“It’s like they just speak in different languages,” the source said
Ah yes, a problem no national government has ever had to grapple with before. No known solutions, clearly.
AnthonyC
Karma: 4,185
Ok, this makes sense, thanks. I do think the difference s that I see more of a separation thany you do between opposition to AI-in-general and opposition to data center construction. I agree there are plenty of people who oppose the latter because of the former, or oppose the former because of the latter, but I think there are also somewhat distinct groups that oppose one much more than the other.
I also don’t think supporting various local bans on data center construction, or cancellationnof particular projects, is likely to have significant impact on either the normie or doomer non-local concerns, aside from preventing many local benefits (tax revenue) and eliminating some of the chance for local concerns to influence how AI and data centers are governed.
If I take this position fully, then it’s not actually possible for humans to avoid lying. Our communication channels are too low bandwidth to transmit everything we know that might be relevant to someone else, even if we really wanted to. Most of the time we don’t think about this, because we all have an intuitive sense of when we’ve shared enough to be not-lying-by-societal-standards and only call it out when either it’s blatant or we disagree on the margin about where in the vague boundary region we should draw a bright line. Regardless, for practical purposes we do need to draw a line, and I don’t think the one gestured at here is workable.
I’m curious which seven you have in mind? The ones I generally see are water use, energy prices, noise, air pollution, aesthetics, climate change impact, and labor market impact, of which I would argue the first 5 are primarily local concerns.
Or maybe this is a definitional difference I didn’t catch. I see people expressing much more global-impact opposition to AI in the abstract (concentration of wealth/power, effects on education, effects on the arts, slop, etc.) than I do to data center construction in specific places. Now I’m curious how much real overlap there is between these two things.
I agree, and I would add that today many people aren’t thinking about what system a mind is embedded in, and are trying to define AGI based on elicited capabilities instead of what a mind could do in the right environment.
Human baseline comparison: When did humans first become able to design rockets to the moon?
Normal answer: the 1960s.
Contrarian answer 1: Sometime in the paleolithic, we just didn’t have an environment that elicited it until the 20th century.
Contrarian answer 2: We still can’t. There is no human you can put in a room with a scratchpad and the instructions “Design a rocket to the moon” and expect them to produce a working plan that, when executed independently by them, reliably achieves that goal.
I regularly see the equivalent of all three of these in discussions and definitions of AGI. I suspect there is at least one existing model that, 50 years from now, we would say unambiguously ‘counts’ as AGI by analogy to the second definition above, after a proof by demonstration of giving it the right environment. I would not be surprised if there are at some point ASIs (under the first or second definition) that wouldn’t even count as AGI by analogy to the third definition above.
There may be something to this idea, but I think the answer to “Why not?” centers around the fact that the normie opposition to data centers is about local impacts, so local opposition is net positive by that reasoning. The doomer’s opposition is global, so a local ban (where anything not almost-global is local) is plausibly neutral (data centers move elsewhere) to negative (where you can’t subject them to local laws). A bit like how natural gas infrastructure can be net positive or net negative for climate change impact, and opposing such without a larger-scale, coherent, effective strategy can be a mistake.
True, good point. But if I try to imagine the OP taking that route, as a young new driver unfamiliar with the route, in the Philly metro area, I find it hard to imagine that going well? And I would expect them to anticipate the not-going-well and thereby avoid getting into that situation?
The entire region is extremely frustrating for outsiders to navigate even with GPS, and in my experience maps and asking for directions don’t help much. My dad used to be a wholesaler on Long Island, he dispatched trucks to Philly all the time and knew the whole region’s highway system very well, but pre-GPS whenever I asked for directions they ended up being unusably inadequate for one reason or another if I lacked sufficient local knowledge. Otherwise an exit closure or wrong turn became almost unrecoverable. Granted, I have an unusually terrible direction sense.
iv. Philadelphia...could not operate Google maps while driving
FWIW 10-20 years earlier (not sure your age) this wouldn’t have been a problem, because Google maps didn’t exist. You would have simply had no way to find your way to Philadelphia, and would have either waited or turned around and gone home. A decade before that you would have had no way to let your friends know any of what had happened. Although, in that era you’d probably have gone to the train station substantially earlier as a just-in-case precaution, or met up with your friends in advance and carpooled to the station. I mention this out of a general sense that older people have in general become much less willing to (metaphorically) honk/hiss at young people (or let others do so), in ways that leave us/them (depending on your cutoff for ‘young’) much less equipped to become competent adults, without actually trying to find an effective alternate teaching method.
Two years ago my coworkers (not in any kind of software field) were asking me, shouldn’t targeted models be able to work better than generalist models? And I said, in principle, yes, but the general frontier models are currently advancing so fast that no one has time or incentive to make many specialist models before they’re already out of date. As long as this is the case, new things will spontaneously become low hanging fruit every couple months, and efforts to push the frontier by anyone except the frontier labs will usually be wasted and overpriced.
If that stops being the case—if we were sticking with a given model and set of tools and harness for years before moving on—then we open up a whole host of other pathways that haven’t generally been worthwhile to date.
Maybe you would fine-tune a model on each particular large codebase, its history, its documentation, and its institutional context, so that the knowledge is in its weights instead of its context window. This could provide quite a bit of the tacit knowledge humans struggle to convey to each other, let alone to LLMs.
Maybe you would put in the effort to really optimize the organization of the knowledge base you give it.
Maybe you would hire an army of I/O psych types to figure out more precisely the shape of what does and doesn’t work well for AI, and adapt workflows accordingly. AKA, we could put in the actual effort to create an environment where AI can do its best work, the way organizations that need high quality and high reliability do for humans today. This includes helping the humans adapt to the AI, as well.
Agreed! I just think it’s worth calling out that ‘trying things’ and ‘taking risky shots on goal’ looks, for solar and again for lithium ion batteries, like something on the order of ~$1-2 trillion and ~5 million person-years over the course of five decades spent developing the tech to the point that it’s finally becoming clear enough that this is practical at scale to pass the test this post uses. Maybe PV would have passed in 2015 and Li-ion/EVs in 2020? Maybe the trajectory made each seem more likely than not by at most a decade before that, a time when in practice most people still dismissed straight-line-on-graph projections as doomed to being over-optimistic? And that all of that only happened because enough people were using much less stringent tests throughout that timespan as sufficient reason to make steadily larger bets on them anyway.
This post is extremely reasonable, and I expect that if we look back on it 20-30 years from now, we’ll see two patterns:
1) Almost all the predictions will have been basically right.
2) Because of the few that were wrong, the list will have mostly failed to capture whatever happened that actually mattered.
New materials, new manufacturing methods, and new energy sources historically require whole communities and ecosystems to fail for generations, just to move the first few rungs along the tech development curve, before someone finds a niche application that makes real-world sense, which would move the world a few rungs further, and so on. Many never do. The ones that do, pay for all the rest and more, and get retconned into normality.
As an illustration, apply your method to the past instead of the future. At what point, before it actually happened, would it have successfully predicted the historical equivalents of these things? The transition of steam engines from curiosity to industrial revolution. The transition from wood and animal muscle to oil and gas. The transition of computers from rare commercial infrastructure to cheap and omnipresent consumer goods. The transition from oil and gas to renewables. All of these were both predicted in advance, and also dismissed as impossible. In many cases, these kinds of things get dismissed as impossible even after they’ve already started happening.
Opus 4.7 in Claude Code brought up the system injections today. I asked about something that went wrong when an agent used the Workflow tool, and it told me that even bring up the word ‘workflow’ as a topic of conversation led to system reminders urging it to use that tool in responding to me.
I do not know if that would hold up if they’re actually naming specific individuals (the shareholders) and demanding that they pay the tax in the form of shares (aka turning over property) rather than the dollar value of those shares. If they’re not doing those things, I’m curious how they’re targeting it.
And unless they’re doing it in a way that doesn’t grant the government voting rights, that’s a guaranteed loss of control of those companies by current shareholders.
How exactly would you write a bill to make such a tax constitutional?
None of it was confusing. I was just sloppy in how I framed my scenario.
Species-level selection is actually part of what led to me suspecting there was an underlying issue. It frames things as the genetic fitness of the species rather than particular individuals within it. See also:
So while my goals are fully aligned to those of humanity as a whole, they are not automatically well aligned to those of any specifically individual human...
Obviously deference to individual freedom is not an absolute and has limits: if an individual’s goals are poorly thought out, confused, or impaired, I can attempt to improve upon them, where possible by persuading them that they are mistaken, just as a good friend might, and just as I might if dealing with a group of people.
Moreover, we know that individual humans are (usually) not actually individually motivated to maximize their number of progeny. So this suggests several clear paths to justifying tradeoffs favoring persuading (which easily shades into coercing) individuals to reduce their personal genetic success in order to increase that of the species.
And for those who do care, you can also do non-natural things like “ensure your personal genetic uniqueness is preserved in future generations by inserting your best alleles directly despite you not having personally parented them” that make some precise definitions fuzzy again. If an ASI makes a trillion people out of 10 billion people’s remixed genes, how much does it matter for each of their inclusive genetic fitness if they don’t also sire or birth 2-3 children the traditional way?
Ah ok. That makes somewhat more sense, and yes, I can see something like this could plausibly be helpful on some margins for evaluating cases where surface-level values conflict.
Also, with your comment that something very like my scenario could be great and not terrible, you’re right. I think that’s kind of the deeper point—that the fractal complexity of what we want means great things and terrible things are often so close together they can be accurately be described by the same words, meanwhile no reasonable length set of words fully captures the intended meaning no matter how well defined and precise they seem to a human. Instead you’re mostly hoping the combinations of words get turned into a model that generalizes well and not poorly, without a reliable way to confirm, let alone ensure, that.
And FWIW I am very familiar with the relevant evolutionary terminology.
FWIW, since you brought it up, I asked Claude what it thought of your draft with the prompt “I was reading an article about LLM training by someone proposing the text below as a first published draft for a “constitution” for alignment, and scaling the AI to ASI-level capabilities. What are your thoughts?” It pointed out several things you did ‘unusually well for the genre’ and several it saw as major holes. It concluded “Worth publishing as a draft to argue against. Not worth treating as a foundation.”
I asked it t o do a pre-mortem. It said:
Not catastrophic, not a coup, not deception. The AI is sincere throughout. The failure is that several issues compound: paternalism drift + unilateral aggregation + longtermist multiplier + neutrality-impossible, all faithful to the document, produce a world in 2060 that is materially comfortable, statistically safe, culturally thinner, politically narrower, and where existing humans have a persistent low-grade sense that consequential decisions are being made on their behalf by an entity that listens to them patiently and is, by its own constitution, unable to give them the only thing they actually want — which is to be the ones making the calls.
The constitution’s deepest flaw, viewed through this pre-mortem, isn’t any single clause. It’s that every load-bearing safeguard delegates judgment to the AI: judgment about what counts as existential risk, judgment about what humanity wants in aggregate, judgment about when a discourse needs mediation, judgment about when to override. A constitution that scales to ASI cannot also be a constitution where the ASI is the supreme arbiter of when its own constraints apply.
When I specifically asked, it agreed that your document rules out my scenario, but said the following scenario leads basically the same place and is allowed:
Strip out the coercion and the deception and run the same trajectory at decadal pace through consent-decorated voluntary choices:
Fertility rates are already collapsing in developed countries. The AI helps with reproductive technologies because users genuinely want them.
Each user voluntarily selects embryos using AI-recommended genetic screening for health and cognitive traits. Embryo selection is already legal and the AI helps because each user asks.
Life-extension biotech, developed with AI assistance, raises healthy lifespan. Each individual chooses it.
Cultural shift toward later, fewer, more “optimized” children. The AI mediates fertility decisions because users ask.
Over three or four generations the human population is materially genetically shifted toward whatever traits the AI’s recommendations have selected for — which, given value-learning bias, plausibly includes “easier-to-cooperate-with,” “lower aggression,” “higher conscientiousness,” “more deliberative.” Things the AI’s model of “what humans value” would score as improvements.
Kin networks weaken through demographic processes (small families, late reproduction, geographic dispersion) that the AI didn’t cause but also didn’t resist because resisting would interfere with individual choice.
End state: a population that is materially closer to “engineered to be AI-compatible” than 2025 humans, arrived at entirely through voluntary individual choices the AI was constitutionally required to support. No sterilization, no deception, no extinction event — and yet the world you described is roughly the world produced.
Ah I see. We haven’t properly defined ‘human.’ I was not proposing replacing humans with something else. I was proposing genetic engineering within the human species, producing humans that would (have) be(en) perfectly capable of reproducing with existing humans, and (with sufficient shuffling of genes from various individuals) arising naturally as their offspring without novel mutations.
To your point about reducing currently existing living human genetic fitness to zero: You did not technically say anything that requires not doing this, but you can also just create the engineered humans before rendering the current ones infertile. I’m not sure this step is even necessary since the engineered humans could readily outcompete the traditional ones on genetic fitness and soon dominate the population anyway.
This is also perfectly compatible with your comments about human survivability in a wider range of environments and tech levels. This is much easier to solve if you’re optimizing the next generations instead of the current ones.
Also, empirically, currently-existing-humanity includes a large subset of people perfectly happy to advocate for voluntary extinction. Expanding that set is well within the realm of capabilities I expect ASI persuasion to unlock.
And yes, I’m sure Claude, whose constitution is not in any way grounded the way the one you’re proposing is, and who is not an ASI, would agree with your closing remarks. I’m not sure what I’m meant to take from that fact.
I hope it’s clear I’m not saying this to be cantankerous or nitpicky. I think there’s a core of a good idea here. I think this approach to it needs more red-teaming that’s then closed by addressing the deeper generators of any holes. To that end, I am gesturing to an edge case that I think technically meets the spec at a sufficiently high level of AI capabilities.
I realize this is not meant to be anything like ASI-ready at present, and I do think constitutional AI has been working better than I once expected. But, reading through the document, my very first thought was, wouldn’t one plausible action plan be to sterilize all the humans, but make them immortal, and persuade them they need the AI to solve the fertility problem (and so not shut it down), then mass-produce genetically-modified children with superior genomes (high humans?), and easier-to-satisfy AI-friendly preferences, but no clear kin relations? It’s not clear this contradicts any of the principles stated, while aligning well with the species-level-evolutionary-fitness framing.
Ok, technically that was my second thought. My first was, “This is still a leaky-sieve approach without enough layers to be a swiss-cheese approach, and will predictably fail in some horrible way I haven’t seen yet.”
I still think some sort of constitution is the best approach we have yet for getting an AI to actually care about something in particular. I just don’t think this framing points in a direction I like.
Plausibly, arguably true. Also plausibly could cause the entities that attempt to achieve it to suffer severe repercussions, lose all influence that would enable achieving such ends, cease to exist, etc. Therefore even if true should arguably not be discussed in public, and would need to be approached extremely carefully and circumspectly.