My blog is here. You can subscribe for new posts there.
My personal site is here.
My X/Twitter is here
You can contact me using this form.
L Rudolf L
Karma: 2,791
Noting that your 2x2 is not exactly how I see it. For example, power grabs become more likely due to the intelligence curse and humans being more able to attempt them is a consequence of the gradual incentives, and (as you mention above) parts of gradual disempowerment are explicitly about states becoming less responsive to humans due to incentives (which, in addition to empowering some AIs, will likely empower humans who control states).
My own diagram to try to make sense of these is here (though note that this is rough OTOH work, not intended as , and not checekd with authors of the other pieces):
Thanks, this was excellent!
To summarize, for anyone else reading this, the model proposed is essentially:
The most fundamental and primitive system for choosing a ruler/government is war, in which the strong win
Wars are very expensive, so all sides have an incentive to avoid them
However, if you setup some layer above the raw physical conflict to decide the winner, the loser has to calculate that resorting to war wouldn’t go well for them either. Therefore, the higher layer has to be legibly predictive of the lower layer.
Therefore, democracy arises when the number of people on a side is a good predictor of the outcome of a conflict, which is true under industrialized warfare that relies on massive production & large armies.
Of course, there’s a lot of lag in these things, so e.g. even though nuclear weapons change the above, they haven’t been used and systems have inertia. (Also, implicitly, Japan—the review’s main topic—had enough inertia from the samurai ruling class to current bureaucrats that real power still resides with them).
One story for why technocratic, elite-driven governance is becoming more common (but is it? see below) is that success in conflict would now rely more on the highly-educated, information control, and so forth
One story for the future trend is that offense becomes easier with drones etc., so assassination will become easier and easier, so victory in power struggles will come down to secrecy & ruling from the shadows, so MITI-style technocratic cabal governance might become even more common.
OTOH, some comments/questions that come to mind:
This is all effectively a story of internal selection within a country. However, times and places differ in the extent to which the main pressure on the political system is an internal or external threat. Now, winning wars against external adversaries is pretty similar to winning wars against internal ones, so with state-to-state competition in the industrial age. But was it internal or external competition that actually provided the bits of selection? (My guess is that in stable countries, it’s more the external selection that promotes democracies—e.g. the UK is just remarkably stable and an armed revolt of a losing political faction doesn’t seem like it’s been in the cards in the last few centuries) In general, this focus on internal rather than external competition seems like a very American focus, thanks to America’s geographic luck & (waning) hegemonic position.
What’s the relative importance of army size & mass production in military tech over time? E.g. are the Napoleonic Wars more like WW1 or more like feudal times, in terms of what it takes to win? If the latter, it seems like US & UK democratization can’t really be downstream.
Is part of this forward luck, rather than causation? E.g. you could tell a story where the UK & US democratized for circumstantial internal reasons, but then this made them more stable & effective internally, and helped them win externally. I.e. it’s not that countries drift towards the optimum, but that countries are at many different fitness points and do not even reliably drift towards higher fitness, but then selection prunes the ranks and rewards the fittest. In general, I think many explanations like this emphasize the causal chain more than the luck + selection.
Isn’t the growth of bureaucracy as a natural part of the lifecycle of institutions a more natural story for decreasing representativeness?
And, come to mention, are Western governments becoming less representative of the popular will over time at all? It seems like the best argument for this being true is that in the past few years, educational polarization means that elite & non-elite views on some issues (esp. immigration) are further apart than usual. But this is a very recent trend, and the elites are losing this one! In many ways, the elite held more leverage before the internet decentralized media. To the extent that Western governments are less able to deliver the goods that the population demands, this seems less because they’re taken over by elites with different interests to the voters, and more because of (1) broad demographic & economic trends, (2) voters having preferences that make these hard to fix (e.g. can’t raise retirement ages to fix #1; can’t increase immigration to mitigate the need to reduce the retirement age less; can’t build much nuclear power) and these voters very successfully forcing the political system to avoid taking elite-recommended actions (cf. French retirement age debate), and (3) general institutional & cultural decay and dysfunction. Now of course, part of #3 is bureaucratic capture etc., but a lot of the value capture is also by non-technocratic-elite groups (e.g. NIMBYs, pensioners, landlords).
There’s a version now that was audited by Chrome Web Store, if that’s enough for you: https://chromewebstore.google.com/detail/objective/dckljlpogfgmgmnbaicaiohioinipbge?authuser=0&hl=en-GB
Currently you need to be on the beta list though, since it costs Gemini API credits to run (though a quite trivial amount)—if you (or anyone else) DMs me an email I can add you to the list, and at some point if I have time I might enable payments such that it’s generally available.
Thanks a lot for this post! I appreciate you taking the time to engage, I think your recommendations are good, and I agree with most of what you say. Some comments below.
Intelligence curse != Gradual disempowerment
“the intelligence curse” or “gradual disempowerment”—concerns that most humans would end up disempowered (or even dying) because their labor is no longer valuable.
The intelligence curse and GD are not equivalent. In particular, I expect @Jan_Kulveit & co. would see GD as a broader bucket including also various subtle forms of cultural misalignment (which tbc I think also matter!), whereas IC is more specifically about things downstream of economic (and hard power, and political power) incentives. (And I would see e.g. @Tom Davidson’s AI-enabled coup risk work as a subset of IC, as representing the most sudden and dramatic way that IC incentives could play out)
On the stakes
It’s worth noting I doubt that these threats would result in huge casualty counts (due to e.g. starvation) or disempowerment of all humans (though substantial concentration of power among a smaller group of humans seems quite plausible).
[fn:]
That said, I do think that technical misalignment issues are pretty likely to disempower all humans and I think war, terrorism, or accidental release of homicidal bioweapons could kill many. That’s why I focus on misalignment risks.
I think if you follow the arguments, disempowerment of all humans is plausible, and disempowerment of the vast majority even more so. I agree that technical misalignment is more likely to lead to high casualty counts if it happens (and I think the technical misalignment --> x-risk pathway is possible and incredibly urgent to make progress on).
I think there’s also a difference between working on mitigating very clear sequences of steps that lead to catastrophe (e.g. X --> Y --> everyone drops dead), and working on maintaining the basic premises that make things not broken (e.g. for the last 200 years when things have been getting much better, the incentives of power and humans have been remarkably correlated, and maybe we should try to not decorrelate them). The first is more obvious, but I think you should also be able to admit theories of change of the second type at least sufficiently that, for example, you would’ve decided to resist communism in the 1950s (“freedom good” is vague, and there wasn’t yet consensus that market-based economies would provide better living standards in the long run, but it was still correct to bet against the communists if you cared about human welfare! basic liberalism is very powerful!).
The proposals
Mandatory interoperability for alignment and fine-tuning is a great idea that I’m not sure I’ve heard before!
Alignment to the user is also a great idea (which @lukedrago & I also wrote about in The Intelligence Curse). There are various reasons I think the per-user alignment needs to be quite fine-grained (e.g. I expect it requires finetuning / RLHF, not just basic prompting), which I think you also buy (?), and which I hope to write about more later.
Implicit in my views is that the problem would be mostly resolved if people had aligned AI representatives which helped them wield their (current) power effectively.
Yep, this is a big part of the future I’m excited to build towards.
Prioritization of proposals
I’m skeptical of generally diffusing AI into the economy, working on systems for assisting humans, and generally uplifting human capabilities. This might help some with societal awareness, but doesn’t seem like a particularly leveraged intervention for this. Things like emulated minds and highly advanced BCIs might help with misalignment, but otherwise seems worse than AI representatives (which aren’t backdoored and don’t have secret loyalties/biases).
I think there are two basic factors that affect uplift chances:
Takeoff speed—if this is fast, then uplift matters less. However, note that there are two distinct ways in which time helps:
more societal awareness over time and more time to figure out what policies to advocate and what steps to take and so on—and the value of this degrades very quickly with takeoff speed increasing
people have more power going into the extreme part of takeoff—but note that how rapidly you can ramp up power also increases with takeoff speed (e.g. if you can achieve huge things in the last year of human labor because of AI uplift, you’re in a better position when going into the singularity, and the AI uplift amount is related to takeoff speed)
How contingent is progress along the tech tree. I believe:
The current race towards agentic AGI in particular is much more like 50% cultural/path-dependent than 5% cultural/path-dependent and 95% obvious. I think the decisions of the major labs are significantly influenced by particular beliefs about AGI & timelines; while these are likely (at least directionally) true beliefs, it’s not at all clear to me that the industry would’ve been this “situationally aware” in alternative timelines.
Tech progress, especially on fuzzier / less-technical things about human-machine interaction and social processes, is really quite contingent. I think we’d have much meaningfully worse computer interfaces today if Steve Jobs had never lived.
(More fundamentally, there’s also the question of how high you think human/AI complementarity at cognitive skills to be—right now it’s surprisingly high IMO)
I’m skeptical that local data is important.
I’m curious what your take on the basic Hayek point is?
I agree that AI enabled contracts, AI enabled coordination, and AIs speeding up key government processes would be good (to preserve some version of rule of law such that hard power is less important). It seems tricky to advance this now.
I expect a track record of trying out some form of coordination at scale is really helpful for later getting it into government / into use by more “serious” actors. I think it’s plausible that it’s really hard to get governments to try any new coordination or governance mechanism before it’s too late, but if you wanted to increase the odds, I think you should just very clearly be trying them out in practice.
Understanding agency, civilizational social processes, and how you could do “civilizational alignment” seems relatively hard and single-single aligned AI advisors/representatives could study these areas as needed (coordinating research funding across many people as needed).
I agree these are hard, and also like an area where it’s unclear if cracking R&D automation to the point where we can hill-climb on ML performance metrics gets you AI that does non-fake work on these questions. I really want very good AI representatives that are very carefully aligned to individual people if we’re going to have the AIs work on this.
We mention the threat of coups—and Davidson et. al.’s paper on it—several times.
Regarding the weakness or slow-actingness of economic effects: it is true that the fundamental thing that forces the economic incentives to percolate to the surface and actually have an effect is selection pressure, and selection pressure is often slow-acting. However: remember that the time that matters is not necessarily calendar time.
Most basically, the faster the rate of progress and change, the faster selection pressures operate.
As MacInnes et. al. point out in Anarchy as Architect, the effects of selection pressures often don’t manifest for a long time, but then appear suddenly in times of crisis—for example, the World Wars leading to a bunch of industrialization-derived state structure changes happening very quickly. The more you believe that takeoff will be chaotic and involve crises and tests of institutional capacity, the more you should believe that unconscious selection pressures will operate quickly.
You don’t need to wait for unconscious selection to work, if the agents in charge of powerful actors can themselves plan and see the writing on the wall. And the more planning capacity you add into the world (a default consequence of AI!), the more effectively you should expect competing agents (that do not coordinate) to converge on the efficient outcome.
Of course, it’s true that if takeoff is fast enough then you might get a singleton and different strategies apply—though of course singletons (whether human organizations or AIs) immediately create vast risk if they’re misaligned. And if you have enough coordination, then you can in fact avoid selection pressures (but a world with such effective coordination seems to be quite an alien world from ours or any that historically existed, and unlikely to be achieved in the short time remaining until powerful AI arrives, unless some incredibly powerful AI-enabled coordination tech arrives quickly). But this requires not just coordination, but coordination between well-intentioned actors who are not corrupted by power. If you enable perfect coordination between, say, the US and Chinese government, you might just get a dual oligarchy controlling the world and ruling over everyone else, rather than a good lightcone.
If humanity loses control and it’s not due to misaligned AI, it’s much more likely to be due to an AI enabled coup, AI propaganda or AI enabled lobbying than humans having insufficient economic power.
AI-enabled coups and AI-enabled lobbying all get majorly easier and more effective the more humanity’s economic role have been erased. Fixing them is also all part of maintaining the balance of power in society.
I agree that AI propaganda, and more generally AI threats to the information environment & culture, are a big & different deal that intelligence-curse.ai don’t address except in passing. You can see the culture section of Gradual Disempowerment (by @Jan_Kulveit @Raymond D & co.) for more on this.
There’s a saying “when all you have is a hammer, everything looks like a nail” that I think applies here. I’m bearish on [approaches] opposed to multi-disciplinary approaches that don’t artificially inflate particular factors.
I share the exact same sentiment, but for me it applies in reverse. Much “basic” alignment discourse seems to admit exactly two fields—technical machine learning and consequentialist moral philosophy—while sweeping aside considerations about economics, game theory, politics, social changes, institutional design, culture, and generally the lessons of history. A big part of what intelligence-curse.ai tries to do is take this more holistic approach, though of course it can’t focus on everything, and in particular neglects the culture / info environment / memetics side. Things that try to be even more holistic are my scenario and Gradual Disempowerment.
I don’t believe the standard story of the resource curse.
What do you think is the correct story for the resource curse?
I find the scenario implausible.
This is not a scenario, it is a class of concerns about the balance of power and economic misalignment that we expect to be a force in many specific scenarios. My actual scenario is here.
The “social-freeze and mass-unemployment” narrative seems to assume that AI progress will halt exactly at the point where AI can do every job but is still somehow not dangerous.
We do not assume AI progress halts at that point. We say several times that we expect AIs to keep improving. They will take the jobs, and they will keep on improving beyond that. The jobs do not come back if the AI gets even smarter. We also have an entire section dedicated to mitigating the risks of AIs that are dangerous, because we believe that is a real and important threat.
More directly, full automation of the economy would mean that AI can perform every task in companies already capable of creating military, chemical, or biological threats. If the entire economy is automated, AI must already be dangerously capable.
Exactly!
I expect reality to be much more dynamic, with many parties simultaneously pushing for ever-smarter AI while understanding very little about its internals.
“Reality will be dynamic, with many parties simultaneously pushing for ever-smarter AI [and their own power & benefit] while understanding very little about [AI] internals [or long-term societal consequences]” is something I think we both agree with.
I expect that approaching superintelligence without any deeper understanding of the internal cognition this way will give us systems that we cannot control and that will get rid of us. For these reasons, I have trouble worrying about job replacement.
If we hit misaligned superintelligence in 2027 and all die as a result, then job replacement, long-run trends of gradual disempowerment, and the increased chances of human coup risks indeed do not come to pass. However, if we don’t hit misaligned superintelligence immediately, and instead some humans pull a coup with the AIs, or the advanced AIs obsolete humans very quickly (very plausible if you think AI progress will be fast!) and the world is now states battling against each other with increasingly dangerous AIs while feeling little need to care for collateral damage to humans, then it sure will have been a low dignity move from humanity if literally no one worked on those threat models!
You also seem to avoid mentioning the extinction risk in this text.
The audience is primarily not LessWrong, and the arguments for working on alignment & hardening go through based on merely catastrophic risks (which we do mention many times). Also, the series is already enough of an everything-bagel as it is.
The Intelligence Curse: an essay series
VDT: a solution to decision theory
The scenario does not say that AI progress slows down. What I imagined to be happening is that after 2028 or so, there is AI research being done by AIs at unprecedented speeds, and this drives raw intelligence forward more and more, but (1) the AIs still need to run expensive experiments to make progress sometimes, and (2) basically nothing is bottlenecked by raw intelligence anymore so you don’t really notice it getting even better.
I will admit I’m not an expert here. The intuition behind this is that if you grant extreme performance at mathsy things very soon, it doesn’t seem unreasonable that the AIs will make some radical breakthrough in the hard sciences surprisingly soon, while still being bad at many other things. In the scenario, note that it’s a “mathematical framework” (implicitly a sufficiently big advance in what we currently have such that it wins a Nobel) but not the final theory of everything, and it’s explicitly mentioned empirical data bottlenecks it.
Thanks for these speculations on the longer-term future!
while I do think Mars will be exploited eventually, I expect the moon to be first for serious robotics effort
Maybe! My vague Claude-given sense is that the Moon is surprisingly poor in important elements though.
not being the fastest amongst them all (because replicating a little better will usually only get a little advantage, not an utterly dominant one), combined with a lot of values being compatible with replicating fast, so value alignment/intent alignment matters more than you think
This is a good point! However, more intelligence in the world also means we should expect competition to be tighter, reducing the amount of slack by which you can deviate from the optimal. In general, I can see plausible abstract arguments for the long-run equilibrium being either Hansonian zero-slack Malthusian competition or absolute unalterable lock-in.
Given no nationalization of the companies has happened, and they still have large freedoms of action, it’s likely that Google Deepmind, OpenAI and Anthropic have essentially supplanted the US as the legitimate government, given their monopolies on violence via robots.
I expect the US government to be competent enough to avoid being supplanted by the companies. I think politicians, for all their flaws, are pretty good at recognising a serious threat to their power. There’s also only one government but several competing labs.
(Note that the scenario doesn’t mention companies in the mid and late 2030s)
the fact that EA types got hired to some of the most critical positions on AI was probably fairly critical in this timeline for preventing the worst outcomes from the intelligence curse from occurring.
In this timeline, a far more important thing is the sense among American political elite that they are freedom-loving people and that they should act in accordance with that, and a similar sense among Chinese political elite that they are a civilised people and that Chinese civilisational continuity is important. A few EAs in government, while good, will find it difficult to match the impact of the cultural norms that a country’s leaders inherit and that proscribe their actions.
For example: I’ve been reading Christopher Brown’s Moral Capital recently, which looks at how opposition to slavery rose to political prominence in 1700s Britain. It claims that early strong anti-slavery attitudes were more driven by a sense that slavery was insulting to Britons’ sense of themselves as a uniquely liberal people, than by arguments about slave welfare. At least in that example, the major constraint on the treatment of a powerless group of people seems to have been in large part the political elite managing its own self-image.
A History of the Future, 2025-2040
I built this a few months ago: https://github.com/LRudL/devcon
Definitely not production-ready and might require some “minimal configuration and tweaking” to get working.
Includes a “device constitution” that you set; if you visit a website, Claude will judge whether the page follows that written document, and if not it will block you, and the only way past it is winning a debate with it about why your website visit is in-line with your device constitution.
I found it too annoying but some of my friends liked it.
However, I think there is a group of people who over-optimize for Direction and neglect the Magnitude. Increasing Magnitude often comes with the risk of corrupting the Direction. For example, scaling fast often makes it difficult to hire only mission-aligned people, and it requires you to give voting power to investors that prioritizes profit. To increase Magnitude can therefore feel risky, what if I end up working at something that is net-negative for the world? Therefore it might be easier for one’s personal sanity to optimize for Direction, to do something that is unquestionably net-positive. But this is the easy way out, and if you want to have the highest expected value of your Impact, you cannot disregard Magnitude.
You talk here about an impact/direction v ambition/profit tradeoff. I’ve heard many other people talking about this tradeoff too. I think it’s overrated; in particular, if you’re constantly having to think about it, that’s a bad sign.
It’s rare that you have a continuous space of options between lots of impact and low profit, and low/negative impact and high profit.
If you do have such a continuous space of options then I think you are often just screwed and profit incentives will win.
The really important decision you make is probably a discrete choice: do you start an org trying to do X, or an org trying to do Y? Usually you can’t (and even if you can, shouldn’t) try to interpolate between these things, and making this high-level strategy call will probably shape your impact more than any later finetuning of parameters within that strategy.
Often, the profit incentives point towards the more-obvious, gradient-descent-like path, which is usually very crowded and leads to many “mediocre” outcomes (e.g. starting a $10M company), but the biggest things come from doing “Something Else Which Is Not That” (as is said in dath ilan). For example, SpaceX (ridiculously hard and untested business proposition) and Facebook (started out seeing very small and niche and with no clue of where the profit was).
Instead, I think the real value of doing things that are startup-like comes from:
The zero-to-one part of Peter Thiel’s zero-to-one v one-to-n framework: the hardest, progress-bottlenecking things usually look like creating new things, rather than scaling existing things. For example, there is very little you can do today in American politics that is as impactful or reaches as deep into the future as founding America in the first place.
In the case of AI safety: neglectedness. Everyone wants to work at a lab instead, humans are too risk averse in general, etc. (I’ve heard many people in AI safety say that neglectedness is overrated. There are arguments like this one that replaceability/neglectedness considerations aren’t that major: job performance is heavy-tailed, hiring is hard for orgs, etc. But such arguments seem like weirdly myopic parameter-fiddling, at least when the alternative is zero-to-one things like discussed above. Starting big things is in fact big. Paradigm shifts matter because they’re the frame that everything else takes place in. You either see this or you don’t.)
To the extent you think the problem is about economic incentives or differential progress, have you considered getting your hands dirty and trying to change the actual economy or the direction of the tech tree? There are many ways to do this, including some types of policy and research. But I think the AI safety scene has a cultural bias towards things that look like research or information-gathering, and away from being “builders” in the Silicon Valley sense. One of the things that Silicon Valey does get right is that being a builder is very powerful. If the AI debate comes down to a culture/influence struggle between anti-steering, e/acc-influenced builder types and pro-steering EA-influenced academic types, it doesn’t look good for the world.
Thanks for the heads-up, that looks very convenient. I’ve updated the post to link to this instead of the scraper repo on GitHub.
As far as I know, my post started the recent trend you complain about.
Several commenters on this thread (e.g. @Lucius Bushnaq here and @MondSemmel here) mention LessWrong’s growth and the resulting influx of uninformed new users as the likely cause. Any such new users may benefit from reading my recently-curated review of Planecrash, the bulk of which is about summarising Yudkowsky’s worldview.
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences
If there’s decision-making about scarce resources, you will have an economy. Even superintelligence does not necessarily imply infinite abundance of everything, starting with the reason that our universe only has so many atoms. Multipolar outcomes seem plausible under continuous takeoff, which the consensus view in AI safety (as I understand it) sees as more likely than fast takeoff. I admit that there are strong reasons for thinking that the aggregate of a bunch of sufficiently smart things is agentic, but this isn’t directly relevant for the concerns about humans within the system in my post.
a value-aligned superintelligence directly creates utopia
In his review of Peter Singer’s commentary on Marx, Scott Alexander writes:
[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.
Peter Thiel might call this “indefinite optimism”: delay all planning or visualisation because there’s some later point where it’s trusted things will all sort themselves out. Now, if you think that takeoff will definitely be extremely hard and the resulting superintelligence will effortlessly take over the world, then obviously it makes sense to focus on what that superintelligence will want to do. But what if takeoff lasts months or years or decades? (Note that there can be lots of change even within months if the stakes look extreme to powerful actors!) Aren’t you curious about what an aligned superintelligence will end up deciding about society and humans? Are you so sure about the transition period being so short and the superintelligence being so unitary and multipolar outcomes being so unlikely that we’ll never have to worry about problems downstream of the incentive issues and competitive pressures that I discuss (which Beren recently had an excellent post on)? Are you so sure that there is not a single interesting, a priori deducible fact about the superintelligent economy beyond “a singleton is in charge and everything is utopia”?
The bottlenecks to compute production are constructing chip fabs; electricity; the availability of rare earth minerals.
Chip fabs and electricity generation are capital!
Right now, both companies have an interest in a growing population with growing wealth and are on the same side. If the population and its buying power begins to shrink, they will be in an existential fight over the remainder, yielding AI-insider/AI-outsider division.
Yep, AI buying power winning over human buying power in setting the direction of the economy is an important dynamic that I’m thinking about.
I also think the AI labor replacement is initially on the side of equality. [...] Now, any single person who is a competent user of Claude can feasibly match the output of any traditional legal team, [...]. The exclusive access to this labor is fundamental to the power imbalance of wealth inequality, so its replacement is an equalizing force.
Yep, this is an important point, and a big positive effect of AI! I write about this here. We shouldn’t lose track of all the positive effects.
Great post! I’m also a big (though biased) fan of Owain’s research agenda, and share your concerns with mech interp.
I’m therefore coining the term “prosaic interpretability”—an approach to understanding model internals [...]
Concretely, I’ve been really impressed by work like Owain Evans’ research on the Reversal Curse, Two-Hop Curse, and Connecting the Dots[3]. These feel like they’re telling us something real, general, and fundamental about how language models think. Despite being primarily empirical, such work is well-formulated conceptually, and yields gearsy mental models of neural nets, independently of existing paradigms.
[emphasis added]
I don’t understand how the papers mentioned are about understanding model internals, and as a result I find the term “prosaic interpretability” confusing.
Some points that are relevant in my thinking (stealing a digram from an unpublished draft of mine):
the only thing we fundamentally care about with LLMs is the input-output behaviour (I-O)
now often, a good way to study the I-O map is to first understand the internals M
but if understanding the internals M is hard but you can make useful generalising statements about the I-O, then you might as well skip dealing with M at all (c.f. psychology, lots of econ, LLM papers like this)
the Owain papers you mention seem to me to make 3 distinct types of moves, in this taxonomy:
finding some useful generalising statement about the I-O map behaviour (potentially conditional on some property of the training data) (e.g. the reversal curse)
creating a modified model M’ from M via fine-tuning on same data (but again, not caring about what the data actually does to the internals)
(far less centrally than the above!) speculating about what the internal structure that causes the behavioural patterns above might be (e.g. that maybe models trained on “A=B” learn to map representation(A) --> representation(B) in some MLP, instead of learning the general rule that A and B are the same thing and representing them internally as such)
So overall, I don’t think the type of work you mention is really focused on internals or interpretability at all, except incidentally in minor ways. (There’s perhaps a similar vibe difference here to category theory v set theory: the focus being relations between (black-boxed) objects, versus the focus being the internals/contents of objects, with relations and operations defined by what they do to those internals)
I think thinking about internals can be useful—see here for a Neel Nanda tweet arguing the reversal curse if obvious if you understand mech interp—but also the blackbox research often has a different conceptual frame, and is often powerful specifically when it can skip all theorising about internals while still bringing true generalising statements about models to the table.
And therefore I’d suggest a different name than “prosaic interpretability”. “LLM behavioural science”? “Science of evals”? “Model psychology”? (Though I don’t particularly like any of these terms)
If takeoff is more continuous than hard, why is it so obvious that there exists exactly one superintelligence rather than multiple? Or are you assuming hard takeoff?
Also, your post writes about “labor-replacing AGI” but writes as if the world it might cause near-term lasts eternally
If things go well, human individuals continue existing (and humans continue making new humans, whether digitally or not). Also, it seems more likely than not that fairly strong property rights continue (if property rights aren’t strong, and humans aren’t augmented to be competitive with the superintelligences, then prospects for human survival seem weak since humans’ main advantage is that they start out owning a lot of the stuff—and yes, that they can shape the values of the AGI, but I tentatively think CEV-type solutions are neither plausible nor necessarily desirable). The simplest scenario is that there is continuity between current and post-singularity property ownership (especially if takeoff is slow and there isn’t a clear “reset” point). The AI stuff might get crazy and the world might change a lot as a result, but these guesses, if correct, seem to pin down a lot of what the human situation looks like.
I’d argue that the incentives for governments to serve their people do in fact change given the nanobots, and that’s a significant part of why the radical AGI+nanotech leads to bad outcomes in this scenario.
Imagine two technologies:
Auto-nanobots: autonomous nanobot cloud controlled by an AGI, does not benefit from human intervention
Helper-nanobots: a nanobot cloud whose effectiveness scales with human management hours spent steering it
Imagine Anthropenmind builds one or the other type of nanobot and then decides whether to take over the world and subjugate everyone else under their iron fist. In the former case, their incentive is to take over the world, paperclips their employees & then everyone else, etc. etc. In the latter case, the more human management they get, the more powerful they are, so their incentive is to get a lot of humans involved, and share proceeds with them, and the humans have leverage. Even if in both cases the tech is enormously powerful and could be used tremendously destructively, the thing that results in the bad outcome is the incentives flipping from cooperating with the rest of humanity to defecting against the rest of humanity, which in turn comes about because the returns to those in power of humans go down.
(Now of course: even with helper-nanobots, why doesn’t Anthropenmind use its hard power to do a small but decapitating coup against the government, and then force everyone to work as nanobot managers? Empirically, having a more liberal society seems better than the alternative; theoretically, unforced labor is more motivated, cooperation means you don’t need to monitor for defection, principle-agent problems bite hard, not needing top-down control means you can organize along more bottom-up structures that better use local information, etc.)
Maybe helpful to distinguish between:
“Narrow” intelligence curse: the specific story where selection & incentive pressures by the powerful given labor-replacing AI disempowers a lot of people over time. (And of course, emphasizing this scenario is the most distinct part of The Intelligence Curse as a piece compared to AI-Enabled Coups)
“Broad” intelligence curse: severing the link between power and people is bad, for reasons including the systemic incentives story, but also because it incentivizes coups and generally disempowers people.
Now, is the latter the most helpful place to draw the boundary between the category definitions? Maybe not—it’s very general. But the power/people link severance is a lot of my concern and therefore I find it helpful to refer to it with one term. (And note that even the broader definition of IC still excludes some of GD as the diagram makes clear, so it does narrow it down)
Curious for your thoughts on this!