Seth, i’m worried your argument proves too much
Many ppl are already more or less “completely safe” from a physical/health perspective
But they still are anxious, and worry about what others think of them.
Similarly, wont’ a dictator want to influence culture and ppl’s views/ideas to maintain their high opinions of him?
Argument for giving most resources to future generations
To get to a great future, we need ppl to be open-minded and reflective. Including about fundamental ideas in ethics, weird simulation possibilities, and all sorts of stuff.
But adults are very close-minded. Society is only open-minded and reflective because the old generations dies out. Young people look at the arguments and evidence with fresh eyes, shedding the biases and prejudices of their elders.
So if the current generation decides how the cosmic endowment is used, the vast majority of people will almost certainly squander it. The best hope is to punt to future generations.
Sceptic: ah, but future generations could be worse in lots of ways. Their values might shift for random cultural reasons, so that by our lights it’s bad to give their values influence. Or maybe they get mind-hacked by mind-viruses while young and impressionable.
My response: Yes, that could all happen. But we know that most ppl in the current generation will get <1% of value (by the lights of someone who values reflection). Future generations could be 100x worse, or they could be 100x better. We don’t know. But given how low the expected value is from the current generation, it’s worth rolling the dice.
Also, this isn’t a dick move to the current generation. Most ppl really value handing something better to their kids! And they value educating their kids to be intelligent and wise and kind.
Handing off most power to future generations, and raising them to use that responsibility wisely and ethically, is something we could all rally behind!
Thanks, great post, and I’m looking forward to reading the rest of the sequence.
But I think many of the readers that you’re looking to convince will be coming in thinking that it is good for the world that Anthropic exists. Yes, capabilities were accelerated. But some high-quality alignment work has been done, at least more than would have been done by the counterfactual frontier AI company by the time ai was this capable. And now there’s a frontier AI company that is deeply concerned about AI takeover risk and will likely support efforts to pause.
So I would be interested to hear more about why you think things are worse than they would be in a counterfactual where AI safety people did not push forward AI capabilities and join frontier AI companies.
And yeah, my impression is that not much has come out of agent foundations, which feeds into the counterfactual
One Q I have:
How would this change if we augmented the Transformer so the first layer can “look back” at the final layer from the previous token?
Optimist: Now the first layer activations can express very complicated concepts that involve many steps of sequential sequencing. So perhaps this helps a lot.
Sceptical: but from the perspective of the weights that produce the first layer activations, all the new concepts still look like gobbledegook.
Optimist: True, but in this set-up the the weights are describing the learning rule, they’re not trying to “understand” the object-level content at all. It is the activations that come to “understand” the new concepts.
Well, if it’s doable after this one simple architectural change, why not via chains of thought like Alex was saying?
Firstly, you can send back detailed and novel representations from the final layer activations, but can’t do either from a new token.
Second, consider the specific weights that create the first layer activations.
With the architectural change, there are specialised weights trained to give sophisticated novel representations to the first layer (stealing them from the final layer of the previous token).
But with current transformers, the same weights that map words to existing concepts (that map the word “dog” to a complex representation of dogs in concept space) would have to do double duty and also help create sophisticated novel concepts.
I think these two differences are pretty big.
Flagging that I think this back-and-forth is very important.
The “limited serial depth” point seems like the core of Steven’s argument for “can’t imitation learn how to continual-learn”. (If he thinks there’s other important parts, i’d be interested to hear it!) And so the more natural statement of the conclusion to me would be “transformers can’t do continual learning at runtime, even if they’ve been trained on trajectories of continual learning”
Fwiw, i feel pretty convinced by Steven’s argument here. Human continual learning in fact involves parameter changes with massive serial depth. It seems likely that an alg that tries to compress that into 10 serial steps of new concept formation will be extremely inefficient and fail in practice.
The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.
Thanks, this is helpful and not an argument i’ve come across before!
One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.
I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.
One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher
I agree that the bureaucracy/scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.
And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).
I guess i’m saying that doing that experiments at small scale could very easily cause large inaccuracies, and this is a plausible reason you might be underestimating progress in your 1.1 bucket!
Cool! I think the UK could play a leading role here, convening a middle powers coalition and leading on its strategy
I thought about this a bit more.
tldr: Under a simple model and reasonable assumptions, if we automate AI R&D and compute growth stays constant then the pace of AI software progress is 3-5X faster. This means the pace of overall AI progress would be 2-3X faster.
Assume AI software R&D is Cobb Douglas:
g_S = L^alpha E^beta S^-1/r
L = cognitive labour
E = experimental compute
r governs ideas getting harder to find as you ramp up both cognitive labour and experimental compute. (Note I’ve often confusingly defined r as the returns when you just ramp up cognitive labour—really I should call that r_cog = r * alpha.)
In this system, before automating away humans, it turns out that:
if L grows exponentially at g_L then (holding E fixed) g_S = g_L * alpha * r
if E grows exponentially at g_E then (holding L fixed) g_S = g_E * beta * r
When you automate AI R&D, L = S. In this situation, it turns out that:
if L grows exponentially at g_L (eg due to exogenously increasing compute), then:
g_S = g_L * alpha * r / (1 - alpha * r)
And if r * alpha > 1 we get a software-only intelligence explosion!
if E grows exponentially at g_E (eg due to exogenously increasing compute), S grows at:
g_S = g_E * beta * r / (1 - alpha * r)
In other words, even absent an SIE, automating AI R&D boosts the standard growth rates by a factor of 1 / (1 - alpha * r) due to the fizzling feedback loop of “better software → better AI researchers → better software”.
This model allows us to ballpark how much faster overall AI progress would be in a regime with full automation but no SIE.
That regime causes two changes:
g_L gets faster. Firstly, compute growth is somewhat faster than the growth of human AI researchers. Secondly, L is superlinear in compute bc you can run faster and smarter models with more compute.
All growth rates are boosted by a factor of 1 / (1 - alpha * r). Ryan used an estimate of r_cog = r * alpha = 0.7. So this is a boost by a factor of 1 / (1 − 0.7) = ~3.
Concretely, let’s make the following assumptions:
g_L = g_E today (conservatively assume compute and human researchers have grown at the same pace)
alpha = beta = 0.5
r = 1.4 (so that we recover Ryan’s r * alpha = 0.7)
After automating AI R&D, L grows twice as fast as compute: g_L = 2 * g_E
Then today the total pace of software progress is :
g_S = progress due to growing labour + progress due to growing compute
= g_E * alpha * r + g_E * beta * r
= g_E * r
And after AI R&D automation g_L is 2X faster and everything gets boosted by a factor of 3. The pace of software progress is:
g_S = progress due to growing labour + progress due to growing compute
= (2 * g_E * alpha * r + g_E * beta * r)*3
= 4.5 * g_E * r
So that’s 4.5x faster software progress, holding the rate of compute growth constant! If software and compute contribute equally to AI progress, that’s a bit under 3x faster total AI progress.
I think not too surprising, given we’re using an r*alpha value close to 1.
If we used r*alpha =0.5, our boost factor shrinks to 2 and we’d get 3x faster software progress and 2x faster total AI progress.
This isn’t what I want because
yeah i adjust for this in my other comment
I don’t understand, the whole point of the experiments is to get us better labor.
yeah, so they do—a doubling of cumulative experiments drives 0.7 doublings of software. And then that better software does more cognitive work to improve software further still. But it doesn’t increase the amount of compute available for experiments, so the feedback loop doesn’t go full circle.
For “cognitive labour” we have: more compute → more cog labour → beter software → more cog labour → better software… So you get and initial boost from extra compute which then ratchets up with the software feedback loop
But for “experimental compute” we have: more compute → more experiments → better software → more cog labour → better software...
So while it feeds into the software feedback loop, it doesn’t loop back to “more experimental compute”.
I’m not sure if my math full priced this in. I’ll think about that more.
Thanks for the push—published!
while 4x more compute at current margins is pretty close to as good as getting compute that’s 4x serially faster
Why?
In fact the above is conservative in assuming one compute doubling yields just one labor doubling. You get that by running more copies. But you’ll train smarter models
If one compute doubling yields two labor doublings (holding software constant) then, rerunning the above analysis:
One compute doubling → two labor doublings (constant software) → in equilibrium, 6 labor doublings drive 4 software doublings (r=0.7)
today one compute doubling is associated with one (or fewer) labor doublings. In the new equ it’s associated with 6
How much faster would ai progress be compared to today?
Suppose that today labor and compute due exp are equally important and produce progress via cobb Douglas. Suppose they grow at equal rates
Then with full automation and r=0.7, labor is growing 6x faster than before. And overall ai progress is 3.5x faster
Of course, compute may be growing more slowly
I haven’t yet found a nice and clean way to model this in isolation
Not sure if this is what you want, but:
Assume r= 0.7
Assume that all that doubling compute does is double labor
--> in equilibrium, doubling compute doubles software 2 times
Why? bc with r=0.7 you need to double labor inputs 3 times in order get 2 software doublings. Which matches the equ: we got 3 labor doublings (one from compute and two from software); and this led to 2 software doublings
One way to think of this: today labor grows more slowly than compute. But I’m this world it grows 3x faster.
(I can explain better if this is unclear)
The above is conservative: it assumes the doubling compute doubles ONLY labor, rather than also doubling experiments
If it also doubles experiments, and experiments and labor are equally important, then I believe that the extra doubling of experiments gives us an extra 0.7 doublings of software. (It doesn’t compound further bc there’s no feedback loop to more experiments like there is to more labor)
So overall, a compute doubling yields 2+0.7= 2.7 software doublings.
There is a gradual boost setting that smooths out the automation returns over a longer period, but I think this period is unrealistically long such that you don’t see one-time speed-up effects
Would it help if we added another param controlling how many years the boost occurs over?
How can the middle powers avoid getting trounced during the intelligence explosion? A plan.
Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.
How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)
On a very high-level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).
That final boded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.
My plan for the middle powers has two stages:
Maintain as much economic and military leverage as possible during the intelligence explosion.
Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.
Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.
I’ll walk through each stage in turn.
The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.
Aside: why access to frontier AI might be sufficient for middle powers to stay economically relevant indefinitely
The hope here is that:
Most of the economic surplus from AI is not captured by AI companies. To create economic value, AI must be combined with complementary inputs: factories, human physical labour, know-how of human experts, relationships with suppliers, trusted brands, etc. How much of the surplus will be captured by AI companies vs the owners of these complementary inputs? Optimistically: producers of general-purpose technologies often capture only a small fraction of surplus; and multiple frontier AI companies might sell similar products and bid each other down on cost.
Most of the economic surplus from AI occurs outside the US. The majority of these complementary inputs are situated outside the US. So most AI-driven economic value-add should occur outside US borders.
If (1) and (2) both hold, a significant fraction of AI’s economic surplus will accrue to non-US actors.
But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:
Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.
Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.
Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.
Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.
Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.
Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.
Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock-in the deal.
(h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)
Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?
Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.
Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.
The catch-all meta-point here is waking middle powers up to superintelligence.
I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.
If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?
First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.
An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.
The optimistic case here is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.
The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the datacentres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.
The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.
The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly. Eg demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.
(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.
I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!
But here’s a brainstorm of pros and cons from a more impartial perspective.
Pros to empowering middle powers:
Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.
More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.
Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” bc they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.
Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.
Pause AI. They could potentially pressure US/China to pause/slowdown reckless AI development.
Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of pro-social norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.
Cons of empowering middle powers. Multipolarity has its own downsides:
more likely to lead to war.
can drive extreme competition, eg racing to develop a dangerous technology, or to hand off power to misaligned AI
Harder to prevent harms from offense-dominant technologies like bioweapons
my plan involves waking up middle powers, which could shorten timelines
But alas, I currently think the most important use-case for AGI is figuring out true important things about the world (esp. related to ASI alignment and strategy) and explaining those things to the human. For this process to be effective, we cannot have an AGI that’s unconcerned with what we wind up believing after the discussion—that’s a recipe for slop-and-doom, or just an AI that’s incomprehensible and unhelpful. Rather, I want the AI to be like a disagreeable nerd that wants us to have a good understanding, notices areas where we’re confused, and is brainstorming and strategizing on how to help set us straight by improving its clarity and pedagogy. This strategizing is clearly a form of optimization, and the target of the optimization is related to the human’s eventual desires (well, it’s nominally about the human’s beliefs, but beliefs and desires are entangled)
What about if AI optimizes for humans fully understanding the evidence and arguments that it has discovered but does not optimize for what humans actually end up believing about the bottom line?
So if it discovered powerful arguments for misalignment, it would make sure to explain them in a way that the humans fully understand those arguments. But if humans had some prior/bias that caused them to dismiss the argument even when they fully understand it, the AI would not try and optimize around that.
If failure of alignment → schemer that wants to seize power, then ASI alignment is one shot.
But if failure of alignment → non-schemer misalignment (eg reward hacking, or flailing misgeneralisation), then we failure isn’t existential
So I think p(scheming | alignment failure) is a crux here
Fwiw it’s unclear to me whether epoch’s methodology would make this mistake
After 2018 they have many data points at different times and compute scales. In principle, that should allow them to disentangle alg progress from the effect of compute. If models trained with the the same compute but later in time are no better, they shouldn’t find any alg progress. (But I haven’t thought about this in a while, and the strong correlation between time and compute in their data makes their results super noisy)
Glad you like it!
As you suggest, I think the parity must be achievable substantially before this point. AI will have such massive advantages in other areas, by the time they match human sample efficiency they will be way way way important than humans to overall R&D. And so parity must come substantially sooner.
Not sure if I follow. I expect sample efficiency to improve continuously and significant acceleration to happen before it reaches human level. If the r<1 then acceleration will be limited, and so maybe AI progress is steady at 2x faster even as sample efficiency reaches and surpasses human-level.