This topic is interesting to me because I expect safety-related decisions during the intelligence explosion to look more like war-time decisions than risk assessments for nuclear power plants: there will be lots of uncertainty about very complex systems with adversarial actors (instead of something where you understand things end-to-end that you can analyze carefully) and no safe action that is realistically reachable (“just don’t build ASI” might be more like “just make peace” than “just use coal instead of nuclear power plants”).
The amount of uncertainty and no-safe-action these books conveyed was roughly what I expected. The amount of bad-in-retrospect decisions was worse than I expected:
Bush is quite defensive of his decisions, but is quite open about how certain decisions seemed quite bad with the benefit of hindsight (Iraq not having a WMD program makes the war less useful, leaving post-war security to local forces ended up being way less successful than what people expected, etc.)
How the war was won (which is quite academic in comparison) also describes many cases of incorrect pre-war or during-war planning:
Air power ended up more powerful than expected in sea warfare, making battleships much less useful than expected
The damage from small amounts of targeted bombing ended up very overestimated, similarly for the effectiveness of area bombing to weaken morale, resulting in a lot of mid-war changes to strategy
The British expected the d-day landing to be less successful than it was and pushed back hard against it
US admirals did not appreciate the effectiveness of convoys (4% of tonnage lost vs 20% for independent boats) against submarine attacks until months after the US entry into war
MacArthur over-prioritized the Philippines, which caused a bunch of casualties while the battles in the Mariana Islands were much more important in winning the war (since it were Islands within bombing range of Japan)
How the war was won also argues that a lot of what helped win the war was “boring” technocratic abilities to understand the war machine and break it where it is the weakest (e.g. well-chosen targets (transports, aircraft manufacturing, and oil production) with massive and regular targeted bombing raids, using convoys, producing more and better planes, …), though it acknowledges that this is not a consensus view within the academic community studying WW2 (which is richer than I expected).
I think most of these decision-making failures seem to be a combination of it being difficult to anticipate somewhat complex dynamics (in a way that you could anticipate if you were just smarter—like a very powerful AI), some amount of organizational dysfunction and motivated reasoning, and some amount of bad or high-uncertainty data. The war is a way more complicated system and has slightly less fog of war than AI development, so maybe this will be much better with AIs, but it doesn’t inspire a lot of confidence, especially in the regime where AIs become better at strategy than humans.
Maybe AI risk reports should focus more on (probably vague) arguments for how much feedback from reality AI companies will get to check their assumptions (before it’s too late) and less on a detailed nuclear-power-plant-style analysis of how good the mitigations are.
Other takeaways from How the war was won:
I am surprised by how much within-alliance lying there was and how chill people seem about it in hindsight.
Roosevelt was trying to make the British think that the US would soon join the war and was trying to make the US population think they wouldn’t. He also did some war-preparation things to help the British right after the election that he only did then because he knew it would damage his election chances. One general also unilaterally strongly misrepresented numbers about the fraction of US effort already in the European Theater, which helped justify a fraction of resources spent in the Pacific Theater bigger than what the British thought (or sth like that, I forgot the exact incident). And Churchill’s strategies to avoid doing a landing in France also seemed quite deceptive.
And people at the time knew there were big tensions within the alliance, so this probably didn’t come as a massive shock. An alliance can be very powerful but it doesn’t remove minor betrayals and big tensions.
I think it’s easy to underestimate how big of a project WW2 was, and what crazy options are available when you try that hard. For example, Germany had (at its peak) roughly a million people working on repairing damage to factories and transports from bombing. The amount of resources spent on sea and air power was also somewhat insane (it’s not that surprising it’s high for the US/UK/Japan, but I was surprised to learn it was over 50% for Germany—its war is less well described as a land war than I thought). When people say “not dying from AGI might be as hard as WW2”, the picture you should have in your head should not just be frontline fighters and regular industry, but also massive efforts on things that are usually not even mentioned in history books.
It’s always a bit surprising to me how contingent war dynamics are on random facts about physics. For example the day-night cycle plays a big role in WW2 air warfare: airplanes can’t see precise targets at night, which weakens air power and enables mobility in a way that is not possible during the day (where troops on the ground would be attacked while on the move), and also weakens anti-air defenses, which made area bombing of cities safer for the attackers than daylight bombing of precise targets.
I listened to 2 books about decision-making during wars: How the War Was Won: Air-Sea Power and Allied Victory in WW2 and Decision Points by George W Bush.
This topic is interesting to me because I expect safety-related decisions during the intelligence explosion to look more like war-time decisions than risk assessments for nuclear power plants: there will be lots of uncertainty about very complex systems with adversarial actors (instead of something where you understand things end-to-end that you can analyze carefully) and no safe action that is realistically reachable (“just don’t build ASI” might be more like “just make peace” than “just use coal instead of nuclear power plants”).
The amount of uncertainty and no-safe-action these books conveyed was roughly what I expected. The amount of bad-in-retrospect decisions was worse than I expected:
Bush is quite defensive of his decisions, but is quite open about how certain decisions seemed quite bad with the benefit of hindsight (Iraq not having a WMD program makes the war less useful, leaving post-war security to local forces ended up being way less successful than what people expected, etc.)
How the war was won (which is quite academic in comparison) also describes many cases of incorrect pre-war or during-war planning:
Air power ended up more powerful than expected in sea warfare, making battleships much less useful than expected
The damage from small amounts of targeted bombing ended up very overestimated, similarly for the effectiveness of area bombing to weaken morale, resulting in a lot of mid-war changes to strategy
The British expected the d-day landing to be less successful than it was and pushed back hard against it
US admirals did not appreciate the effectiveness of convoys (4% of tonnage lost vs 20% for independent boats) against submarine attacks until months after the US entry into war
MacArthur over-prioritized the Philippines, which caused a bunch of casualties while the battles in the Mariana Islands were much more important in winning the war (since it were Islands within bombing range of Japan)
How the war was won also argues that a lot of what helped win the war was “boring” technocratic abilities to understand the war machine and break it where it is the weakest (e.g. well-chosen targets (transports, aircraft manufacturing, and oil production) with massive and regular targeted bombing raids, using convoys, producing more and better planes, …), though it acknowledges that this is not a consensus view within the academic community studying WW2 (which is richer than I expected).
I think most of these decision-making failures seem to be a combination of it being difficult to anticipate somewhat complex dynamics (in a way that you could anticipate if you were just smarter—like a very powerful AI), some amount of organizational dysfunction and motivated reasoning, and some amount of bad or high-uncertainty data. The war is a way more complicated system and has slightly less fog of war than AI development, so maybe this will be much better with AIs, but it doesn’t inspire a lot of confidence, especially in the regime where AIs become better at strategy than humans.
Maybe AI risk reports should focus more on (probably vague) arguments for how much feedback from reality AI companies will get to check their assumptions (before it’s too late) and less on a detailed nuclear-power-plant-style analysis of how good the mitigations are.
Other takeaways from How the war was won:
I am surprised by how much within-alliance lying there was and how chill people seem about it in hindsight.
Roosevelt was trying to make the British think that the US would soon join the war and was trying to make the US population think they wouldn’t. He also did some war-preparation things to help the British right after the election that he only did then because he knew it would damage his election chances. One general also unilaterally strongly misrepresented numbers about the fraction of US effort already in the European Theater, which helped justify a fraction of resources spent in the Pacific Theater bigger than what the British thought (or sth like that, I forgot the exact incident). And Churchill’s strategies to avoid doing a landing in France also seemed quite deceptive.
And people at the time knew there were big tensions within the alliance, so this probably didn’t come as a massive shock. An alliance can be very powerful but it doesn’t remove minor betrayals and big tensions.
I think it’s easy to underestimate how big of a project WW2 was, and what crazy options are available when you try that hard. For example, Germany had (at its peak) roughly a million people working on repairing damage to factories and transports from bombing. The amount of resources spent on sea and air power was also somewhat insane (it’s not that surprising it’s high for the US/UK/Japan, but I was surprised to learn it was over 50% for Germany—its war is less well described as a land war than I thought). When people say “not dying from AGI might be as hard as WW2”, the picture you should have in your head should not just be frontline fighters and regular industry, but also massive efforts on things that are usually not even mentioned in history books.
It’s always a bit surprising to me how contingent war dynamics are on random facts about physics. For example the day-night cycle plays a big role in WW2 air warfare: airplanes can’t see precise targets at night, which weakens air power and enables mobility in a way that is not possible during the day (where troops on the ground would be attacked while on the move), and also weakens anti-air defenses, which made area bombing of cities safer for the attackers than daylight bombing of precise targets.