This is tangential, but I wanted to note that, a slow takeoff is, at least in some ways, a worse scenario than fast takeoff in terms of AI risk.
Designing an aligned AI (probably) involves trade-offs between safety and capability. In a fast takeoff scenario, the FAI “only” has to be capable enough to prevent UFAI from ever coming into being, in order for AI risk to be successfully mitigated. The most powerful potentially adversarial agents in its environment would be mere humans (and, yes, humans should be counted as somewhat adversarial since they are the source of the UFAI risk). In a (sufficiently) slow takeoff scenario, there will be a number of competing superintelligences, and it seems likely that at least some of them will not be aligned. This means that the FAIs have to be capable enough to effectively compete with UFAIs that are already deployed (while remaining safe).
Yes. I agree that neither scenario is strictly easier than the other. There are some plans that only work if takeoff is slow, and some plans that only work if takeoff is fast. Unfortunately, if you make the real world messier, I expect that neither of those categories is particularly likely to work well in practice. If you have a plan that works across several simple situations, then I think it’s reasonable to hope it will work well in the (complicated) real situation. But if you have N plans, each of which works well in 1 of N simple situations, then you are liable to be overfitting.
This is tangential, but I wanted to note that, a slow takeoff is, at least in some ways, a worse scenario than fast takeoff in terms of AI risk.
Designing an aligned AI (probably) involves trade-offs between safety and capability. In a fast takeoff scenario, the FAI “only” has to be capable enough to prevent UFAI from ever coming into being, in order for AI risk to be successfully mitigated. The most powerful potentially adversarial agents in its environment would be mere humans (and, yes, humans should be counted as somewhat adversarial since they are the source of the UFAI risk). In a (sufficiently) slow takeoff scenario, there will be a number of competing superintelligences, and it seems likely that at least some of them will not be aligned. This means that the FAIs have to be capable enough to effectively compete with UFAIs that are already deployed (while remaining safe).
Yes. I agree that neither scenario is strictly easier than the other. There are some plans that only work if takeoff is slow, and some plans that only work if takeoff is fast. Unfortunately, if you make the real world messier, I expect that neither of those categories is particularly likely to work well in practice. If you have a plan that works across several simple situations, then I think it’s reasonable to hope it will work well in the (complicated) real situation. But if you have N plans, each of which works well in 1 of N simple situations, then you are liable to be overfitting.