the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully.
I also wonder of the extent to which rationalism helped or didn’t help to avert this. I suspect that the counterfactual AI safety would be accidentally restricted to product safety like Agent-4 obeying the instructions (until Agent-4 commits genocide). However, there are issues like “The current bottleneck is political will, not research” with which rationalism didn’t help or is unable to help because the society cannot grasp the extent and likelihoodness of the threat.
I also wonder of the extent to which rationalism helped or didn’t help to avert this. I suspect that the counterfactual AI safety would be accidentally restricted to product safety like Agent-4 obeying the instructions (until Agent-4 commits genocide). However, there are issues like “The current bottleneck is political will, not research” with which rationalism didn’t help or is unable to help because the society cannot grasp the extent and likelihoodness of the threat.