Trying to avoid capability contributions also means avoiding alignment contributions.
I actually agree with you on this. My most-preferred future is a bit different: slow down AI overall (both capabilities and alignment) so that other things can happen in the meanwhile. If building AI is widely seen as a bad thing, talented folks feel dirty for joining it (instead of feeling virtuous because 80K hours is recommending AI careers), society has more time to build defenses (like algorithmic liability), near-AI gets planted more widely in society before growing too fast (thus making unilateral takeoffs harder), technologies that are complementary rather than substitute for humans get comparatively more time and investment (like intelligence amplification, genetic engineering, or thought interfaces), and failing all that, at least humanity gets a little more time to survive.
I’m not sure I have a most-preferred future—I guess it would be everyone becoming aware of AI risk and the U.N. with US and China at the helm co-ordinating a joint pause and eventual AGI project. But people who point out this is super dangerous itself due to power centralization are correct, and on top of that I think it’s possible that instead of co-ordination, everyone becoming aware of AI risk leads to racing, sabotage, and other drastic actions.
The current situation where the leading labs are moderately risk aware and the rest of the world doesn’t seem that freaked out, while still being freaked out enough to potentially apply some brakes and regulations, is pretty good I think.
Your preferred future seems both nearly-impossible, not that good if it happens, and if we fall short of total victory on your goal then it comes out strongly negative—if we shame AI researchers and only some people worried about AI risk quit, and most people not worried about it stay, then all we’ve done is slow things down 1% and decreased the awareness of the field by 10%. I agree that more time to adjust and develop things like algorithmic liability would be very helpful though, I just think the price of buying that extra time through researcher-exit is absurdly high.
I actually agree with you on this. My most-preferred future is a bit different: slow down AI overall (both capabilities and alignment) so that other things can happen in the meanwhile. If building AI is widely seen as a bad thing, talented folks feel dirty for joining it (instead of feeling virtuous because 80K hours is recommending AI careers), society has more time to build defenses (like algorithmic liability), near-AI gets planted more widely in society before growing too fast (thus making unilateral takeoffs harder), technologies that are complementary rather than substitute for humans get comparatively more time and investment (like intelligence amplification, genetic engineering, or thought interfaces), and failing all that, at least humanity gets a little more time to survive.
I’m not sure I have a most-preferred future—I guess it would be everyone becoming aware of AI risk and the U.N. with US and China at the helm co-ordinating a joint pause and eventual AGI project. But people who point out this is super dangerous itself due to power centralization are correct, and on top of that I think it’s possible that instead of co-ordination, everyone becoming aware of AI risk leads to racing, sabotage, and other drastic actions.
The current situation where the leading labs are moderately risk aware and the rest of the world doesn’t seem that freaked out, while still being freaked out enough to potentially apply some brakes and regulations, is pretty good I think.
Your preferred future seems both nearly-impossible, not that good if it happens, and if we fall short of total victory on your goal then it comes out strongly negative—if we shame AI researchers and only some people worried about AI risk quit, and most people not worried about it stay, then all we’ve done is slow things down 1% and decreased the awareness of the field by 10%. I agree that more time to adjust and develop things like algorithmic liability would be very helpful though, I just think the price of buying that extra time through researcher-exit is absurdly high.