I found the “which comes first?” framing helpful. I don’t think it changes my takeaways but it’s a new gear to think about.
A thing I keep expecting you to say next but you haven’t quite said something like, is:
“Sure, there are differences when the AI becomes able to actually take over. But the shape of how the AI is able to take over, and how long we get to leverage somewhat-superintelligence, and how super that somewhat-superintellignce is, is not a fixed quantity. Our ability to study scheming and build control systems and get partial-buy-in from labs/government/culture.
And the Yudkowskian framing makes it sound like there’s a discrete scary moment, and the Schlegeris framing is that both where-that-point-is and how-scary-it-is are quite variable, which changes the strategic landscape noticeably
Does that feel like a real/relevant characterization of stuff you believe?
(I find that pretty plausible, and I could imagine it buying us like 10-50 years of knifes-edge-gradualist-takeoff-that-hasn’t-killed-us-yet, but that seems to me to have, in practice, >60% likelihood that by the end of those 50 years, AIs are running everything, they still aren’t robustly aligned, they gradually squeeze us out)
I found the “which comes first?” framing helpful. I don’t think it changes my takeaways but it’s a new gear to think about.
A thing I keep expecting you to say next but you haven’t quite said something like, is:
Does that feel like a real/relevant characterization of stuff you believe?
(I find that pretty plausible, and I could imagine it buying us like 10-50 years of knifes-edge-gradualist-takeoff-that-hasn’t-killed-us-yet, but that seems to me to have, in practice, >60% likelihood that by the end of those 50 years, AIs are running everything, they still aren’t robustly aligned, they gradually squeeze us out)
A more important argument is the one I give briefly here.