I consider this post as one of the most important ever written on issues of timelines and AI doom scenario. Not because it’s perfect (some of its assumptions are unconvincing), but because it highlights a key aspect of AI Risk and the alignment problem which is so easy to miss coming from a rationalist mindset: it doesn’t require an agent to take over the whole world. It is not about agency.
What RAAPs show instead is that even in a purely structural setting, where agency doesn’t matter, these problem still crop up!
This insight was already present in Drexler’s work, but however insightful Eric is in person, CAIS is completely unreadable and so no one cared. But this post is well written. Not perfectly once again, but it gives short, somewhat minimal proofs of concept for this structural perspective on alignment. And it also managed to tie alignment with key ideas in sociology, opening ways for interdisciplinarity.
I have made every person I have ever mentored on alignment study this post. And I plan to continue doing so. Despite the fact that I’m unconvinced by most timeline and AI risk scenarios post. That’s how good and important it is.
Just registering my own disagreement here—I don’t think it’s a key aspect, because I don’t think it’s necessary; the bulk of the problem IS about agency & this post encourages us to focus on the wrong problems.
I do agree that this post is well written and that it successfully gives proofs of concept for the structural perspective, for there being important problems that don’t have to do with agency, etc. I just think that the biggest problems do have to do with agency & this post is a distraction from them.
(My opinion is similar to what Kosoy and Christiano said in their comments)
I was mostly thinking of the efficiency assumption underlying almost all the scenarios. Critch assumes that a significant chunk of the economy always can and does make the most efficient change (everyone replacing the job, automated regulations replacing banks when they can’t move fast enough). Which neglects many potential factors, like big economic actors not having to be efficient for a long time, backlash from customers, and in general all factors making economic actors and market less than efficient.
I expect that most of these factors could be addressed with more work on the scenarios.
I consider this post as one of the most important ever written on issues of timelines and AI doom scenario. Not because it’s perfect (some of its assumptions are unconvincing), but because it highlights a key aspect of AI Risk and the alignment problem which is so easy to miss coming from a rationalist mindset: it doesn’t require an agent to take over the whole world. It is not about agency.
What RAAPs show instead is that even in a purely structural setting, where agency doesn’t matter, these problem still crop up!
This insight was already present in Drexler’s work, but however insightful Eric is in person, CAIS is completely unreadable and so no one cared. But this post is well written. Not perfectly once again, but it gives short, somewhat minimal proofs of concept for this structural perspective on alignment. And it also managed to tie alignment with key ideas in sociology, opening ways for interdisciplinarity.
I have made every person I have ever mentored on alignment study this post. And I plan to continue doing so. Despite the fact that I’m unconvinced by most timeline and AI risk scenarios post. That’s how good and important it is.
Just registering my own disagreement here—I don’t think it’s a key aspect, because I don’t think it’s necessary; the bulk of the problem IS about agency & this post encourages us to focus on the wrong problems.
I do agree that this post is well written and that it successfully gives proofs of concept for the structural perspective, for there being important problems that don’t have to do with agency, etc. I just think that the biggest problems do have to do with agency & this post is a distraction from them.
(My opinion is similar to what Kosoy and Christiano said in their comments)
I’d be interested in which bits strike you as notably “imperfect.”
I was mostly thinking of the efficiency assumption underlying almost all the scenarios. Critch assumes that a significant chunk of the economy always can and does make the most efficient change (everyone replacing the job, automated regulations replacing banks when they can’t move fast enough). Which neglects many potential factors, like big economic actors not having to be efficient for a long time, backlash from customers, and in general all factors making economic actors and market less than efficient.
I expect that most of these factors could be addressed with more work on the scenarios.