It’s been about 8 months since this post and Buck’s comment above.
At the time, I didn’t bother replying to Buck’s comment because it didn’t really say much. My post basically said “this control agenda doesn’t seem to address anything important”, illustrated with a slew of specific examples of ways-things-go-wrong which seem-to-me to account for far more probability mass than scheming in early transformative AGI. Buck’s response was basically “yeah, those are real limitations, but IDK man scheming seems intuitively important?”. There’s a substantive argument to be had here about why I expect scheming of early transformative AGI to be either unlikely or easy to fix (relative to other problems), whereas Buck expects the opposite, and we haven’t had that debate.
Anyway, I’m leaving this comment now because I think some people saw that I had a critique, that Buck had responded, and that I hadn’t responded back, and therefore assumed that the ball was in my court and I wasn’t engaging. That’s not my understanding of what’s going on here; I think Buck has basically not argued back substantively against the core of the critique, but also I haven’t argued strongly against his core crux either. We’ve identified a crux, and that’s the state of things. (And to be clear that’s fine, not every debate is worth having.)
It’s been about 8 months since this post and Buck’s comment above.
At the time, I didn’t bother replying to Buck’s comment because it didn’t really say much. My post basically said “this control agenda doesn’t seem to address anything important”, illustrated with a slew of specific examples of ways-things-go-wrong which seem-to-me to account for far more probability mass than scheming in early transformative AGI. Buck’s response was basically “yeah, those are real limitations, but IDK man scheming seems intuitively important?”. There’s a substantive argument to be had here about why I expect scheming of early transformative AGI to be either unlikely or easy to fix (relative to other problems), whereas Buck expects the opposite, and we haven’t had that debate.
Anyway, I’m leaving this comment now because I think some people saw that I had a critique, that Buck had responded, and that I hadn’t responded back, and therefore assumed that the ball was in my court and I wasn’t engaging. That’s not my understanding of what’s going on here; I think Buck has basically not argued back substantively against the core of the critique, but also I haven’t argued strongly against his core crux either. We’ve identified a crux, and that’s the state of things. (And to be clear that’s fine, not every debate is worth having.)