Not entirely related to this post, but I wonder how much of a crux takeover difficulty is. You talk a lot about how we don’t seem on track to have AIs capable of genuinely creative insights or anything close to sample efficiency with humans, and while I’m sympathetic to that take I don’t entirely understand your reasons for thinking it’s so important.
Even if we don’t have AIs close to sample efficiency with humans, they will still be able to self replicate and be given control over large swaths of our economy and be (in my view) capable of killing us all (via bioweapon most likely, but other possibilities exist). I doubt that they will take over unless they can be fully self sufficient, which I imagine to be a much higher bar than killing us all, but it still doesn’t seem to necessarily require truly novel creative insights (like cantors theorem) or sample efficiency.
Also (if by chance you are familiar with MTG) do you think we’ll get general-purpose AI systems which can beat humans in a new draft format? I suspect under your view of the world we won’t, though I’m not sure if this is particularly load bearing.
If humans decide to disempower themselves, sure. That seems a lot slower and more reversible than a scenario where the AI systems are actually preventing humans taking back power.
Re/ MtG, that specific question has occurred to me as interesting re/ AI capabilities. But I don’t really know enough to figure out how hard it is. I would not especially bet against “an AI will win a new draft format tournament by 2035” or something.
If killing everyone, or killing enough that civilization is heavily destabilized, is on the table then preventing humans from taking back power is as well. The main thing that would prevent this imo is an AI being incapable of self sufficiency. Do you not think we are particularly close to either level of capability?
I think it’s quite difficult to kill all humans / forcibly permanently disempower all humans. So yeah, to answer your original question, that’s probably a crux for something.
Not entirely related to this post, but I wonder how much of a crux takeover difficulty is. You talk a lot about how we don’t seem on track to have AIs capable of genuinely creative insights or anything close to sample efficiency with humans, and while I’m sympathetic to that take I don’t entirely understand your reasons for thinking it’s so important.
Even if we don’t have AIs close to sample efficiency with humans, they will still be able to self replicate and be given control over large swaths of our economy and be (in my view) capable of killing us all (via bioweapon most likely, but other possibilities exist). I doubt that they will take over unless they can be fully self sufficient, which I imagine to be a much higher bar than killing us all, but it still doesn’t seem to necessarily require truly novel creative insights (like cantors theorem) or sample efficiency.
Also (if by chance you are familiar with MTG) do you think we’ll get general-purpose AI systems which can beat humans in a new draft format? I suspect under your view of the world we won’t, though I’m not sure if this is particularly load bearing.
If humans decide to disempower themselves, sure. That seems a lot slower and more reversible than a scenario where the AI systems are actually preventing humans taking back power.
Re/ MtG, that specific question has occurred to me as interesting re/ AI capabilities. But I don’t really know enough to figure out how hard it is. I would not especially bet against “an AI will win a new draft format tournament by 2035” or something.
If killing everyone, or killing enough that civilization is heavily destabilized, is on the table then preventing humans from taking back power is as well. The main thing that would prevent this imo is an AI being incapable of self sufficiency. Do you not think we are particularly close to either level of capability?
apologies if you’ve addressed this elsewhere
I think it’s quite difficult to kill all humans / forcibly permanently disempower all humans. So yeah, to answer your original question, that’s probably a crux for something.
Makes sense. Not sure I agree but thanks for the responses.