I don’t think your first point is obvious. We’ve had super smart humans (e.g. with IQ >200) and they haven’t been able to take over the world. (Although they didn’t have many of the advantages an AI might have, such as mass copying themselves over the internet.)
In general, the power(intelligence) curve is a big crux for me that we can’t fill in with data points yet (of course intelligence is also spiky). Imo we also have no idea where takeover-level intelligence is, what takeover-shape intelligence is, and what maximum AI would be.
I don’t think your first point is obvious. We’ve had super smart humans (e.g. with IQ >200) and they haven’t been able to take over the world. (Although they didn’t have many of the advantages an AI might have, such as mass copying themselves over the internet.)
In general, the power(intelligence) curve is a big crux for me that we can’t fill in with data points yet (of course intelligence is also spiky). Imo we also have no idea where takeover-level intelligence is, what takeover-shape intelligence is, and what maximum AI would be.
What do you mean by soft limit?