I think this continues to miss many of the objections and dismissals, and would benefit from some numeric estimates. I’m either a hopeless doomer or a polyanna optimist, depending on who’s asking and what their non-AI estimates of doom are. I’ve estimated between 0.25% and 1% chance of civilizational disaster (mostly large-scale war or irreversible climate tipping point) since the mid-90s. That’s at least 5% per decade (with a fair variance). With AI to accelerate things, I put it marginally higher, but still kind of expect that humans do most of the destruction.
Further, I think the first two steps are often the most controversial/non-obvious ones.
We’re going to build superintelligent AI.
I haven’t seen an operational definition yet, and I don’t know what the scaling curve looks like, or whether the current (impressive, but not super) progress actually is the same dimension. I’d give it less than 10% in the next decade that a transformative, creative superintelligence on the same scale as Eliezer seems to be imagining exists..
It will be agent-like, in the sense of having long-term goals it tries to pursue.
I have seen no progress on this, and don’t think it’s likely that a fully-orthogonal long-term goal set will happen. I DO think it’s likely that somewhat longer-term contexts and goals will happen, perhaps to weeks or months without human intervention/validation, but probably not fully so.
NOTE: this does NOT argue against AI being very powerful, and misused (accidentally or intentionally) by humans to do catastrophic harms. I don’t think it’s automatic, but I think it’s a huge risk of building such a powerful tool.
Call it 10x more risky than nuclear weapons. You don’t need to argue that it’s autonomously misaligned, just that HUMANS are prone to such things and this tool could be horrifically effective.
I think this continues to miss many of the objections and dismissals, and would benefit from some numeric estimates. I’m either a hopeless doomer or a polyanna optimist, depending on who’s asking and what their non-AI estimates of doom are. I’ve estimated between 0.25% and 1% chance of civilizational disaster (mostly large-scale war or irreversible climate tipping point) since the mid-90s. That’s at least 5% per decade (with a fair variance). With AI to accelerate things, I put it marginally higher, but still kind of expect that humans do most of the destruction.
Further, I think the first two steps are often the most controversial/non-obvious ones.
I haven’t seen an operational definition yet, and I don’t know what the scaling curve looks like, or whether the current (impressive, but not super) progress actually is the same dimension. I’d give it less than 10% in the next decade that a transformative, creative superintelligence on the same scale as Eliezer seems to be imagining exists..
I have seen no progress on this, and don’t think it’s likely that a fully-orthogonal long-term goal set will happen. I DO think it’s likely that somewhat longer-term contexts and goals will happen, perhaps to weeks or months without human intervention/validation, but probably not fully so.
NOTE: this does NOT argue against AI being very powerful, and misused (accidentally or intentionally) by humans to do catastrophic harms. I don’t think it’s automatic, but I think it’s a huge risk of building such a powerful tool.
Call it 10x more risky than nuclear weapons. You don’t need to argue that it’s autonomously misaligned, just that HUMANS are prone to such things and this tool could be horrifically effective.