It seems to me that there’s a counter-argument available to the “hardware overhang” argument for discontinuous takeoff that doesn’t apply to the “hacking” argument, namely that for any AI that achieves a high level of capability by taking advantage of hardware overhang, there will be an AI that arrives a bit earlier and achieves a somewhat lower level of capability by taking advantage of the same hardware overhang (e.g., because it has somewhat worse algorithms, or somewhat less or lower quality training data). Unlike the “hacking” scenario, in the generic “hardware overhang” scenario, there’s not an apparent threshold effect that could cause a discontinuity.
(Curiously, Paul Christiano’s and AI Impacts’s posts arguing against discontinuous takeoff both ignore “hardware overhang” and neither give this counter-argument. Neither of them mention the “hacking” argument either, AFAICT.)
Wasn’t hardware overhang the argument that if AGI is more bottlenecked by software than hardware, then conceptual insights on the software side could cause a discontinuity as people suddenly figured out how to use that hardware effectively? I’m not sure how your counterargument really works there, since the AI that arrives “a bit earlier” either precedes or follows that conceptual breakthrough. If it precedes the breakthrough, then it doesn’t benefit from that conceptual insight so won’t be powerful enough to take advantage of the overhang, and if it follows it, then it has a discontinuous advantage over previous systems and can take advantage of hardware overhang.
---
Separately, your comment also feels related to my argument that focusing on just superintelligence is a useful simplifying assumption, since a superintelligence is almost by definition capable of taking over the world. But it simplifies things a little too much, because if we focus too much on just the superintelligence case, we might miss the emergence of a “dumb” AGI which nevertheless had the “crucial capabilities” necessary for a world takeover.
In those terms, “having sufficient offensive cybersecurity capability that a hacking attempt can snowball into a world takeover” would be one such crucial capability that allowed for a discontinuity.
I think Bostrom uses the term “hardware overhang” in Superintelligence to point to a cluster of discontinuous takeoff scenarios including this one
It seems to me that there’s a counter-argument available to the “hardware overhang” argument for discontinuous takeoff that doesn’t apply to the “hacking” argument, namely that for any AI that achieves a high level of capability by taking advantage of hardware overhang, there will be an AI that arrives a bit earlier and achieves a somewhat lower level of capability by taking advantage of the same hardware overhang (e.g., because it has somewhat worse algorithms, or somewhat less or lower quality training data). Unlike the “hacking” scenario, in the generic “hardware overhang” scenario, there’s not an apparent threshold effect that could cause a discontinuity.
(Curiously, Paul Christiano’s and AI Impacts’s posts arguing against discontinuous takeoff both ignore “hardware overhang” and neither give this counter-argument. Neither of them mention the “hacking” argument either, AFAICT.)
Wasn’t hardware overhang the argument that if AGI is more bottlenecked by software than hardware, then conceptual insights on the software side could cause a discontinuity as people suddenly figured out how to use that hardware effectively? I’m not sure how your counterargument really works there, since the AI that arrives “a bit earlier” either precedes or follows that conceptual breakthrough. If it precedes the breakthrough, then it doesn’t benefit from that conceptual insight so won’t be powerful enough to take advantage of the overhang, and if it follows it, then it has a discontinuous advantage over previous systems and can take advantage of hardware overhang.
---
Separately, your comment also feels related to my argument that focusing on just superintelligence is a useful simplifying assumption, since a superintelligence is almost by definition capable of taking over the world. But it simplifies things a little too much, because if we focus too much on just the superintelligence case, we might miss the emergence of a “dumb” AGI which nevertheless had the “crucial capabilities” necessary for a world takeover.
In those terms, “having sufficient offensive cybersecurity capability that a hacking attempt can snowball into a world takeover” would be one such crucial capability that allowed for a discontinuity.