One scenario I’ve thought of is the Hackening, i.e. in 3-12 months, someone releases an open source model equivalent to Mythos, different from previous models for having the Juice (autonomous end-to-end exploit construction), and the world is inadequately prepared for the mass of newly minted or empowered hackers. Many would be individuals that don’t care about causing international incidents if it gives short-term gain, and all of them can now probe the world’s entire attack surface. Even with efforts like Glasswing, there’s still likely to be a very long tail of targets that don’t have defense budgets or prepared SOCs. I haven’t seen anyone try to model this in detail, but what comes to my mind is that there’d be a gold rush of attacks while systems are still undefended, and enough ransomware/credential harvests/blackmails/lockouts could happen to trigger a recession. Attribution could be hard enough, and enough states tempted to join the chaos, that this causes heightened international tensions.
This seems like the sort of wake-up call that if nothing else of similar magnitude happened by then, it would be what causes AI policy to go mainstream. I don’t know what the default crisis response to that looks like, or what improvements on it could be. There might also be moderating factors, like control of compute, offense/defense asymmetries I didn’t think of, the economics of running attacks, or other things, that could reduce the severity of the scenario. It’s also possible that someone manages to make a harness that has the Juice even when a model in a simple agentic harness doesn’t.
One scenario I’ve thought of is the Hackening, i.e. in 3-12 months, someone releases an open source model equivalent to Mythos, different from previous models for having the Juice (autonomous end-to-end exploit construction), and the world is inadequately prepared for the mass of newly minted or empowered hackers. Many would be individuals that don’t care about causing international incidents if it gives short-term gain, and all of them can now probe the world’s entire attack surface. Even with efforts like Glasswing, there’s still likely to be a very long tail of targets that don’t have defense budgets or prepared SOCs. I haven’t seen anyone try to model this in detail, but what comes to my mind is that there’d be a gold rush of attacks while systems are still undefended, and enough ransomware/credential harvests/blackmails/lockouts could happen to trigger a recession. Attribution could be hard enough, and enough states tempted to join the chaos, that this causes heightened international tensions.
This seems like the sort of wake-up call that if nothing else of similar magnitude happened by then, it would be what causes AI policy to go mainstream. I don’t know what the default crisis response to that looks like, or what improvements on it could be. There might also be moderating factors, like control of compute, offense/defense asymmetries I didn’t think of, the economics of running attacks, or other things, that could reduce the severity of the scenario. It’s also possible that someone manages to make a harness that has the Juice even when a model in a simple agentic harness doesn’t.