I feel like people haven’t fully internalized what the world would look like if computer security actually broke.
There are some varying opinions on this, but a window of time without real computer security seems plausible. I was recently speaking with a computer security professor who I deeply respect and he was literally like “I think we are fucked and I don’t think there is anything we can do.”
This is similar to how people believe there is a 20% chance of extinction via AI but don’t really internalize “No really. You will die and your girlfriend too. And your dog. And …” In the cybersecurity case, some people believe (me included) that you cannot just patch all the bugs before releasing the model[1] but then don’t internalize “No really. It would be chaos. You may not be able to get into your bank account. Industrial plants could be compromised. Power could go out for several days at a time. [...]”
The current plan seems to be to let companies use the models to fix all the bugs a model can find before a public release of that model. But there are so many companies that would need to do this properly for this to work, and the more companies you release to, the more opportunities there are for some bad actor to get their hands on the model. There are also many systems running legacy code that is very hard to update. In some cases you may need to go to a physical location to properly update a machine. Even if it is in theory possible to fix all the bugs, it would be pretty hard and I expect humanity to drop the ball for a while.
I strongly agree, and I’ve been working on a top level post trying to paint a picture of what this would look like. I’ve been calling it “the Hackening” to people I talk to. Do you have ideas for what would be good to put in the post?
This is a great idea for a post! I wanted to do something somewhat similar but probably will never get done (or even started).
To me, the main interesting thing to think about here is how much of the global financial system is vulnerable to cyber attacks. Obviously you could cause a lot of damage but hacking banks and locking people out of their accounts and wiping servers containing important data. But would the entire economy stop working? Like maybe people wouldn’t be able to get paid, supply chains would break down etc.? What does this do to international relations / does the public start raiding stores and steeling stuff? Or maybe the damage within the financial sector is pretty localized and the economy is able to continue functioning even though markets are doing crazy things. But I think that doing a good job thinking through this question would be a great contribution.
Another thought: I would love to see a AI 2027 / Plan A pair of pieces describing what may happen by default vs what would happen in optimistic but realistic world where things go well. Having lots of detail about where the vulnerabilities are and what would be required to prepare the world would be very valuable. (I may try to help get some people to work on this so people can DM if they are interested.)
I know nothing of cybersecurity, but I’ve been wondering exactly the things Zephaniah talks about. Like “wait, won’t there be a total digital apocalypse?” So I’m waiting for your post, feel free to DM me when it’s out (or if you want to share a draft).
I’d recommend just addressing the most radical and outlandish scenarios. To start at the upper bound of harm and go downwards.
It won’t be possible to go on the internet at all, unless you have the most modern systems? Databases get hacked and tons of old info disappears from the internet?Hospitals get hacked and people die? Entire countries get destroyed? CIA/KGB start hunting hackers extra hard? Countries form pacts to hunt hackers, because it’s the only way to avoid societal collapse?
It’s interesting that public reaction was so different to this, as opposed to Y2K. People seem to either have more faith in the current tech ecosystem than the one in 1999 (which seems unfounded) or sufficient skepticism about AI Safety claims that they’re willing to dismiss it out of the gate.
I think this is a good question and I don’t have especially strong feelings on this but I don’t think any attempt here would work. It’s just really hard to direct chaos in the direction you want.
A tangentially related and perhaps interesting intuition I have:
I have communist friends who think that if the world got bad enough (the particular reason doesn’t matter) this could actually be good because there would finally be a good enough reason to revolt and communism would win. But they are assuming the public would use the chaos as an opportunity to do communism when it seems just as likely that they would do authoritarianism or direct the anger towards an ethnic or religious minority, etc. The chaos/anger/suffering more likely then not won’t be directed in the exact direction they want. In general if you are advocating for a specific law or ideal, chaos is probably really bad news and I think its better to try to minimize the chaos then try to harness it in the direction you are hoping for.
I feel like people haven’t fully internalized what the world would look like if computer security actually broke.
There are some varying opinions on this, but a window of time without real computer security seems plausible. I was recently speaking with a computer security professor who I deeply respect and he was literally like “I think we are fucked and I don’t think there is anything we can do.”
This is similar to how people believe there is a 20% chance of extinction via AI but don’t really internalize “No really. You will die and your girlfriend too. And your dog. And …” In the cybersecurity case, some people believe (me included) that you cannot just patch all the bugs before releasing the model[1] but then don’t internalize “No really. It would be chaos. You may not be able to get into your bank account. Industrial plants could be compromised. Power could go out for several days at a time. [...]”
The current plan seems to be to let companies use the models to fix all the bugs a model can find before a public release of that model. But there are so many companies that would need to do this properly for this to work, and the more companies you release to, the more opportunities there are for some bad actor to get their hands on the model. There are also many systems running legacy code that is very hard to update. In some cases you may need to go to a physical location to properly update a machine. Even if it is in theory possible to fix all the bugs, it would be pretty hard and I expect humanity to drop the ball for a while.
I strongly agree, and I’ve been working on a top level post trying to paint a picture of what this would look like. I’ve been calling it “the Hackening” to people I talk to. Do you have ideas for what would be good to put in the post?
This is a great idea for a post! I wanted to do something somewhat similar but probably will never get done (or even started).
To me, the main interesting thing to think about here is how much of the global financial system is vulnerable to cyber attacks. Obviously you could cause a lot of damage but hacking banks and locking people out of their accounts and wiping servers containing important data. But would the entire economy stop working? Like maybe people wouldn’t be able to get paid, supply chains would break down etc.? What does this do to international relations / does the public start raiding stores and steeling stuff? Or maybe the damage within the financial sector is pretty localized and the economy is able to continue functioning even though markets are doing crazy things. But I think that doing a good job thinking through this question would be a great contribution.
Another thought: I would love to see a AI 2027 / Plan A pair of pieces describing what may happen by default vs what would happen in optimistic but realistic world where things go well. Having lots of detail about where the vulnerabilities are and what would be required to prepare the world would be very valuable. (I may try to help get some people to work on this so people can DM if they are interested.)
I know nothing of cybersecurity, but I’ve been wondering exactly the things Zephaniah talks about. Like “wait, won’t there be a total digital apocalypse?” So I’m waiting for your post, feel free to DM me when it’s out (or if you want to share a draft).
I’d recommend just addressing the most radical and outlandish scenarios. To start at the upper bound of harm and go downwards.
It won’t be possible to go on the internet at all, unless you have the most modern systems? Databases get hacked and tons of old info disappears from the internet?Hospitals get hacked and people die? Entire countries get destroyed? CIA/KGB start hunting hackers extra hard? Countries form pacts to hunt hackers, because it’s the only way to avoid societal collapse?
Thank you, will do!
It’s interesting that public reaction was so different to this, as opposed to Y2K. People seem to either have more faith in the current tech ecosystem than the one in 1999 (which seems unfounded) or sufficient skepticism about AI Safety claims that they’re willing to dismiss it out of the gate.
Thoughts on leveraging this rather plausible scenario to stimulate governmental action towards doing something productive?
I think this is a good question and I don’t have especially strong feelings on this but I don’t think any attempt here would work. It’s just really hard to direct chaos in the direction you want.
A tangentially related and perhaps interesting intuition I have:
I have communist friends who think that if the world got bad enough (the particular reason doesn’t matter) this could actually be good because there would finally be a good enough reason to revolt and communism would win. But they are assuming the public would use the chaos as an opportunity to do communism when it seems just as likely that they would do authoritarianism or direct the anger towards an ethnic or religious minority, etc. The chaos/anger/suffering more likely then not won’t be directed in the exact direction they want. In general if you are advocating for a specific law or ideal, chaos is probably really bad news and I think its better to try to minimize the chaos then try to harness it in the direction you are hoping for.
It’s not only communists, accelerationism is a strategy applicable more generally to any sufficiently anti-status-quo ideology.
Yes I agree. This is a very interesting pattern. I would love a satisfying explanation for why so many people seem to think in this way.
Been saying this for years.