humanity survives if and only if that researcher is careful and takes safety seriously
Here’s where I’d stick in the 10^-3 penalty. It’s reasonable to assume that taking safety seriously will keep you safe from accidental leaks of toxic chemicals, deadly viruses, etc. because these are well-understood phenomena that pose a single, predictable risk. If you can keep the muriatic acid off your skin, it won’t burn you. If you can keep the swine flu out of your lungs, it won’t infect you.
A truly general AI, though, almost by definition, would be able to think up countless ways of overpowering you. It’s very unlikely that you could adequately guard against all of those ways, and the AI only needs to succeed once to cause an existential risk. Thus, it’s not enough that the AI be ‘securely’ boxed; the AI also has to be provably friendly. And that means we have to figure out what ‘friendly’ even means. And while there are certainly researchers out there who can be convinced to invest in a bit of safety, proving that an AI is friendly requires way more resources than just putting on a pair of gloves or working in an airtight room.
A truly general AI, though, almost by definition, would be able to think up countless ways of overpowering you. It’s very unlikely that you could adequately guard against all of those ways, and the AI only needs to succeed once to cause an existential risk.
It isn’t likely to be you vs the superintelligence, though. People keep imagining that—and then wringing their hands. The restraints on intelligent agents while they are being developed and tested are likely to consist of a prison built by the last generation of intelligent agents, featuring them as guards.
Here’s where I’d stick in the 10^-3 penalty. It’s reasonable to assume that taking safety seriously will keep you safe from accidental leaks of toxic chemicals, deadly viruses, etc. because these are well-understood phenomena that pose a single, predictable risk. If you can keep the muriatic acid off your skin, it won’t burn you. If you can keep the swine flu out of your lungs, it won’t infect you.
A truly general AI, though, almost by definition, would be able to think up countless ways of overpowering you. It’s very unlikely that you could adequately guard against all of those ways, and the AI only needs to succeed once to cause an existential risk. Thus, it’s not enough that the AI be ‘securely’ boxed; the AI also has to be provably friendly. And that means we have to figure out what ‘friendly’ even means. And while there are certainly researchers out there who can be convinced to invest in a bit of safety, proving that an AI is friendly requires way more resources than just putting on a pair of gloves or working in an airtight room.
It isn’t likely to be you vs the superintelligence, though. People keep imagining that—and then wringing their hands. The restraints on intelligent agents while they are being developed and tested are likely to consist of a prison built by the last generation of intelligent agents, featuring them as guards.