I love the way you explained the iterative design failure modes! Accessible and clear. Well, the lead gasoline inventors knew what they were doing was harmful to humans, but they were driven by profits. But there are plenty of examples where the unexpected long-term effects kill you, like the CFC ozone depletion.
My main takeaway is similar to what I had gestured at some years ago: you can’t use an AGI as a tool. Instead an AGI’s goal would be to understand the universe, including humans in it, and only act well within the boundaries of its understanding. So, no little fusicles in every phone unless it cannot be exploited by malevolent humans, without anyone thinking to check. And refuse to do anything unexpectedly x-risky humans may ask or try:
That would imply a frustrating experience where an AGI would refuse a seemingly perfectly reasonable request for the reasons unfathomable to us. May not quite be the “CEV”, but at least the chances of survival will go up.
I love the way you explained the iterative design failure modes! Accessible and clear. Well, the lead gasoline inventors knew what they were doing was harmful to humans, but they were driven by profits. But there are plenty of examples where the unexpected long-term effects kill you, like the CFC ozone depletion.
My main takeaway is similar to what I had gestured at some years ago: you can’t use an AGI as a tool. Instead an AGI’s goal would be to understand the universe, including humans in it, and only act well within the boundaries of its understanding. So, no little fusicles in every phone unless it cannot be exploited by malevolent humans, without anyone thinking to check. And refuse to do anything unexpectedly x-risky humans may ask or try:
Or even https://archive.is/30TA6.
That would imply a frustrating experience where an AGI would refuse a seemingly perfectly reasonable request for the reasons unfathomable to us. May not quite be the “CEV”, but at least the chances of survival will go up.