Nothing. The arguments towards any course of action have very low external probabilities (which I assign when I see equally plausible but contradicting arguments), resulting in very low expected utilities even if the bad AI is presumed to do some drastically evil stuff vs good AI doing some drastically nice stuff. There are many problems for which efforts have larger expected payoff.
edit:
I do subscribe to the school of thought that the irregular connectionist AIs (neural networks, brain emulations of various kind and the like) are the ones least likely to engage in highly structured effort like maximization of some scary utility to the destruction of everything else. I’m very dubious that the agent can have foresight so good as to decide humans are not worth preserving, as part of general “gather more interesting information” heuristics.
While the design space near the FAI is a minefield of monster AIs and a bugged FAI represents a worst case scenario. There is a draft of my article on the topic. Note: I am a software developer, and I am very sceptical about our ability to write FAI that is not bugged, as well as of ability to detect substantial problems in FAI goal system, as regardless of the goal system the FAI will do all it can to pretend to be working correctly.
Nothing. The arguments towards any course of action have very low external probabilities (which I assign when I see equally plausible but contradicting arguments), resulting in very low expected utilities even if the bad AI is presumed to do some drastically evil stuff vs good AI doing some drastically nice stuff. There are many problems for which efforts have larger expected payoff.
edit:
I do subscribe to the school of thought that the irregular connectionist AIs (neural networks, brain emulations of various kind and the like) are the ones least likely to engage in highly structured effort like maximization of some scary utility to the destruction of everything else. I’m very dubious that the agent can have foresight so good as to decide humans are not worth preserving, as part of general “gather more interesting information” heuristics.
While the design space near the FAI is a minefield of monster AIs and a bugged FAI represents a worst case scenario. There is a draft of my article on the topic. Note: I am a software developer, and I am very sceptical about our ability to write FAI that is not bugged, as well as of ability to detect substantial problems in FAI goal system, as regardless of the goal system the FAI will do all it can to pretend to be working correctly.
I can’t see this draft. I think only those who write them can see drafts.
Hmm, weird. I thought the hide button would hide it from public, and un-hide button would unhide. How do i make it public as a draft?
Just post it to Discussion and immediately use “Delete”. It’ll still be readable and linkable, but not seen in the index.
Hmm, can you see it now? (I of course kept a copy of the text on my computer, in case you were joking, so i do have the draft reposted as well)
It is now readable at the previous link, yes.