1: We don’t know that AI can go FOOM. It may be just as hard to prevent self improving AI from wireheading (when it becomes super-intelligent) as it is to ensure friendliness. Note: perfect wireheading has infinite utility according to agent prone to wireheading; the length of wireheading experience in time (or it’s volume in space) is then irrelevant. The whole premise of fear of UFAI is that intelligence (human intelligence) can have faulty self improvement; it’s inconsistent to assume that about human intelligence but not about any AI.
2: We don’t know that the AI would likely to be substantially unfriendly. Other humans, and especially groups of humans (corporations, governments) are non-you non-friendly-to-you intelligences too, with historical examples of extreme unfriendliness (i’m going to coin a law that the (un)friendly intelligence discussion is incomplete without mention of nazis), yet they can be friendly enough—permitting you to live normal life while paying taxes (but note the military draft, which happens when meta-organism is threatened). It is plausible enough that the AI would be friendly enough. Humans would be cheap to store.
3: We may get there by mind uploading, which seems to me like the safest option.
4: We don’t actually know if FAI attempt is more, or less dangerous than messy AI like ‘replicate function of cortical columns, simulate a lot of cortical columns’. FAI attempt could just as well be more dangerous. You get it wrong, AI euthanizes you with the best intentions. The extrapolated volition idea btw entirely ignores fact that you are a massively parallel system that can have different goals in different parts of itself (and the mankind too is that kind of system, albeit less well connected).
The argumentation everywhere has very low external probabilities (when I evaluate probabilities if I see conflicting arguments that are opposite and both look similarly plausible, I assume external probability of zero, even if its 1 argument vs 10; much more so for 10 arguments vs 10), and so acting upon those arguments has rather low utility values.
Well, the critique I have:
1: We don’t know that AI can go FOOM. It may be just as hard to prevent self improving AI from wireheading (when it becomes super-intelligent) as it is to ensure friendliness. Note: perfect wireheading has infinite utility according to agent prone to wireheading; the length of wireheading experience in time (or it’s volume in space) is then irrelevant. The whole premise of fear of UFAI is that intelligence (human intelligence) can have faulty self improvement; it’s inconsistent to assume that about human intelligence but not about any AI.
2: We don’t know that the AI would likely to be substantially unfriendly. Other humans, and especially groups of humans (corporations, governments) are non-you non-friendly-to-you intelligences too, with historical examples of extreme unfriendliness (i’m going to coin a law that the (un)friendly intelligence discussion is incomplete without mention of nazis), yet they can be friendly enough—permitting you to live normal life while paying taxes (but note the military draft, which happens when meta-organism is threatened). It is plausible enough that the AI would be friendly enough. Humans would be cheap to store.
3: We may get there by mind uploading, which seems to me like the safest option.
4: We don’t actually know if FAI attempt is more, or less dangerous than messy AI like ‘replicate function of cortical columns, simulate a lot of cortical columns’. FAI attempt could just as well be more dangerous. You get it wrong, AI euthanizes you with the best intentions. The extrapolated volition idea btw entirely ignores fact that you are a massively parallel system that can have different goals in different parts of itself (and the mankind too is that kind of system, albeit less well connected).
The argumentation everywhere has very low external probabilities (when I evaluate probabilities if I see conflicting arguments that are opposite and both look similarly plausible, I assume external probability of zero, even if its 1 argument vs 10; much more so for 10 arguments vs 10), and so acting upon those arguments has rather low utility values.