To me it looks like the universe made of computronium and devoid of living humans. With the only difference with the unaligned Foom being that some of that computronium calculates our digital imitations.
EDIT: I don’t claim that “me is meat me” view is objectively right. It’s just according to my purely subjective values people are biological people and me is a biological me. Digital being can be our children and successors, but I don’t identify myself with them.
You may view digital you as your true self. I respect that. But I really don’t want an AI that forces your values on me (or my on yours). Or AI that makes people compete with AIs for the right to be alive, because it’s obvious that we have no chance in that competition. If we have AI that maximizes intelligence, is it really that different from “papperclip optimizer” that “can find a better use for your atoms”?
To me it looks like the universe made of computronium and devoid of living humans. With the only difference with the unaligned Foom being that some of that computronium calculates our digital imitations.
EDIT: I don’t claim that “me is meat me” view is objectively right. It’s just according to my purely subjective values people are biological people and me is a biological me. Digital being can be our children and successors, but I don’t identify myself with them.
You may view digital you as your true self. I respect that. But I really don’t want an AI that forces your values on me (or my on yours). Or AI that makes people compete with AIs for the right to be alive, because it’s obvious that we have no chance in that competition. If we have AI that maximizes intelligence, is it really that different from “papperclip optimizer” that “can find a better use for your atoms”?