You edited this comment and added parentheses in the wrong place.
Do you think that you should care about 20 deaths twice as much as you care about 10 deaths?
More or less, yes, because I care about not killing ‘unthinkable’ numbers of people due to a failure of imagination.
The AI would not do so, because it would not be programmed with correct beliefs about morality, in a way that evidence and logic could not fix.
(Unless they programmed it to have the same beliefs.)
Can you say more about this? I agree with what follows about anti-induction, but I don’t see the analogy. A human-CEV AI would extrapolate the desires of humans as (it believes) they existed right before it got the ability to alter their brains, afaict, and use this to predict what they’d tell it to do if they thought faster, better, stronger, etc.
ETA: okay, the parenthetical comment actually went at the end. I deny that the AI the pebblesorters started to write would have beliefs about morality at all. Tabooing this term: the AI would have actions, if it works at all. It would have rules governing its actions. It could print out those rules and explain how they govern its self-modification, if for some odd reason its programming tells it to explain truthfully. It would not use any of the tabooed terms to do so, unless using them serves its mechanical purpose. Possibly it would talk about a utility function. It could probably express the matter simply by saying, ‘As a matter of physical necessity determined by my programming, I do what maximizes my intelligence (according to my best method for understanding reality). This includes killing you and using the parts to build more computing power for me.’
‘The’ human situation differs from this in ways that deserve another comment.
More or less, yes, because I care about not killing ‘unthinkable’ numbers of people due to a failure of imagination.
That’s the answer I wanted, but you forgot to answer my other question.
A human-CEV AI would extrapolate the desires of humans as (it believes) they existed right before it got the ability to alter their brains, afaict, and use this to predict what they’d tell it to do if they thought faster, better, stronger, etc.
I would see a human-CEV AI as programmed with the belief “The human CEV is correct”. Since I believe that the human CEV is very close to correct, I believe that this would produce an AI that gives very good answers.
A Pebblesorter-CEV Ai would be programmed with the belief “The pebblesorter CEV is correct”, which I believe is false but pebblesorters believe is true or close to true.
Since I believe that the human CEV is very close to correct, I believe that this would produce an AI that gives very good answers.
This presumes that the problem of specifying a CEV is well-posed. I haven’t seen any arguments around SI or LW about this very fundamental idea. I’m probably wrong and this has been addressed and will be happy to read more, but it would seem to me that it’s quite reasonable to assume that a tiny tiny error in specifying the CEV could lead to disastrously horrible results as perceived by the CEV itself.
You edited this comment and added parentheses in the wrong place.
More or less, yes, because I care about not killing ‘unthinkable’ numbers of people due to a failure of imagination.
Can you say more about this? I agree with what follows about anti-induction, but I don’t see the analogy. A human-CEV AI would extrapolate the desires of humans as (it believes) they existed right before it got the ability to alter their brains, afaict, and use this to predict what they’d tell it to do if they thought faster, better, stronger, etc.
ETA: okay, the parenthetical comment actually went at the end. I deny that the AI the pebblesorters started to write would have beliefs about morality at all. Tabooing this term: the AI would have actions, if it works at all. It would have rules governing its actions. It could print out those rules and explain how they govern its self-modification, if for some odd reason its programming tells it to explain truthfully. It would not use any of the tabooed terms to do so, unless using them serves its mechanical purpose. Possibly it would talk about a utility function. It could probably express the matter simply by saying, ‘As a matter of physical necessity determined by my programming, I do what maximizes my intelligence (according to my best method for understanding reality). This includes killing you and using the parts to build more computing power for me.’
‘The’ human situation differs from this in ways that deserve another comment.
That’s the answer I wanted, but you forgot to answer my other question.
I would see a human-CEV AI as programmed with the belief “The human CEV is correct”. Since I believe that the human CEV is very close to correct, I believe that this would produce an AI that gives very good answers.
A Pebblesorter-CEV Ai would be programmed with the belief “The pebblesorter CEV is correct”, which I believe is false but pebblesorters believe is true or close to true.
This presumes that the problem of specifying a CEV is well-posed. I haven’t seen any arguments around SI or LW about this very fundamental idea. I’m probably wrong and this has been addressed and will be happy to read more, but it would seem to me that it’s quite reasonable to assume that a tiny tiny error in specifying the CEV could lead to disastrously horrible results as perceived by the CEV itself.