I don’t that’s a good description of the orthogonality thesis.
Probably not, but it highlights the relevant (or at least related) portion. I suppose I could have been more precise by specifying terminal values, since things like paperclips are obviously instrumental values, at least for us.
Human’s don’t effectively self modify
Agreed, except in the trivial case where we can condition ourselves to have different emotional responses. That’s substantially less dangerous, though.
Partial optimization for human values is easier than total optimization.
Why do you believe that?
I’m not sure I do, in the sense that I wouldn’t assign the proposition >50% probability. However, I might put the odds at around 25% for a Reduced Impact AI architecture providing a useful amount of shortcuts.
That seems like decent odds of significantly boosting expected utility. If such an AI would be faster to develop by even just a couple years, that could make the difference between winning and loosing an AI arms race. Sure, it’d be at the cost of a utopia, but if it boosted the odds of success enough it’d still have enough expected utility to compensate.
Probably not, but it highlights the relevant (or at least related) portion. I suppose I could have been more precise by specifying terminal values, since things like paperclips are obviously instrumental values, at least for us.
Agreed, except in the trivial case where we can condition ourselves to have different emotional responses. That’s substantially less dangerous, though.
I’m not sure I do, in the sense that I wouldn’t assign the proposition >50% probability. However, I might put the odds at around 25% for a Reduced Impact AI architecture providing a useful amount of shortcuts.
That seems like decent odds of significantly boosting expected utility. If such an AI would be faster to develop by even just a couple years, that could make the difference between winning and loosing an AI arms race. Sure, it’d be at the cost of a utopia, but if it boosted the odds of success enough it’d still have enough expected utility to compensate.