There is no known, or readily conceivable, basis for rational agents to all converge on the same course of action.
Hm. I don’t think you quite mean that as stated?
I mean, I agree that a basis for rational agents to converge on values is difficult to imagine.
But it’s certainly possible for two agents with different values to converge on a course of action. E.g., “I want everything to be red, am OK with things being purple, and hate all other colors; you want everything to be blue, are OK with things being purple, and hate all other colors.” We have different values, but we can still agree on pragmatic grounds that we should paint everything purple.
Hence the “all”. Certainly agents can happen to have areas in which their goals are compatible, and choose to exert their efforts e.g. synergistically in such win-win situations of mutual benefit.
The same does not hold true for agents whose primary goals are strictly antagonistic. “I maximize the number of paperclips!”—“I minimize the number of paperclips!” will have … trouble … getting along, and mutually exchanging treatises about their respective ethics wouldn’t solve the impasse.
(A pair of “I make paperclips!”—“I destroy paperclips!” may actually enter a hugely beneficial relationship.)
Didn’t think there was anyone—apart from the PawnOfFaith and I—still listening in. :)
But of course this only works if the pair of agents both dislike war/murder even more than they like their colors, and/or if neither of them are powerful enough to murder the other one and thus paint everything their own colors.
Not neccesarily. An agent that values X and doesn’t have a stupid prior will invariably strive towards finding the best way to accomplish X. If X requires information about an ouside world, it will build epistemology and sensors, if it requires planning, it will build manipulators and a way of evaluating hypotheticals for X-ness.
All for want of X. It will be rational because it helps attaining X.
Omega offers you two boxes, each box contains a statement, upon choosing a box you will instantly belive that statement: One contains somthing true which you currently belive to be false, tailored to cause maximum disutility in your preferred ethical system; the other contains something false which you currently belive to be true, tailored to cause maximum utility.
Truth with negative consequences or Falsehood with positive ones? If you value nothing over truth you will realise something terrible upon opening the first box, that will maybe make you kill your family. If you value something other than truth, you will end up believing that the programming code you are writing will make pie, when it will in fact make a FAI.
Do you mean this as a general principle, along the lines of “If I am constructed so as to operate a particular way, it follows that I value operating that way”? Or as something specific about rationality?
If the former, I disagree, but if the latter I’m interested in what you have in mind.
Complete tangential point...
Hm. I don’t think you quite mean that as stated?
I mean, I agree that a basis for rational agents to converge on values is difficult to imagine.
But it’s certainly possible for two agents with different values to converge on a course of action. E.g., “I want everything to be red, am OK with things being purple, and hate all other colors; you want everything to be blue, are OK with things being purple, and hate all other colors.” We have different values, but we can still agree on pragmatic grounds that we should paint everything purple.
Hence the “all”. Certainly agents can happen to have areas in which their goals are compatible, and choose to exert their efforts e.g. synergistically in such win-win situations of mutual benefit.
The same does not hold true for agents whose primary goals are strictly antagonistic. “I maximize the number of paperclips!”—“I minimize the number of paperclips!” will have … trouble … getting along, and mutually exchanging treatises about their respective ethics wouldn’t solve the impasse.
(A pair of “I make paperclips!”—“I destroy paperclips!” may actually enter a hugely beneficial relationship.)
Didn’t think there was anyone—apart from the PawnOfFaith and I—still listening in. :)
Yup, that’s fair.
And I read the Recent Comments list every once in a while.
But of course this only works if the pair of agents both dislike war/murder even more than they like their colors, and/or if neither of them are powerful enough to murder the other one and thus paint everything their own colors.
Rational agents all need to value rationality.
Not neccesarily. An agent that values X and doesn’t have a stupid prior will invariably strive towards finding the best way to accomplish X. If X requires information about an ouside world, it will build epistemology and sensors, if it requires planning, it will build manipulators and a way of evaluating hypotheticals for X-ness.
All for want of X. It will be rational because it helps attaining X.
Good epistemological rationality requires avoidance of bias, contradiction, arbitrariness, etc. That is just what my rationality-based ethics needs.
I will defer to the problem of
Truth with negative consequences or Falsehood with positive ones? If you value nothing over truth you will realise something terrible upon opening the first box, that will maybe make you kill your family. If you value something other than truth, you will end up believing that the programming code you are writing will make pie, when it will in fact make a FAI.
Do you mean this as a general principle, along the lines of “If I am constructed so as to operate a particular way, it follows that I value operating that way”? Or as something specific about rationality?
If the former, I disagree, but if the latter I’m interested in what you have in mind.
Only instrumentally.
Epistemic rationality has instrumental value. That’s where the trouble starts.