The question is, of course, silly. It is perfectly rational to decline to answer. I choose to try to answer.
It is also perfectly rational to say “it depends”. If you really think “a dust speck in 3^^^3 eyes” gives a uniquely defined probability distribution over different subsets of possibilityverse, you are being ridiculous. But let’s pretend it did—let’s pretend we had 3^^^^3 parallel Eleizers, standing on flat golden surfaces in 1G and one atmosphere, for just long enough to ask each other enough enough questions to define the problem properly. (I’m sorry, Eleizer, if by stating that possibility, I’ve increased the “true”ness of that part of the probabilityverse by ((3^^^3+1)/3^^^3) :) ).
You can or “I’ve thought about it, but I don’t trust my thought processes”. That is not my position.
My position is that this question does not, in fact, have an answer. I think that that fact is very important.
It’s not that the numbers are meaningless. 3^^^3 is a very exact number, and you can prove any number of things about it. A different question using ridiculous numbers—say, would you rather torture 4^^^4 people for 5 minutes or 3^^^3 of them for 50 years—has a single correct answer which is very clear (of course, the 3^^^3 ones; 4^^^4 >>> (3^^^3)^2). (Unless there were very bizarre extra conditions on the problem.)
It’s just that there is no universal moral utility function which inputs a probability distribution over a finite subset of the possibilityverse and outputs a number. It’s more like relativistic causality—substitute “better” for “after”. A is after B and B is a spacelike distance from C, but C can also be spacelike from A. The dust specks and the torture are incomparable, a spacelike distance.
I think that, philosophically, that makes a big difference. If you pilosophically can’t always go around morally comparing near-infinite sets, then it’s silly to try to approximate how you would behave if you could. Which means you consider the moral value of the consequences which you could possibly anticipate. So yeah, if you are working on AI, you are morally obligated to think about FAI, because that’s intentional action, and you would have to be a monster to say you didn’t care. But you don’t get to use FAI and the singularity to trump the here-and-now, because in many ways they’re just not comparable.
Which means, to me, for instance, that people can understand the singularity idea and believe it has a non-0 probability, and have abilities or resources that would be meaningful to the FAI effort, and still morally choose to simply live as “good people” in a more traditional sense (have a good life in which they make the people with whom they interact overall happier). It’s not just a lack of ability to trace the consequences; it’s also the possibility that the consequences of this or that outcome will be literally incomparable by any finite halting algorithm, whereas even our desperately-limited brains have decent approximations of algorithms for morally comparing the effect of, say, posting on OB versus washing the dishes.
The question is, of course, silly. It is perfectly rational to decline to answer. I choose to try to answer.
It is also perfectly rational to say “it depends”. If you really think “a dust speck in 3^^^3 eyes” gives a uniquely defined probability distribution over different subsets of possibilityverse, you are being ridiculous. But let’s pretend it did—let’s pretend we had 3^^^^3 parallel Eleizers, standing on flat golden surfaces in 1G and one atmosphere, for just long enough to ask each other enough enough questions to define the problem properly. (I’m sorry, Eleizer, if by stating that possibility, I’ve increased the “true”ness of that part of the probabilityverse by ((3^^^3+1)/3^^^3) :) ).
You can or “I’ve thought about it, but I don’t trust my thought processes”. That is not my position.
My position is that this question does not, in fact, have an answer. I think that that fact is very important.
It’s not that the numbers are meaningless. 3^^^3 is a very exact number, and you can prove any number of things about it. A different question using ridiculous numbers—say, would you rather torture 4^^^4 people for 5 minutes or 3^^^3 of them for 50 years—has a single correct answer which is very clear (of course, the 3^^^3 ones; 4^^^4 >>> (3^^^3)^2). (Unless there were very bizarre extra conditions on the problem.)
It’s just that there is no universal moral utility function which inputs a probability distribution over a finite subset of the possibilityverse and outputs a number. It’s more like relativistic causality—substitute “better” for “after”. A is after B and B is a spacelike distance from C, but C can also be spacelike from A. The dust specks and the torture are incomparable, a spacelike distance.
I think that, philosophically, that makes a big difference. If you pilosophically can’t always go around morally comparing near-infinite sets, then it’s silly to try to approximate how you would behave if you could. Which means you consider the moral value of the consequences which you could possibly anticipate. So yeah, if you are working on AI, you are morally obligated to think about FAI, because that’s intentional action, and you would have to be a monster to say you didn’t care. But you don’t get to use FAI and the singularity to trump the here-and-now, because in many ways they’re just not comparable.
Which means, to me, for instance, that people can understand the singularity idea and believe it has a non-0 probability, and have abilities or resources that would be meaningful to the FAI effort, and still morally choose to simply live as “good people” in a more traditional sense (have a good life in which they make the people with whom they interact overall happier). It’s not just a lack of ability to trace the consequences; it’s also the possibility that the consequences of this or that outcome will be literally incomparable by any finite halting algorithm, whereas even our desperately-limited brains have decent approximations of algorithms for morally comparing the effect of, say, posting on OB versus washing the dishes.
Going to wash the dishes now.