I agree with your first statement: it is enjoyable to read fiction that can convincingly propose a difficult idea (here, that my fundamental assumptions are false), but the point behind (maybe about what is matter?) is too obfuscated to be applicable in another context, and so I do not grow as a rationalist, only as a sci-fi reader.
For your pro-paperclip argument, the probability is wrong: it assumes that people’s beliefs are independent, which is a dubious model. Perhaps a way to salvage your reasoning would be to consider the amount of classes of people who have the same opinion on paperclipping the universe for the same reason. I’d expect two such classes to have somewhat independent beliefs (as in, an argument that convinces one would not necessarily convince another). There aren’t a million ways to justify or refuse paperclipping the universe, though.
Using the honoured tradition of pulling numbers out of my arse, I’ll assume there are a thousand such classes. It does not seem that extreme to assign a probability less than e-4 that a random class would embrace paperclipping. It’s not outright out of my range of intuitively considerable probabilities the way e-10 is.
But regardless, Z. M. Davis’s argument does not hinge on unanimous agreement, it’s easy to see how even partial agreement can still be reworked into an argument Z. M. Davis would probably agree with: ”A certain amount of people agree that we should not paperclip the universe (for reasons unknown or too complicated to understand or that I just don’t want to disclose(*)); in the absence of information about the preferences of the rest of the people and in the absence of an object-level reason to paperclip the universe or not, that is evidence we should not paperclip the universe (exactly how much evidence depends on how confident you are about your own ability to discern the answer to this question relative to a random person’s ability, and how you adjust the information just given to take biases into account). Thanks to conservation of expected evidence, this means that even in the case where we are aware of other arguments, this one still holds some weight, if a reduced one.”
I agree with your first statement: it is enjoyable to read fiction that can convincingly propose a difficult idea (here, that my fundamental assumptions are false), but the point behind (maybe about what is matter?) is too obfuscated to be applicable in another context, and so I do not grow as a rationalist, only as a sci-fi reader.
For your pro-paperclip argument, the probability is wrong: it assumes that people’s beliefs are independent, which is a dubious model.
Perhaps a way to salvage your reasoning would be to consider the amount of classes of people who have the same opinion on paperclipping the universe for the same reason. I’d expect two such classes to have somewhat independent beliefs (as in, an argument that convinces one would not necessarily convince another). There aren’t a million ways to justify or refuse paperclipping the universe, though.
Using the honoured tradition of pulling numbers out of my arse, I’ll assume there are a thousand such classes. It does not seem that extreme to assign a probability less than e-4 that a random class would embrace paperclipping. It’s not outright out of my range of intuitively considerable probabilities the way e-10 is.
But regardless, Z. M. Davis’s argument does not hinge on unanimous agreement, it’s easy to see how even partial agreement can still be reworked into an argument Z. M. Davis would probably agree with:
”A certain amount of people agree that we should not paperclip the universe (for reasons unknown or too complicated to understand or that I just don’t want to disclose(*)); in the absence of information about the preferences of the rest of the people and in the absence of an object-level reason to paperclip the universe or not, that is evidence we should not paperclip the universe (exactly how much evidence depends on how confident you are about your own ability to discern the answer to this question relative to a random person’s ability, and how you adjust the information just given to take biases into account).
Thanks to conservation of expected evidence, this means that even in the case where we are aware of other arguments, this one still holds some weight, if a reduced one.”
Remember you have to face not the argument given by Z. M. Davis, but the best one that he could have come up with if you are to show his reasoning is not sound.