Posting here since the other post is now at exactly 50 replies:
Re michael vassar:
Sane utility functions pay attention to base rates, not just evidence, so even if it’s impossible to measure a difference in principle one can still act according to a probability distribution over differences.
You’re right, in principle. But how would you estimate a base rate in the absence of all empirical data? By simply using your priors?
I pretty much completely agree with the rest of your paragraph.
Re Nick Tarleton:
(1) an entity without E can have identical outward behavior to an entity with E (but possibly different physical structure); and
(2) you assign intrinsic value to at least some entities with E, but none without it?
If so, do you have property E?
As phrased, this is too vague to answer; for one thing, “identical outward behaviour” under what circumstances? Presumably not all conceivable ones (“What if you take it apart atom by atom using MNT?”), otherwise it couldn’t have a different physical structure.
If you rephrased it to be precise, I strongly suspect that I would genuinely not know the answer without a lot of further research; in fact, without that research, I couldn’t even be sure that there is any E for which both of your premises hold. I’m a human, and I don’t really know how my value system works in edge cases. Estimating the intrinsic value of general information-processing devices with a given behaviour is pretty far removed from the cases it was originally optimized to judge.
Posting here since the other post is now at exactly 50 replies: Re michael vassar: Sane utility functions pay attention to base rates, not just evidence, so even if it’s impossible to measure a difference in principle one can still act according to a probability distribution over differences. You’re right, in principle. But how would you estimate a base rate in the absence of all empirical data? By simply using your priors? I pretty much completely agree with the rest of your paragraph.
Re Nick Tarleton: (1) an entity without E can have identical outward behavior to an entity with E (but possibly different physical structure); and (2) you assign intrinsic value to at least some entities with E, but none without it? If so, do you have property E? As phrased, this is too vague to answer; for one thing, “identical outward behaviour” under what circumstances? Presumably not all conceivable ones (“What if you take it apart atom by atom using MNT?”), otherwise it couldn’t have a different physical structure. If you rephrased it to be precise, I strongly suspect that I would genuinely not know the answer without a lot of further research; in fact, without that research, I couldn’t even be sure that there is any E for which both of your premises hold. I’m a human, and I don’t really know how my value system works in edge cases. Estimating the intrinsic value of general information-processing devices with a given behaviour is pretty far removed from the cases it was originally optimized to judge.