I don’t understand what you mean by a revealed preference. If you mean “that which is rewarded,” then it seems pretty straightforward to me that a reinforcement learner can’t optimize anything other than that which is rewarded (in the limit).
That’s why I said the “right” thing to do if you asked about cryonics “I will give you something to deny. I’ll create a perfect reality and you will be cured afterward.”
I don’t understand what you mean by a revealed preference. If you mean “that which is rewarded,” then it seems pretty straightforward to me that a reinforcement learner can’t optimize anything other than that which is rewarded (in the limit).
Yes, that’s basically what I mean. I think I’m trying to refer to the same issue that Paul mentioned here: https://www.lesswrong.com/posts/pZhDWxDmwzuSwLjou/asymptotically-benign-agi#ZWtTvMdL8zS9kLpfu
That’s why I said the “right” thing to do if you asked about cryonics “I will give you something to deny. I’ll create a perfect reality and you will be cured afterward.”