“We don’t need the t.
It’s redundant.
The r never repeats itself.”
While this seems to be true given the expansion of the universe, is it strictly necessary? What if some value R does repeat, throwing the universe into an endless loop? At some point, the chains of r’s leading up to R0 and R1 would differ; wouldn’t we need another variable to encode that?
″...if you would prefer not to become orgasmium, then why should you?”
I’d prefer not to become orgasmium, because I value my consciousness and humanity, my ability to think, decide, and interact. However, it’s unclear to me what exactly preference is, other than the traversal of pathways we’ve constructed, whether we’re aware of them or not, leading to pleasure, or away from pain. To drastically oversimplify, those values exist in me as a result of beliefs I’ve constructed, linking the lack of those things to an identity that I don’t want, which in turn is eventually linked to an emotional state of sadness and loss that I’d like to avoid. There’s also probably a link to a certain identity that I do want, which leads to a certain sense of pride and rightness, which leads me to a positive emotional state.
Eliezer, you said there was nothing higher to override your preference to increase intelligence gradually. But what about the preferences that led you to that one? What was it about the gradual increase of intelligence, and your beliefs about what that means, that compelled you to prefer it? Isn’t that deeper motivation closer to your actual volition? How far down can we chase this? What is the terminal value of fun, if not orgasmium?
Or is “fun” in this context the pursuit specifically of those preferences that we’re consciously aware of as goals?