Roko, preferences are not flat, they depend and act on the state of the world in general and on themselves in particular. They can grow very detailed, and include a state quite remote from current world as desirable. The problem with derailed aspect of transhumanism in not in remoteness from currently human, but in mistaken preferences arrived at mostly by blind leaps of imagination. We define the preferences over remote future implicitly, without being able to imagine it, only gradually becoming able to actually implement them, preserving or refining the preference through growth.
Roko, preferences are not flat, they depend and act on the state of the world in general and on themselves in particular. They can grow very detailed, and include a state quite remote from current world as desirable. The problem with derailed aspect of transhumanism in not in remoteness from currently human, but in mistaken preferences arrived at mostly by blind leaps of imagination. We define the preferences over remote future implicitly, without being able to imagine it, only gradually becoming able to actually implement them, preserving or refining the preference through growth.