My understanding of habryka’s take is that it’s a bit more like:
The thing we want to steer the future is not current human values but an extrapolation of those values after enough reflection, and even if (current) AIs understand our current values fairly well, their extrapolation would probably diverge pretty substantially from ours, enough that most value gets lost.
I think there’s also a kernel that’s like:
A big part of what matters for humans is the process that generated our values (e.g. a messy evolutionary history) rather than the snapshot. Mind uploading might cut it; more brain-like AIs might cut it; intense RL on top of pretraining is really not great for this.
I’ve gleaned this sentiment from @habryka and @Kaarel, so they would be good candidates for explaining.
My understanding of habryka’s take is that it’s a bit more like:
I think there’s also a kernel that’s like:
Some pieces I think of as making similar points are Thou Art Godshatter and The Tails Coming Apart as a Metaphor for Life.
Sorry, but that is a bad explanation. It would’ve been better to provide no explanation than to provide a bad one.
I am a bit confused about in what way this is a bad explanation. It would be helpful for me if you could spell it out.