I will go further, and say the human universals are nowhere near strong enough to assume that alignment of much more powerful people will automatically/likely happen, or that not aligning them produces benevolent results, and the reason for this is humans are already misaligned, in many cases very severely to each other, so allowing human augmentation without institutional reform makes things a lot worse by default.
It is better to solve the AI alignment problem first, then have a legal structure created by AIs that can make human genetic editing safe, rather than try to solve the human alignment problem:
I honestly think the EV of superhumans is lower than the EV for AI. sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else. i think people oftentimes act like the risk ends at existential risks, which i strongly disagree with. i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
with AIs we’re essentially putting them through selective pressures to promote benevolence (as a hedge by the labs in case they don’t figure out intent alignment). that seems like a massive advantage compared to the evolutionary baggage associated with humans.
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby. but people wouldn’t want their children to make them feel like shitty people so they would want them to “be more normal.”
sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else.
I don’t think the result of intelligence enhancement would be “multiplying those instincts” for the vast majority of people; humans don’t seem to end up more sadistic as they get smarter and have more options.
i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
I’m curious what value you assign to the ratio [U(paperclipped) - U(worst future)] / [U(best future) - U(paperclipped)]? It can’t be literally infinity unless U(paperclipped) = U(best future).
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby.
So your model is that we need to eradicate any last trace of sadism before superbabies is a good idea?
I will go further, and say the human universals are nowhere near strong enough to assume that alignment of much more powerful people will automatically/likely happen, or that not aligning them produces benevolent results, and the reason for this is humans are already misaligned, in many cases very severely to each other, so allowing human augmentation without institutional reform makes things a lot worse by default.
It is better to solve the AI alignment problem first, then have a legal structure created by AIs that can make human genetic editing safe, rather than try to solve the human alignment problem:
https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies#jgDtAPXwSucQhPBwf
I honestly think the EV of superhumans is lower than the EV for AI. sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else. i think people oftentimes act like the risk ends at existential risks, which i strongly disagree with. i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
with AIs we’re essentially putting them through selective pressures to promote benevolence (as a hedge by the labs in case they don’t figure out intent alignment). that seems like a massive advantage compared to the evolutionary baggage associated with humans.
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby. but people wouldn’t want their children to make them feel like shitty people so they would want them to “be more normal.”
I don’t think the result of intelligence enhancement would be “multiplying those instincts” for the vast majority of people; humans don’t seem to end up more sadistic as they get smarter and have more options.
I’m curious what value you assign to the ratio [U(paperclipped) - U(worst future)] / [U(best future) - U(paperclipped)]? It can’t be literally infinity unless U(paperclipped) = U(best future).
So your model is that we need to eradicate any last trace of sadism before superbabies is a good idea?