I honestly think the EV of superhumans is lower than the EV for AI. sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else. i think people oftentimes act like the risk ends at existential risks, which i strongly disagree with. i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
with AIs we’re essentially putting them through selective pressures to promote benevolence (as a hedge by the labs in case they don’t figure out intent alignment). that seems like a massive advantage compared to the evolutionary baggage associated with humans.
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby. but people wouldn’t want their children to make them feel like shitty people so they would want them to “be more normal.”
sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else.
I don’t think the result of intelligence enhancement would be “multiplying those instincts” for the vast majority of people; humans don’t seem to end up more sadistic as they get smarter and have more options.
i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
I’m curious what value you assign to the ratio [U(paperclipped) - U(worst future)] / [U(best future) - U(paperclipped)]? It can’t be literally infinity unless U(paperclipped) = U(best future).
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby.
So your model is that we need to eradicate any last trace of sadism before superbabies is a good idea?
I honestly think the EV of superhumans is lower than the EV for AI. sadism and wills to power are baked into almost every human mind (with the exception of outliers of course). force multiplying those instincts is much worse than an AI which simply decides to repurpose the atoms in a human for something else. i think people oftentimes act like the risk ends at existential risks, which i strongly disagree with. i would argue that everyone dying is actually a pretty great ending compared to hyperexistential risks. it is effectively +inf relative utility.
with AIs we’re essentially putting them through selective pressures to promote benevolence (as a hedge by the labs in case they don’t figure out intent alignment). that seems like a massive advantage compared to the evolutionary baggage associated with humans.
with humans you’d need the will and capability to engineer in at least +5sd empathy and −10sd sadism into every superbaby. but people wouldn’t want their children to make them feel like shitty people so they would want them to “be more normal.”
I don’t think the result of intelligence enhancement would be “multiplying those instincts” for the vast majority of people; humans don’t seem to end up more sadistic as they get smarter and have more options.
I’m curious what value you assign to the ratio [U(paperclipped) - U(worst future)] / [U(best future) - U(paperclipped)]? It can’t be literally infinity unless U(paperclipped) = U(best future).
So your model is that we need to eradicate any last trace of sadism before superbabies is a good idea?