Yeah I agree, but one of the points of this post was meant to be take as assumption “good people at Anthropic or whatever do a good job building an actually-more-moral-than-average-for-most-practical purposes human level intelligence” (i.e. via the mechanisms like “the weak superintelligence can just decide to self-modify into the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.”)
(and, like, I do buy the arguments that if we’re assuming the first bunch of IMO optimistic assumptions about getting to humanish-level-alignment being easy, it’s actually not that hard to do that step)
But then, argue: yeah, even if we assume that one, it’s really not great.
“the weak superintelligence can just decide to self-modify into the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.”
I don’t think this will work. Today’s billionaires can already do something similar to a binding self-modification: donate most of their money to good causes. Not just take a non-binding “pledge”, but actually donate. Few do that. Most of them spend more on increasing their own power than on any kind of charity. For the same reason, I expect future human-like AIs to shy away from moral self-modification. They’ll keep issuing press releases saying “I’ll do it tomorrow” and so on. And as for corporation-like AIs—we can just forget it.
If we want a future where AIs are more moral than people, we need to build them that way from the start.
I mean I don’t believe most of the leadup-assumptions to this world in the first place.
But, AI would be way more competent at self-modification than billionaires. I think hopes that routes through: “build a aligned-ish medium strength AI, in a corrigibility basin”, then “ask it to help you make it more aligned” is the the sort of thing that might work if you actually succeed at the first step.
Just, you have to then progress quickly to fully aligned wise powerful longterm safeguards, which implies a degree of ‘fully solve alignment’ that it didn’t seem like everyone was properly imagining as necesssary.”
Yeah I agree, but one of the points of this post was meant to be take as assumption “good people at Anthropic or whatever do a good job building an actually-more-moral-than-average-for-most-practical purposes human level intelligence” (i.e. via the mechanisms like “the weak superintelligence can just decide to self-modify into the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.”)
(and, like, I do buy the arguments that if we’re assuming the first bunch of IMO optimistic assumptions about getting to humanish-level-alignment being easy, it’s actually not that hard to do that step)
But then, argue: yeah, even if we assume that one, it’s really not great.
I don’t think this will work. Today’s billionaires can already do something similar to a binding self-modification: donate most of their money to good causes. Not just take a non-binding “pledge”, but actually donate. Few do that. Most of them spend more on increasing their own power than on any kind of charity. For the same reason, I expect future human-like AIs to shy away from moral self-modification. They’ll keep issuing press releases saying “I’ll do it tomorrow” and so on. And as for corporation-like AIs—we can just forget it.
If we want a future where AIs are more moral than people, we need to build them that way from the start.
I mean I don’t believe most of the leadup-assumptions to this world in the first place.
But, AI would be way more competent at self-modification than billionaires. I think hopes that routes through: “build a aligned-ish medium strength AI, in a corrigibility basin”, then “ask it to help you make it more aligned” is the the sort of thing that might work if you actually succeed at the first step.
Just, you have to then progress quickly to fully aligned wise powerful longterm safeguards, which implies a degree of ‘fully solve alignment’ that it didn’t seem like everyone was properly imagining as necesssary.”