I think a lot of people would be willing to die for their children’s sakes though. And if making unaligned ASI means risking their children’s lives for their own...? I think most parents would take the safer option. And most policymakers are parents.
I know you said this isn’t the best outcome, but this vision still seems like “humans in a zoo”, except the cage is gilded? I also worry that this vision relies overmuch on the God-Emperor never, say, getting tired of the rest of humanity existing, or something. What if he becomes convinced that the universe is better off by replacing all of us with clones of himself? As a source of inspiration, it feels like a tough pill to swallow, and only really looks better in comparison to x- or s-risk outcomes.
Thanks for engaging. You point out one huge problem and some other lesser ones.
This isn’t a very motivating scenario. “We’re ruled by someone barely nice enough to not kill us” does not sound like victory.
One approach is to emphasize that we can put better people in charge and they can hand power to an aligned ASI as soon as it’s safe and we’re ready. Then frame the “barely good enough person” as a backup plan to emphasize that full success on societal alignment isn’t needed.
Yes, some decision-makers would die for their children. I don’t think it’s that many; that’s a pretty big ask. If they would, that’s great. Advocating for a positive vision of success at alignment isn’t in opposition to advocating for a pause. I think that we should stop building AGI right now, and that we may squeak by on alignment even if we don’t, if we get our shit together rapidly in some critical but not large ways.
It’s like saying “you should definitely not try that motorcycle jump but if you do you should really try to land it by doing these things, because it would be awesome if you did!” You can go ahead and say how awesome it would be if you didn’t try the jump until you’d practiced and planned enough, too.
I think a lot of people would be willing to die for their children’s sakes though. And if making unaligned ASI means risking their children’s lives for their own...? I think most parents would take the safer option. And most policymakers are parents.
I know you said this isn’t the best outcome, but this vision still seems like “humans in a zoo”, except the cage is gilded? I also worry that this vision relies overmuch on the God-Emperor never, say, getting tired of the rest of humanity existing, or something. What if he becomes convinced that the universe is better off by replacing all of us with clones of himself? As a source of inspiration, it feels like a tough pill to swallow, and only really looks better in comparison to x- or s-risk outcomes.
Thanks for engaging. You point out one huge problem and some other lesser ones.
This isn’t a very motivating scenario. “We’re ruled by someone barely nice enough to not kill us” does not sound like victory.
One approach is to emphasize that we can put better people in charge and they can hand power to an aligned ASI as soon as it’s safe and we’re ready. Then frame the “barely good enough person” as a backup plan to emphasize that full success on societal alignment isn’t needed.
Yes, some decision-makers would die for their children. I don’t think it’s that many; that’s a pretty big ask. If they would, that’s great. Advocating for a positive vision of success at alignment isn’t in opposition to advocating for a pause. I think that we should stop building AGI right now, and that we may squeak by on alignment even if we don’t, if we get our shit together rapidly in some critical but not large ways.
It’s like saying “you should definitely not try that motorcycle jump but if you do you should really try to land it by doing these things, because it would be awesome if you did!” You can go ahead and say how awesome it would be if you didn’t try the jump until you’d practiced and planned enough, too.