An ASI aligned to a group of people likely should dedicate sovereign slivers of compute (optimization domains) for each of those people, and those people could do well with managing their domain with their own ASIs aligned to each of them separately. Optimization doesn’t imply a uniform pureed soup, it’s also possible to optimize autonomy, coordination, and interaction, without mixing them up.
An ASI perfectly aligned to me must literally be a smarter version of myself.
Values judge what should be done, but also what you personally should be doing. An ASI value aligned to you will be doing the things that should be done (according to you, on reflection), but you wouldn’t necessarily endorse that you personally should be doing those things. Like, I want the world to be saved, but I don’t necessarily want to be in a position to need to try to save the world personally.
So an ASI perfectly aligned to you might help uplift you into a smarter version of yourself as one of its top priorities, and then go on to do various other things you’d approve of on reflection. But you wouldn’t necessarily endorse that it’s the smarter version of yourself that is doing those other things, you are merely endorsing that they get done.
An ASI aligned to a group of people likely should dedicate sovereign slivers of compute (optimization domains) for each of those people, and those people could do well with managing their domain with their own ASIs aligned to each of them separately. Optimization doesn’t imply a uniform pureed soup, it’s also possible to optimize autonomy, coordination, and interaction, without mixing them up.
Values judge what should be done, but also what you personally should be doing. An ASI value aligned to you will be doing the things that should be done (according to you, on reflection), but you wouldn’t necessarily endorse that you personally should be doing those things. Like, I want the world to be saved, but I don’t necessarily want to be in a position to need to try to save the world personally.
So an ASI perfectly aligned to you might help uplift you into a smarter version of yourself as one of its top priorities, and then go on to do various other things you’d approve of on reflection. But you wouldn’t necessarily endorse that it’s the smarter version of yourself that is doing those other things, you are merely endorsing that they get done.