Yes, and expecting it any one will try to narrow the group with which CEV-AI will align. For example we have a CEV-AI with following function: given presented with group X, he will calculate CEV(X). So if I want to manipulate it, I will present him as small group as possible, which will mostly include people like me.
But it is also possible to imagine superCEV, which will be able to calculate not only CEV, but also the best group for CEV.
For me CEV is not pleasant because it kills me as subject of history. I am not the one who rules the universe, but just a pet for which someone else knows what is good and bad. It kills all my potential for future development, which I see as natural evolution of my own values in complex environment.
I think also that there are infinitely many possible SEVs for a given group, which means that any CEV is mostly random.
The CEV-AI will have to create many simulations to calculate CEV. So most observers will still find themselves in CEV-modeling simulation where they fight another group of people with different values, and see their own values evolving someway. If the values of all people will not converge, the simulation will be terminated. I don’t see it as peasant scenario also.
My opinion is that most natural solution for aligning problem is to create AI as extension of his creator (using uploading and Tool AIs, or running AI-master algorithm on biological brain). In this case any evolution of goal system will by mine evolution, so there will be no problem of aligning something with something. There will be always one evolving thing. (Another question if it will be safe for others or stable).
Yes, and expecting it any one will try to narrow the group with which CEV-AI will align. For example we have a CEV-AI with following function: given presented with group X, he will calculate CEV(X). So if I want to manipulate it, I will present him as small group as possible, which will mostly include people like me.
But it is also possible to imagine superCEV, which will be able to calculate not only CEV, but also the best group for CEV.
For me CEV is not pleasant because it kills me as subject of history. I am not the one who rules the universe, but just a pet for which someone else knows what is good and bad. It kills all my potential for future development, which I see as natural evolution of my own values in complex environment.
I think also that there are infinitely many possible SEVs for a given group, which means that any CEV is mostly random.
The CEV-AI will have to create many simulations to calculate CEV. So most observers will still find themselves in CEV-modeling simulation where they fight another group of people with different values, and see their own values evolving someway. If the values of all people will not converge, the simulation will be terminated. I don’t see it as peasant scenario also.
My opinion is that most natural solution for aligning problem is to create AI as extension of his creator (using uploading and Tool AIs, or running AI-master algorithm on biological brain). In this case any evolution of goal system will by mine evolution, so there will be no problem of aligning something with something. There will be always one evolving thing. (Another question if it will be safe for others or stable).