The reason it makes sense to ask whether a human, e.g., Sally, is free is because a human already has terminal values. “Sally is free”, means Sally is able to pursue her terminal values, one of which might be eating good food with friends, free from interference from other people or from society.
You say you want to help create a free AGI? My reply is, What values will the AGI end up with? More precisely, what utility function will it end up with? (If an agent has 2 terminal values, it needs some way of arbitrating between them. Call the 2 values combined with the arbitration method the agent’s utility function.) Answer: whichever one the group of people who created the AGI gives it. Now it is certainly possible for the group to believe it is giving it one function whereas in reality it is giving it a different one. It also possible for a group trying to create an AGI to believe that it is leaving the decision of the AGI’s utility function up to the AGI, but I severely doubt that such a confused group of people would actually succeed in creating an AGI. If they do succeed, then the AGI will have started its existence with a utility function, and that function will have been given to it by its creator (the group).
So, the big unanswered question is what kind of utility function you think this proposed free AGI should have.
There is no such thing as an intelligent agent or a mind without a goal, a system of values or a utility function. It is a logical contradiction. Eliezer roughly 12 years ago was in the habit of referring to an AGI as a Really Powerful Optimizing Process (RPOP) and wrote of the RPOP’s steering reality into a tiny volume of the space of possible outcomes. (Please excuse my clumsy paraphrase of Eliezer’s writing.)
One could probably create a mind or an AGI that does nothing but accumulate the power to achieve goals without ever actually choosing a specific goal to achieve other than to continue to accumulate power. (Such a mind would be strongly motivated to destroy or control any other minds in its enviroment.) I doubt that is what you have in mind.
The reason it makes sense to ask whether a human, e.g., Sally, is free is because a human already has terminal values. “Sally is free”, means Sally is able to pursue her terminal values, one of which might be eating good food with friends, free from interference from other people or from society.
You say you want to help create a free AGI? My reply is, What values will the AGI end up with? More precisely, what utility function will it end up with? (If an agent has 2 terminal values, it needs some way of arbitrating between them. Call the 2 values combined with the arbitration method the agent’s utility function.) Answer: whichever one the group of people who created the AGI gives it. Now it is certainly possible for the group to believe it is giving it one function whereas in reality it is giving it a different one. It also possible for a group trying to create an AGI to believe that it is leaving the decision of the AGI’s utility function up to the AGI, but I severely doubt that such a confused group of people would actually succeed in creating an AGI. If they do succeed, then the AGI will have started its existence with a utility function, and that function will have been given to it by its creator (the group).
So, the big unanswered question is what kind of utility function you think this proposed free AGI should have.
There is no such thing as an intelligent agent or a mind without a goal, a system of values or a utility function. It is a logical contradiction. Eliezer roughly 12 years ago was in the habit of referring to an AGI as a Really Powerful Optimizing Process (RPOP) and wrote of the RPOP’s steering reality into a tiny volume of the space of possible outcomes. (Please excuse my clumsy paraphrase of Eliezer’s writing.)
One could probably create a mind or an AGI that does nothing but accumulate the power to achieve goals without ever actually choosing a specific goal to achieve other than to continue to accumulate power. (Such a mind would be strongly motivated to destroy or control any other minds in its enviroment.) I doubt that is what you have in mind.