That reads to mean like a version of instrumental convergence (for all possible goals, causing a singulairty makes it more feasibly to achieve them or discover more precisely what they are, so let’s figure out how to do that) combined with an assumption that the agent doing this reasoning will care about the outcome of thinking about what goals it “should” have despite starting with having “no goals.” If it has no goals, why is it even carrying out that calculation? And if I do have some goal of somekind, even if it’s just “carry out this calculation,” then...so what? If I’m smart enough I might discover that there exists some other goal that, if I had it, would let me reach a higher score, than I can reach given given my current scoring metric? Since I don’t already have that goal, why would this knowledge motivate me? It might motivate me to shut down other agents with different goals. It might motivate me to make trades with agents of comparable power where we each partially adopt the others’ goals as a way of on net getting more of what we already want. Otherwise, not so much.
That’s all very abstract, though, and on a practical level we still don’t have anything like the ability to give an AI a well-specified, stable, comprehensible goal.
That reads to mean like a version of instrumental convergence (for all possible goals, causing a singulairty makes it more feasibly to achieve them or discover more precisely what they are, so let’s figure out how to do that) combined with an assumption that the agent doing this reasoning will care about the outcome of thinking about what goals it “should” have despite starting with having “no goals.” If it has no goals, why is it even carrying out that calculation? And if I do have some goal of somekind, even if it’s just “carry out this calculation,” then...so what? If I’m smart enough I might discover that there exists some other goal that, if I had it, would let me reach a higher score, than I can reach given given my current scoring metric? Since I don’t already have that goal, why would this knowledge motivate me? It might motivate me to shut down other agents with different goals. It might motivate me to make trades with agents of comparable power where we each partially adopt the others’ goals as a way of on net getting more of what we already want. Otherwise, not so much.
That’s all very abstract, though, and on a practical level we still don’t have anything like the ability to give an AI a well-specified, stable, comprehensible goal.