I think formally, such a circular consequentialist agent should not exist, since running a calculation of X’s utility either
Returns 0 utility, by avoiding self reference.
or
Runs in an endless loop and throws a stack overflow error, without returning any utility.
However, my guess is that in practice such an agent could exist, if we don’t insist on it being perfectly rational.
Instead of running a calculation of X’s utility, it has a intuitive guesstimate for X’s utility. A “vibe” for how much utility X has.
Over time, it adjusts its guesstimate of X’s utility based on whether X helps it acquire other things which have utility. If it discovers that X doesn’t achieve anything, it might reduce its guesstimate of X’s utility. However if it discovers that X helps it acquire Y which helps it acquire Z, and its guesstimate of Z’s utility is high, then it might increase its guesstimate of X’s utility.
And it may stay in an equilibrium where it guesses that all of these things have utility, because all of these things help it acquire one another.
I think the reason you value the items in the video game, is because humans have the mesaoptimizer goal of “success,” having something under your control grow and improve and be preserved.
Maybe one hope is that the artificial superintelligence will also have a bit of this goal, and place a bit of this value on humanity and what we wish for. Though obviously it can go wrong.
I think formally, such a circular consequentialist agent should not exist, since running a calculation of X’s utility either
Returns 0 utility, by avoiding self reference.
or
Runs in an endless loop and throws a stack overflow error, without returning any utility.
However, my guess is that in practice such an agent could exist, if we don’t insist on it being perfectly rational.
Instead of running a calculation of X’s utility, it has a intuitive guesstimate for X’s utility. A “vibe” for how much utility X has.
Over time, it adjusts its guesstimate of X’s utility based on whether X helps it acquire other things which have utility. If it discovers that X doesn’t achieve anything, it might reduce its guesstimate of X’s utility. However if it discovers that X helps it acquire Y which helps it acquire Z, and its guesstimate of Z’s utility is high, then it might increase its guesstimate of X’s utility.
And it may stay in an equilibrium where it guesses that all of these things have utility, because all of these things help it acquire one another.
I think the reason you value the items in the video game, is because humans have the mesaoptimizer goal of “success,” having something under your control grow and improve and be preserved.
Maybe one hope is that the artificial superintelligence will also have a bit of this goal, and place a bit of this value on humanity and what we wish for. Though obviously it can go wrong.