UDT is drawing attention to issues with how algorithms influence each other, how we should reason with uncertain knowledge about such influence, and how decisions under that uncertainty should be made by those algorithms. After an agent updates, these problems don’t go away, so being “updateless” is less central to the point of UDT than all the rest, even if a lot of discussion of UDT and proposed solutions to decision problems involve an unusual amount of not-updating.
For example, consider an outcome W = C(A(O())), where W is an algorithm/term that’s a composition of continuation C, agent A, and observation O (let’s say it’s also given as an algorithm, but A directly observes only the value it computes). When A wants to reason about how to influence W, it needs to know something about C, even though it doesn’t even observe its value. It’s not obvious what about C should interest A, its value C(-) as a function isn’t necessarily relevant for A’s decisions if C has other instances of A as its parts (for example as subterms within C itself rather than only of C(A(O()))). Now C and O seem to play a similar role in connecting A to W, the only difference is that A gets to observe the value of O (in some not obviously relevant sense, once A “becomes” the composition A(O())). So similarly A might need to know something about O that is not just its value, even “prior” to observing its value (when A is just A itself rather than A(O()), especially if O has other instances of A as its parts. The Absent-Minded Driver problem illustrates this, where one instance of the agent has the other instance in its continuation, while that other instance has the first instance of the agent in its observation-as-algorithm.
It makes sense that A has already updated on some knowledge about C and O, even if that knowledge doesn’t include an already-computed value of O. For example A might already know some of the code in C and O, or facts about their code, which is often assumed in decision problems. So agents are already not perfectly undateless, in the sense that they already know the decision problem, which can involve knowing something about observation-as-algorithm.
Updating on observations seems to ask about how A(O()) should behave, as opposed to how A(-) should behave, in order to influence the value of W. But A(O()) still has the same problems with C as A(-) did (for example C could have other instances of A(O()) as its parts), it only got rid of A’s problems with O, and the problems with C seem largely analogous to the problems with O (considered as an algorithm), so it’s not even a crucial change.
I’m reasoning about updatelessness because I’ve recently been investigating an updateful theory of embedded agency, not because I think it’s the only embeddedness problem.
UDT is drawing attention to issues with how algorithms influence each other, how we should reason with uncertain knowledge about such influence, and how decisions under that uncertainty should be made by those algorithms. After an agent updates, these problems don’t go away, so being “updateless” is less central to the point of UDT than all the rest, even if a lot of discussion of UDT and proposed solutions to decision problems involve an unusual amount of not-updating.
For example, consider an outcome W = C(A(O())), where W is an algorithm/term that’s a composition of continuation C, agent A, and observation O (let’s say it’s also given as an algorithm, but A directly observes only the value it computes). When A wants to reason about how to influence W, it needs to know something about C, even though it doesn’t even observe its value. It’s not obvious what about C should interest A, its value C(-) as a function isn’t necessarily relevant for A’s decisions if C has other instances of A as its parts (for example as subterms within C itself rather than only of C(A(O()))). Now C and O seem to play a similar role in connecting A to W, the only difference is that A gets to observe the value of O (in some not obviously relevant sense, once A “becomes” the composition A(O())). So similarly A might need to know something about O that is not just its value, even “prior” to observing its value (when A is just A itself rather than A(O()), especially if O has other instances of A as its parts. The Absent-Minded Driver problem illustrates this, where one instance of the agent has the other instance in its continuation, while that other instance has the first instance of the agent in its observation-as-algorithm.
It makes sense that A has already updated on some knowledge about C and O, even if that knowledge doesn’t include an already-computed value of O. For example A might already know some of the code in C and O, or facts about their code, which is often assumed in decision problems. So agents are already not perfectly undateless, in the sense that they already know the decision problem, which can involve knowing something about observation-as-algorithm.
Updating on observations seems to ask about how A(O()) should behave, as opposed to how A(-) should behave, in order to influence the value of W. But A(O()) still has the same problems with C as A(-) did (for example C could have other instances of A(O()) as its parts), it only got rid of A’s problems with O, and the problems with C seem largely analogous to the problems with O (considered as an algorithm), so it’s not even a crucial change.
I’m reasoning about updatelessness because I’ve recently been investigating an updateful theory of embedded agency, not because I think it’s the only embeddedness problem.