For the world-model, it’s not actually incoherent because we cut the link and update the distribution of the subsequent agent.
I’m gonna see if I can explain this in more detail—you can correct me if I’m wrong.
In common sense, I would say “Suppose I burn the kite. What happens in the future? Is it good or bad? OK, suppose I don’t burn the kite. What happens in the future? Is it good or bad?” And then decide on that basis.
But that’s EDT.
CDT is different.
In CDT I can have future expectations that follow logically from burning the kite, but they don’t factor in as considerations, because they don’t causally flow from the decision according to the causal diagram in my head.
Smoking lesion is a pretty intuitive example for us to think about, because smoking lesion involves a plausible causal diagram of the world.
Here we’re taking the same idea, but I (=the LCDT agent) have a wildly implausible causal diagram of the world. “If I burn the kite, then the person won’t move the kite, but c’mon, that’s not because I burned the kite!”
Just like the smoking lesion, I have the idea that the kite might or might not be there, but that’s a fact about the world that’s somehow predetermined before decision time, not because of my decision, and therefore doesn’t factor into my decision.
…Maybe. Did I get that right?
Anyway, I usually think of a world-model as having causality in it, as opposed to causal diagrams being a separate layer that exists on top of a world model. So I would disagree with “not actually incoherent”. Specifically, I think if an agent can do the kind of reasoning that would allow it to create a causal world-model in the first place, then the same kind of reasoning would lead it to realize that there is in fact supposed to be a link at each of the places where we manually cut it—i.e., that the causal world-model is incoherent.
Specifically, I think if an agent can do the kind of reasoning that would allow it to create a causal world-model in the first place, then the same kind of reasoning would lead it to realize that there is in fact supposed to be a link at each of the places where we manually cut it—i.e., that the causal world-model is incoherent.
An LCDT agent should certainly be aware of the fact that those causal chains actually exist—it just shouldn’t care about that. If you want to argue that it’ll change to not using LCDT to make decisions anymore, you have to argue that, under the decision rules of LCDT, it will choose to self-modify in some particular situation—but LCDT should rule out its ability to ever believe that any self-modification will do anything, thus ensuring that, once an agent starts making decisions using LCDT, it shouldn’t stop.
In addition to Evan’s answer (with which I agree), I want to make explicit an assumption I realized after reading your last paragraph: we assume that the causal graph is the final result of the LCDT agent consulting its world model to get a “model” of the task at hand. After that point (which includes drawing causality and how the distributions impacts each other, as well as the sources’ distributions), the LCDT agent only decides based on this causal graph. In this case it cuts the causal links to agent and then decide CDT style.
None of this result in an incoherent world model because the additional knowledge that could be used to realize that the cuts are not “real”, is not available in the truncated causal model, and thus cannot be accessed while making the decision.
I honestly feel this is the crux of our talking past each other (same with Joe) in the last few comments. Do you think that’s right?
I’m gonna see if I can explain this in more detail—you can correct me if I’m wrong.
In common sense, I would say “Suppose I burn the kite. What happens in the future? Is it good or bad? OK, suppose I don’t burn the kite. What happens in the future? Is it good or bad?” And then decide on that basis.
But that’s EDT.
CDT is different.
In CDT I can have future expectations that follow logically from burning the kite, but they don’t factor in as considerations, because they don’t causally flow from the decision according to the causal diagram in my head.
The classic example is smoking lesion.
Smoking lesion is a pretty intuitive example for us to think about, because smoking lesion involves a plausible causal diagram of the world.
Here we’re taking the same idea, but I (=the LCDT agent) have a wildly implausible causal diagram of the world. “If I burn the kite, then the person won’t move the kite, but c’mon, that’s not because I burned the kite!”
Just like the smoking lesion, I have the idea that the kite might or might not be there, but that’s a fact about the world that’s somehow predetermined before decision time, not because of my decision, and therefore doesn’t factor into my decision.
…Maybe. Did I get that right?
Anyway, I usually think of a world-model as having causality in it, as opposed to causal diagrams being a separate layer that exists on top of a world model. So I would disagree with “not actually incoherent”. Specifically, I think if an agent can do the kind of reasoning that would allow it to create a causal world-model in the first place, then the same kind of reasoning would lead it to realize that there is in fact supposed to be a link at each of the places where we manually cut it—i.e., that the causal world-model is incoherent.
Sorry if I’m confused.
An LCDT agent should certainly be aware of the fact that those causal chains actually exist—it just shouldn’t care about that. If you want to argue that it’ll change to not using LCDT to make decisions anymore, you have to argue that, under the decision rules of LCDT, it will choose to self-modify in some particular situation—but LCDT should rule out its ability to ever believe that any self-modification will do anything, thus ensuring that, once an agent starts making decisions using LCDT, it shouldn’t stop.
In addition to Evan’s answer (with which I agree), I want to make explicit an assumption I realized after reading your last paragraph: we assume that the causal graph is the final result of the LCDT agent consulting its world model to get a “model” of the task at hand. After that point (which includes drawing causality and how the distributions impacts each other, as well as the sources’ distributions), the LCDT agent only decides based on this causal graph. In this case it cuts the causal links to agent and then decide CDT style.
None of this result in an incoherent world model because the additional knowledge that could be used to realize that the cuts are not “real”, is not available in the truncated causal model, and thus cannot be accessed while making the decision.
I honestly feel this is the crux of our talking past each other (same with Joe) in the last few comments. Do you think that’s right?