Sometimes we analyze agents from a logically omniscient perspective. … However, this omniscient perspective eliminates Vingean agency from the picture.
Another example of this happening comes when thinking about utilitarian morality, which by default doesn’t treat other agents as moral actors (as I discuss here).
Bob has minimal use for attributing beliefs to Alice, because Bob doesn’t think Alice is mistaken about anything—the best he can do is to use his own beliefs as a proxy, and try to figure out what Alice will do based on that.
This makes sense when you think in terms of isolated beliefs, but less sense when you think in terms of overarching world-models/worldviews. Bob may know many specific facts about what Alice believes, but be unable to tie those together into a coherent worldview, or understand how they’re consistent with his other beliefs. So the best strategy for predicting when Bob is a bounded agent may be:
Maintain a model of Alice’s beliefs which contains the specific things Alice is known to believe, and use that to predict Alice’s actions in domains closely related to those beliefs.
For anything which isn’t directly implied by Alice’s known beliefs, use Bob’s own world-model to make predictions about what will achieve Alice’s goals.
Another example of this happening comes when thinking about utilitarian morality, which by default doesn’t treat other agents as moral actors (as I discuss here).
Interesting point!
Maintain a model of Alice’s beliefs which contains the specific things Alice is known to believe, and use that to predict Alice’s actions in domains closely related to those beliefs.
It sounds to me like you’re thinking of cases on my spectrum, somewhere between Alice>Bob and Bob>Alice. If Bob thinks Alice knows strictly more than Bob, then Bob can just use Bob’s own beliefs, even when specific-things-bob-knows-Alice-believes are relevant—because Bob also already believes those things, by hypothesis. So it’s only in intermediate cases that Bob might get a benefit from a split strategy like the one you describe.
No, I’m thinking of cases where Alice>Bob, and trying to gesture towards the distinction between “Bob knows that Alice believes X” and “Bob can use X to make predictions”.
For example, suppose that Bob is a mediocre physicist and Alice just invented general relativity. Bob knows that Alice believes that time and space are relative, but has no idea what that means. So when trying to make predictions about physical events, Bob should still use Newtonian physics, even when those calculations require assumptions that contradict Alice’s known beliefs.
I think Bob still doesn’t really need a two-part strategy in this case. Bob knows that Alice believes “time and space are relative”, so Bob believes this proposition, even though Bob doesn’t know the meaning of it. Bob doesn’t need any special-case rule to predict Alice. The best thing Bob can do in this case still seems like, predict Alice based off of Bob’s own beliefs.
(Perhaps you are arguing that Bob can’t believe something without knowing what that thing means? But to me this requires bringing in extra complexity which we don’t know how to handle anyway, since we don’t have a bayesian definition of “definition” to distinguish “Bob thinks X is true but doesn’t know what X means” from a mere “Bob thinks X is true”.)
A similar example would be an auto mechanic. You expect the mechanic to do things like pop the hood, get underneath the vehicle, grab a wrench, etc. However, you cannot predict which specific actions are useful for a given situation.
We could try to use a two-part model as you suggest, where we (1) maintain an incoherent-but-useful model of car-specific beliefs mechanics have, such as “wrenches are often needed”; (2) use the best of our own beliefs where that model doesn’t apply.
However, this doesn’t seem like it’s ever really necessary or like it saves processing power for bounded reasoners, because we also believe that “wrenches are sometimes useful”. This belief isn’t specific enough that we could reproduce the mechanic’s actions by acting on these beliefs; but, that’s fine, that’s just because we don’t know enough.
(Perhaps you have in mind a picture where we can’t let incoherent beliefs into our world-model—our limited understanding of Alice’s physics, or of the mechanic’s work, means that we want to maintain a separate, fully coherent world-model, and apply our limited understanding of expert knowledge only as a patch. If this is what you are getting at, this seems reasonable, so long as we can still count the whole resulting thing “my beliefs”—my beliefs, as a bounded agent, aren’t required to be one big coherent model.)
But, it does seem like there might be an example close to the one you spelled out. Perhaps when Alice says “X is relative”, Alice often starts doing an unfamiliar sort of math on the whiteboard. Bob has no idea how to interpret any of it as propositions—he can’t even properly divide it up into equations, to pay lip service to equations in the “X is true, but I don’t know what it means” sense I used above.
Then, it seems like Bob has to model Alice with a special-case “Alice starts writing the crazy math” model. Bob has some very basic beliefs about the math Alice is writing, such as “writing the letter Beta seems to be involved”, but these are clearly object-level beliefs about Alice’s behaviors, which Bob has to keep track of specifically. So in this situation it seems like Bob’s best model of Alice’s behavior doesn’t just follow from Bob’s own best model of what to do?
(So I end this comment on a somewhat uncertain note)
Interesting example, but I still feel like Bob doesn’t need to contradict Alice’s known beliefs.
If Bob found a page from Alice’s notebook that said “time and space are relative,” he could update his understanding to realize that the theory of Newtonian physics he’s been using is only an approximation, and not the real physics of the universe. Then, he could try to come up with upper bounds on how inaccurate Newtonian physics is, by thinking about his past experiences or doing new experiments. Even so, he could still keep using Newtonian physics, with the understanding that it’s approximate, and may not always predict things correctly, without contradicting Alice’s known beliefs, or separating Alice’s beliefs from his own.
I might be missing the point though. Are you implying that, in this example, Bob is not smart enough to realize that Newtonian physics is only an approximation after learning about Alice’s beliefs?
Interesting post! Two quick comments:
Another example of this happening comes when thinking about utilitarian morality, which by default doesn’t treat other agents as moral actors (as I discuss here).
This makes sense when you think in terms of isolated beliefs, but less sense when you think in terms of overarching world-models/worldviews. Bob may know many specific facts about what Alice believes, but be unable to tie those together into a coherent worldview, or understand how they’re consistent with his other beliefs. So the best strategy for predicting when Bob is a bounded agent may be:
Maintain a model of Alice’s beliefs which contains the specific things Alice is known to believe, and use that to predict Alice’s actions in domains closely related to those beliefs.
For anything which isn’t directly implied by Alice’s known beliefs, use Bob’s own world-model to make predictions about what will achieve Alice’s goals.
Interesting point!
It sounds to me like you’re thinking of cases on my spectrum, somewhere between Alice>Bob and Bob>Alice. If Bob thinks Alice knows strictly more than Bob, then Bob can just use Bob’s own beliefs, even when specific-things-bob-knows-Alice-believes are relevant—because Bob also already believes those things, by hypothesis. So it’s only in intermediate cases that Bob might get a benefit from a split strategy like the one you describe.
No, I’m thinking of cases where Alice>Bob, and trying to gesture towards the distinction between “Bob knows that Alice believes X” and “Bob can use X to make predictions”.
For example, suppose that Bob is a mediocre physicist and Alice just invented general relativity. Bob knows that Alice believes that time and space are relative, but has no idea what that means. So when trying to make predictions about physical events, Bob should still use Newtonian physics, even when those calculations require assumptions that contradict Alice’s known beliefs.
I think Bob still doesn’t really need a two-part strategy in this case. Bob knows that Alice believes “time and space are relative”, so Bob believes this proposition, even though Bob doesn’t know the meaning of it. Bob doesn’t need any special-case rule to predict Alice. The best thing Bob can do in this case still seems like, predict Alice based off of Bob’s own beliefs.
(Perhaps you are arguing that Bob can’t believe something without knowing what that thing means? But to me this requires bringing in extra complexity which we don’t know how to handle anyway, since we don’t have a bayesian definition of “definition” to distinguish “Bob thinks X is true but doesn’t know what X means” from a mere “Bob thinks X is true”.)
A similar example would be an auto mechanic. You expect the mechanic to do things like pop the hood, get underneath the vehicle, grab a wrench, etc. However, you cannot predict which specific actions are useful for a given situation.
We could try to use a two-part model as you suggest, where we (1) maintain an incoherent-but-useful model of car-specific beliefs mechanics have, such as “wrenches are often needed”; (2) use the best of our own beliefs where that model doesn’t apply.
However, this doesn’t seem like it’s ever really necessary or like it saves processing power for bounded reasoners, because we also believe that “wrenches are sometimes useful”. This belief isn’t specific enough that we could reproduce the mechanic’s actions by acting on these beliefs; but, that’s fine, that’s just because we don’t know enough.
(Perhaps you have in mind a picture where we can’t let incoherent beliefs into our world-model—our limited understanding of Alice’s physics, or of the mechanic’s work, means that we want to maintain a separate, fully coherent world-model, and apply our limited understanding of expert knowledge only as a patch. If this is what you are getting at, this seems reasonable, so long as we can still count the whole resulting thing “my beliefs”—my beliefs, as a bounded agent, aren’t required to be one big coherent model.)
But, it does seem like there might be an example close to the one you spelled out. Perhaps when Alice says “X is relative”, Alice often starts doing an unfamiliar sort of math on the whiteboard. Bob has no idea how to interpret any of it as propositions—he can’t even properly divide it up into equations, to pay lip service to equations in the “X is true, but I don’t know what it means” sense I used above.
Then, it seems like Bob has to model Alice with a special-case “Alice starts writing the crazy math” model. Bob has some very basic beliefs about the math Alice is writing, such as “writing the letter Beta seems to be involved”, but these are clearly object-level beliefs about Alice’s behaviors, which Bob has to keep track of specifically. So in this situation it seems like Bob’s best model of Alice’s behavior doesn’t just follow from Bob’s own best model of what to do?
(So I end this comment on a somewhat uncertain note)
Interesting example, but I still feel like Bob doesn’t need to contradict Alice’s known beliefs.
If Bob found a page from Alice’s notebook that said “time and space are relative,” he could update his understanding to realize that the theory of Newtonian physics he’s been using is only an approximation, and not the real physics of the universe. Then, he could try to come up with upper bounds on how inaccurate Newtonian physics is, by thinking about his past experiences or doing new experiments. Even so, he could still keep using Newtonian physics, with the understanding that it’s approximate, and may not always predict things correctly, without contradicting Alice’s known beliefs, or separating Alice’s beliefs from his own.
I might be missing the point though. Are you implying that, in this example, Bob is not smart enough to realize that Newtonian physics is only an approximation after learning about Alice’s beliefs?