That is, the deal only makes sense for you to propose if you have a stake in something other than the accuracy of your own beliefs—if there’s something it would benefit you for me to believe.
Yes, Alice may be talking with Bob because she wants Bob to believe something that Alice thinks is true. And Bob may be talking with Alice because he wants Alice to believe something that Bob thinks is true.
But then if I’m truthseeking, then there’s no reason for me to accept the deal (because whether a proposed compute reallocation improves my beliefs shouldn’t depend on what you’re doing with your compute).
Bob may accept the deal in order to have Alice reciprocate and spend compute on seriously pondering whether Bob might be right. Again, this makes sense to Bob even if he just cares about Alice being right, if he has reason to believe she’s currently wrong.
It seems like the deal should only go through if we both have a stake in the other person’s beliefs that isn’t about the truth or falsity of those beliefs
No, it can be about both people wanting the other one to have true beliefs, if they come in with separate beliefs about what’s true.
I think maybe you think this situation shouldn’t be stable because of something about Aumann’s agreement theorem making this persistent disagreement unstable if both Alice and Bob were actually only concerned with the truth. But Aumann’s theorem doesn’t just require mutual rationality but common knowledge of mutual rationality. That’s a high bar that approximately no conversation meets because it’s always plausible that your counter-party is acting in bad-faith. (Or that they think it’s plausible that you act in bad faith, or that they think it’s plausible that you think it’s plausible that they are acting in bad faith, etc.) At which points it seems consistent to me to (i) have mutual knowledge of disagreement, and (ii) assign high probability to your conversation partner being good-faith and rational. [Edit: Well, it’s unlikely that any human is perfect on those aspects. But sufficiently good-faith and rational to be worthwhile to keep talking with in a pretty assuming-good-faith kind of way.]
This doesn’t at all seem like an easy operation that one participant in the conversation can do with 0 effort from the other participant. Language is not very high band-width. People come into conversations with different abstractions and world models etc. It’s not feasible to dump all the reasons you believe something, because it probably hooks into deep parts of your world model. It’s not sufficient to dump an argument that convinced you because those words were converted to concepts in your brain and may not correspond to the same concepts in someone else’s brain and/or because the argument may have relied on previous beliefs you had which aren’t shared by your conversational partner. If Alice is going to convince Bob of her view, there’s definitely a lot of computational labor involved in that which is more efficiently done by Bob than by Alice.
That’s a high bar that approximately no conversation meets because it’s always plausible that your counter-party is acting in bad-faith.
Yes, and those plausible worlds are exactly why I’m so suspicious of the proposed compute reallocation deal: I’m worried that “you’re being uncharitable; try to think harder about why I’m right” is something people around here say to conceal the fact that they don’t actually have an argument.
Maybe you’re right that the deal could make sense (and my argument that it doesn’t is wrong), but it gets scuttled by the adverse selection problem (where good faith actors would prefer to make such deals with each other, but they have no way to credibly distinguish themselves from bad faith actors who are just bluffing)?
Well sometimes it gets scuttled and sometimes it doesn’t. Cars do in fact get sold second-hand, despite the adverse selection problem. Often the gains-from-trade are large enough to justify some risk.
I think there are often big gains from people putting in some work to understand where their conversation partner is coming from.
Also, “no way to credibly distinguish themselves” seems like a big overstatement. You can do way better than chance at telling whether someone is reasonable or not, especially in repeated interactions:
For one, if someone says “you’re being uncharitable; try to think harder about why I’m right”—you can see that as a reputational bet that you’ll learn something interesting from thinking harder about why they’re right. If you try that a couple of times and you don’t learn anything new, then that’s reasonable inductive evidence that you won’t benefit from that exercise with that person again. You’re not infinitely exploitable here.
And from the perspective of the “deal”—where you put in some effort to understand their perspective, and they put in some effort on understanding yours, even if you don’t expect your part of the deal to be worthwhile in isolation—I think it’s pretty easy to get evidence for whether they’ve held up their part of the deal. Like: Can they demonstrate any understanding of what you’re trying to say? (In the limit this is “can they pass your ITT” but conversations often involve much weaker versions, like “so you’re saying [blah]”, or even less explicit demonstrations of understanding, like engaging with what the other person see as the central point.)
This seems wrong to me.
Yes, Alice may be talking with Bob because she wants Bob to believe something that Alice thinks is true. And Bob may be talking with Alice because he wants Alice to believe something that Bob thinks is true.
Bob may accept the deal in order to have Alice reciprocate and spend compute on seriously pondering whether Bob might be right. Again, this makes sense to Bob even if he just cares about Alice being right, if he has reason to believe she’s currently wrong.
No, it can be about both people wanting the other one to have true beliefs, if they come in with separate beliefs about what’s true.
I think maybe you think this situation shouldn’t be stable because of something about Aumann’s agreement theorem making this persistent disagreement unstable if both Alice and Bob were actually only concerned with the truth. But Aumann’s theorem doesn’t just require mutual rationality but common knowledge of mutual rationality. That’s a high bar that approximately no conversation meets because it’s always plausible that your counter-party is acting in bad-faith. (Or that they think it’s plausible that you act in bad faith, or that they think it’s plausible that you think it’s plausible that they are acting in bad faith, etc.) At which points it seems consistent to me to (i) have mutual knowledge of disagreement, and (ii) assign high probability to your conversation partner being good-faith and rational. [Edit: Well, it’s unlikely that any human is perfect on those aspects. But sufficiently good-faith and rational to be worthwhile to keep talking with in a pretty assuming-good-faith kind of way.]
Also...
This doesn’t at all seem like an easy operation that one participant in the conversation can do with 0 effort from the other participant. Language is not very high band-width. People come into conversations with different abstractions and world models etc. It’s not feasible to dump all the reasons you believe something, because it probably hooks into deep parts of your world model. It’s not sufficient to dump an argument that convinced you because those words were converted to concepts in your brain and may not correspond to the same concepts in someone else’s brain and/or because the argument may have relied on previous beliefs you had which aren’t shared by your conversational partner. If Alice is going to convince Bob of her view, there’s definitely a lot of computational labor involved in that which is more efficiently done by Bob than by Alice.
Yes, and those plausible worlds are exactly why I’m so suspicious of the proposed compute reallocation deal: I’m worried that “you’re being uncharitable; try to think harder about why I’m right” is something people around here say to conceal the fact that they don’t actually have an argument.
Maybe you’re right that the deal could make sense (and my argument that it doesn’t is wrong), but it gets scuttled by the adverse selection problem (where good faith actors would prefer to make such deals with each other, but they have no way to credibly distinguish themselves from bad faith actors who are just bluffing)?
Well sometimes it gets scuttled and sometimes it doesn’t. Cars do in fact get sold second-hand, despite the adverse selection problem. Often the gains-from-trade are large enough to justify some risk.
I think there are often big gains from people putting in some work to understand where their conversation partner is coming from.
Also, “no way to credibly distinguish themselves” seems like a big overstatement. You can do way better than chance at telling whether someone is reasonable or not, especially in repeated interactions:
For one, if someone says “you’re being uncharitable; try to think harder about why I’m right”—you can see that as a reputational bet that you’ll learn something interesting from thinking harder about why they’re right. If you try that a couple of times and you don’t learn anything new, then that’s reasonable inductive evidence that you won’t benefit from that exercise with that person again. You’re not infinitely exploitable here.
And from the perspective of the “deal”—where you put in some effort to understand their perspective, and they put in some effort on understanding yours, even if you don’t expect your part of the deal to be worthwhile in isolation—I think it’s pretty easy to get evidence for whether they’ve held up their part of the deal. Like: Can they demonstrate any understanding of what you’re trying to say? (In the limit this is “can they pass your ITT” but conversations often involve much weaker versions, like “so you’re saying [blah]”, or even less explicit demonstrations of understanding, like engaging with what the other person see as the central point.)