I don’t think Aumann’s agreement theorem is a good way to motivate your normative judgments, though I basically agree with your conclusions. I read Duncan’s post as well and did not really understand why he called you out. You both seem non-malevolent to me.
Bayesianism generalizes logical reasoning to uncertain claims, subject to certain consistency assumptions. Obviously humans are not ideal Bayesians. But in a deeper sense, maybe we’re not supposed to be. Not in an instrumental sense where being Bayesian is incompatible with some kind of good life, but rather in an epistemic sense. Maybe there is some mathematical theory of reasoning, we’ll call it Glorpism, of which humans are an approximation, and it is easier for humans to become more Glorpish than it is for us to become more Bayesian, and becoming more Glorpish is powerful and general in the sense we expect epistemic rationality to be. Glorpism may not have agreement guarantees in the way that Bayesianism does.
Sam Eisenstat’s Condensation is an example of something like this, although I don’t think it’s The Thing. Importantly, Condensation only has the translation theorem to the extent that models are hierarchically organized in a nice way, which does not always hold. (Apologies for any errors, feel free to correct me.)
I also think a purely functionalist account of reasoning error deletes a lot of information. For example, a Ruby that says, “oh, my bad” upon being confronted with evidence from computer analysis of photographs that the different images are all grey is different from a Ruby who changes the topic or flies into a rage. Among the first type of Ruby, those that systematically downgrade or restructure how they assign credence to their color-intuitions after admitting their error is different from those who “bounce back” to their original epistemic state. The best one of these is well-modelled by mistake theory. The worst two, conflict theory.
In real life, I think honest humans often agree to disagree. I do not fully understand why this is and consider this an important problem in the theory of powerful reasoners. I think part of it is that humans perform reasoning using words. Honest words correspond to natural categories but natural categories have an intrinsic misgeneralization problem. If you have two objects, korgs and spangs, which both have exactly half the properties each of bleggs and rubes, but different sets of these properties, then honest people might categorize them differently as bleggs and rubes. But this process is happening below the level of introspective access, so dissolving the question / debucketing has to be done “out loud” in the chamber of consciousness. The act of debucketing / rectifying definitions is a constraint problem with the constraints supplied by one’s introspection on hypotheticals. In general this can take exponential time in the number of traits used to define bleggs and rubes. (I do not have a proof of this, and expect the answer is sensitive to the formulation of the problem. This last claim is purely mathematical intuition.)
Also, our equivalent of Bayesian evidence is our sense-data, which is stored in an extremely unreliable compression system.
I don’t think Aumann’s agreement theorem is a good way to motivate your normative judgments, though I basically agree with your conclusions. I read Duncan’s post as well and did not really understand why he called you out. You both seem non-malevolent to me.
Bayesianism generalizes logical reasoning to uncertain claims, subject to certain consistency assumptions. Obviously humans are not ideal Bayesians. But in a deeper sense, maybe we’re not supposed to be. Not in an instrumental sense where being Bayesian is incompatible with some kind of good life, but rather in an epistemic sense. Maybe there is some mathematical theory of reasoning, we’ll call it Glorpism, of which humans are an approximation, and it is easier for humans to become more Glorpish than it is for us to become more Bayesian, and becoming more Glorpish is powerful and general in the sense we expect epistemic rationality to be. Glorpism may not have agreement guarantees in the way that Bayesianism does.
Sam Eisenstat’s Condensation is an example of something like this, although I don’t think it’s The Thing. Importantly, Condensation only has the translation theorem to the extent that models are hierarchically organized in a nice way, which does not always hold. (Apologies for any errors, feel free to correct me.)
I also think a purely functionalist account of reasoning error deletes a lot of information. For example, a Ruby that says, “oh, my bad” upon being confronted with evidence from computer analysis of photographs that the different images are all grey is different from a Ruby who changes the topic or flies into a rage. Among the first type of Ruby, those that systematically downgrade or restructure how they assign credence to their color-intuitions after admitting their error is different from those who “bounce back” to their original epistemic state. The best one of these is well-modelled by mistake theory. The worst two, conflict theory.
In real life, I think honest humans often agree to disagree. I do not fully understand why this is and consider this an important problem in the theory of powerful reasoners. I think part of it is that humans perform reasoning using words. Honest words correspond to natural categories but natural categories have an intrinsic misgeneralization problem. If you have two objects, korgs and spangs, which both have exactly half the properties each of bleggs and rubes, but different sets of these properties, then honest people might categorize them differently as bleggs and rubes. But this process is happening below the level of introspective access, so dissolving the question / debucketing has to be done “out loud” in the chamber of consciousness. The act of debucketing / rectifying definitions is a constraint problem with the constraints supplied by one’s introspection on hypotheticals. In general this can take exponential time in the number of traits used to define bleggs and rubes. (I do not have a proof of this, and expect the answer is sensitive to the formulation of the problem. This last claim is purely mathematical intuition.)
Also, our equivalent of Bayesian evidence is our sense-data, which is stored in an extremely unreliable compression system.