In retrospect, I think this comment of mine didn’t address Jan’s key point, which is that we often form intuitions/emotions by running a process analagous to aggregating data into a summary statistic and then throwing away the data. Now the evidence we saw is quite incommunicable—we no longer have the evidence ourselves.
Ray Arnold gave me a good example the other day of two people—one an individualist libertarian, the other a communitarian Christian. In the example these two people deeply disagree on how society should be set up, and this is entirely because they’re two identical RL systems built on different training sets (one has repeatedly seen the costs of trying to trust others with your values, and the other has repeatedly seen it work out brilliantly). Their brains have compressed the data into a single emotion, that they feel in groups trying to coordinate (say). Overall they might be able to introspect enough to communicate the causes of their beliefs, but they might not—they might just be stuck this way (until we reach the glorious transhumanist future, that is). Scott might expect them to say they just have fundamental value differences.
I agree that I have not in the OP given a full model of the different parts of the brain, how they do reasoning, and which parts are (or aren’t) in principle communicable or trustworthy. I at least claim that I’ve pointed to a vague mechanism that’s more true than the simple model where everyone just has the outputs of their beliefs. There are important gears that are hard-but-possible to communicate, and they’re generally worth focusing on over and above the credences they output. (Will write more on this in a future post about Aumann’s Agreement Theorem.)
In retrospect, I think this comment of mine didn’t address Jan’s key point, which is that we often form intuitions/emotions by running a process analagous to aggregating data into a summary statistic and then throwing away the data. Now the evidence we saw is quite incommunicable—we no longer have the evidence ourselves.
Ray Arnold gave me a good example the other day of two people—one an individualist libertarian, the other a communitarian Christian. In the example these two people deeply disagree on how society should be set up, and this is entirely because they’re two identical RL systems built on different training sets (one has repeatedly seen the costs of trying to trust others with your values, and the other has repeatedly seen it work out brilliantly). Their brains have compressed the data into a single emotion, that they feel in groups trying to coordinate (say). Overall they might be able to introspect enough to communicate the causes of their beliefs, but they might not—they might just be stuck this way (until we reach the glorious transhumanist future, that is). Scott might expect them to say they just have fundamental value differences.
I agree that I have not in the OP given a full model of the different parts of the brain, how they do reasoning, and which parts are (or aren’t) in principle communicable or trustworthy. I at least claim that I’ve pointed to a vague mechanism that’s more true than the simple model where everyone just has the outputs of their beliefs. There are important gears that are hard-but-possible to communicate, and they’re generally worth focusing on over and above the credences they output. (Will write more on this in a future post about Aumann’s Agreement Theorem.)