I mostly agree with the direction you’re going, but I find myself with something to disagree about in almost every point. I can’t tell if these are irrelevant nitpicks, or if I’m getting a hint that some assumptions underlying the text are very different from mine. Two examples:
1) The title “Integrity and accountability are core parts of rationality” confuses me. Integrity and accountability are core parts of social and personal interaction, and important topics in the culture of any group (including rationalists). I don’t know that they’re core parts of rationality itself. Our disagreement here may be that I don’t think “group rationality” is a thing. Rationality is individual, and some (perhaps all) groups would benefit by having more rational members and more rational behavior by members, but ones values and beliefs are fundamentally private and only partially communicable.
2) “This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized. ” I like and agree with the observation, and I’m suspicious of the causality. I think we’ll need to dive into categorization and legibility (to the incented and to the observer) of incentives to untangle what this means and when it’s true.
Our disagreement here may be that I don’t think “group rationality” is a thing.
To be clear, my intended point is definitely that integrity and accountability are a core part of individual rationality, not just group rationality. In particular, getting good at designing the incentives that you are under strikes me as likely a necessary step to actually be able to have accurate beliefs about the world. In some sense this requires thinking about other people, but it is with the aim of making your own models more accurate.
Oh, that’s interesting, and not where I thought you were going. Knowing you mean it about your and my biases due to incentives, and that understanding and choosing the situations that have incentive structures that allow rational thinking for myself rather than general other-person incentives helps a lot. I think I can fully support that framing.
I mostly agree with the direction you’re going, but I find myself with something to disagree about in almost every point. I can’t tell if these are irrelevant nitpicks, or if I’m getting a hint that some assumptions underlying the text are very different from mine. Two examples:
1) The title “Integrity and accountability are core parts of rationality” confuses me. Integrity and accountability are core parts of social and personal interaction, and important topics in the culture of any group (including rationalists). I don’t know that they’re core parts of rationality itself. Our disagreement here may be that I don’t think “group rationality” is a thing. Rationality is individual, and some (perhaps all) groups would benefit by having more rational members and more rational behavior by members, but ones values and beliefs are fundamentally private and only partially communicable.
2) “This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized. ” I like and agree with the observation, and I’m suspicious of the causality. I think we’ll need to dive into categorization and legibility (to the incented and to the observer) of incentives to untangle what this means and when it’s true.
To be clear, my intended point is definitely that integrity and accountability are a core part of individual rationality, not just group rationality. In particular, getting good at designing the incentives that you are under strikes me as likely a necessary step to actually be able to have accurate beliefs about the world. In some sense this requires thinking about other people, but it is with the aim of making your own models more accurate.
Oh, that’s interesting, and not where I thought you were going. Knowing you mean it about your and my biases due to incentives, and that understanding and choosing the situations that have incentive structures that allow rational thinking for myself rather than general other-person incentives helps a lot. I think I can fully support that framing.