I’d like to check my understanding of the last two transitions a little. If someone could check the below I’d be grateful.
When we move to the utilitarianism-like quadrant, individual actors have preference ordering over actions (not states). So if they were the kind of utilitarians we often think about (with utilities attached to states), that would be something like ordering the actions by their expected utility (as they see it). So we actually get something like combined epistemic modesty and utilitarianism here?
Then, when we move to the futarchy-like quadrant, we additionally elicit probabilities for (all possible?) observations from each agent. We give them less and less weight in the decision calculus when their predictions are wrong. This stops agents that have crazy beliefs reliably voting us into world states they wouldn’t like anyway (though I think that world states aren’t present in the formalism in this quadrant).
Does the above paragraph mean that people with unique preferences and crazy beliefs eventually end up without having their preferences respected (whereas someone with unique preferences and accurate beliefs would still have their preferences respected)?
(Some extra questions. I’m more interested in answers to questions above the line, so feel free to skip these)
Also, do we have to treat the agents as well-calibrated across all domains? Or is the system able to learn that their thoughts should be given weight in some circumstances and not others? The reason I think we can’t do that is because it seems like there is just one number that represents the agent’s overall accuracy (theta_i)
A possible fix to the above is that individual agents could do this subject-specific evaluation of other agents and would update their credences based on partially-accurate agents, thus the information still gets preserved. But I think this leads to another problem: could there be a double-counting when both Critch’s mechanism and other agents pick up on the accuracy of an agent? Or are we fine because agents who over-update on others’ views also get their vote penalised?
Does the above paragraph mean that people with unique preferences and crazy beliefs eventually end up without having their preferences respected (whereas someone with unique preferences and accurate beliefs would still have their preferences respected)?
Yes. This might be too harsh. The “libertarian” argument in favor of it is: who are you to keep someone from betting away all of their credit in the system? If you make a rule preventing this, agents will tend to want to find some way around it. If you just give some free credit to agents who are completely out, this harms the calibration of the system by reducing the incentive to be sane about your bets.
On the other hand, there may well be a serious game-theoretic reason why it is “too harsh”: someone who is getting to cooperation from the system has no reason to cooperate in turn. I’m curious if a CCT-adjacent formalism could capture this (or some other reason to be gentler). That would be the kind of thing which might have interesting analogues when we try to import insights back into decision theory.
Also, do we have to treat the agents as well-calibrated across all domains? Or is the system able to learn that their thoughts should be given weight in some circumstances and not others?
In the formalism, no, you just win or lose points across all domains. Realistically, it seems prudent to introduce stuff like that.
A possible fix to the above is that individual agents could do this subject-specific evaluation of other agents and would update their credences based on partially-accurate agents, thus the information still gets preserved.
That’s exactly what could happen in a logical-induction like setting.
could there be a double-counting when both Critch’s mechanism and other agents pick up on the accuracy of an agent?
There might temporarily be all sorts of crazy stuff like this, but we know it would (somehow) self-correct eventually.
On the other hand, there may well be a serious game-theoretic reason why it is “too harsh”: someone who is getting to cooperation from the system has no reason to cooperate in turn.
Is there a typo here? (“getting to cooperation” → “getting no cooperation”). And the idea is that there are other ways of making an impact on the world than the decisions of the “futarchy”, so people who have no stake in the futarchy could mess things up other ways, right?
I’d like to check my understanding of the last two transitions a little. If someone could check the below I’d be grateful.
When we move to the utilitarianism-like quadrant, individual actors have preference ordering over actions (not states). So if they were the kind of utilitarians we often think about (with utilities attached to states), that would be something like ordering the actions by their expected utility (as they see it). So we actually get something like combined epistemic modesty and utilitarianism here?
Then, when we move to the futarchy-like quadrant, we additionally elicit probabilities for (all possible?) observations from each agent. We give them less and less weight in the decision calculus when their predictions are wrong. This stops agents that have crazy beliefs reliably voting us into world states they wouldn’t like anyway (though I think that world states aren’t present in the formalism in this quadrant).
Does the above paragraph mean that people with unique preferences and crazy beliefs eventually end up without having their preferences respected (whereas someone with unique preferences and accurate beliefs would still have their preferences respected)?
(Some extra questions. I’m more interested in answers to questions above the line, so feel free to skip these)
Also, do we have to treat the agents as well-calibrated across all domains? Or is the system able to learn that their thoughts should be given weight in some circumstances and not others? The reason I think we can’t do that is because it seems like there is just one number that represents the agent’s overall accuracy (theta_i)
A possible fix to the above is that individual agents could do this subject-specific evaluation of other agents and would update their credences based on partially-accurate agents, thus the information still gets preserved. But I think this leads to another problem: could there be a double-counting when both Critch’s mechanism and other agents pick up on the accuracy of an agent? Or are we fine because agents who over-update on others’ views also get their vote penalised?
Yes. This might be too harsh. The “libertarian” argument in favor of it is: who are you to keep someone from betting away all of their credit in the system? If you make a rule preventing this, agents will tend to want to find some way around it. If you just give some free credit to agents who are completely out, this harms the calibration of the system by reducing the incentive to be sane about your bets.
On the other hand, there may well be a serious game-theoretic reason why it is “too harsh”: someone who is getting to cooperation from the system has no reason to cooperate in turn. I’m curious if a CCT-adjacent formalism could capture this (or some other reason to be gentler). That would be the kind of thing which might have interesting analogues when we try to import insights back into decision theory.
In the formalism, no, you just win or lose points across all domains. Realistically, it seems prudent to introduce stuff like that.
That’s exactly what could happen in a logical-induction like setting.
There might temporarily be all sorts of crazy stuff like this, but we know it would (somehow) self-correct eventually.
Is there a typo here? (“getting to cooperation” → “getting no cooperation”). And the idea is that there are other ways of making an impact on the world than the decisions of the “futarchy”, so people who have no stake in the futarchy could mess things up other ways, right?