https://arxiv.org/abs/1605.01021: “Our framework pays every agent a measure of mutual information between her signal and a peer’s signal”—any strategy besides truth telling will decrease every agent’s expected payment, under certain conditions
https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism if each agent in an auction gets rewarded a function of the values reported by other agents, plus the sum of the values reported by other agents, then the “winning strategy” to get the most reward is to report your own value function honestly. this implements the utilitarian welfare function (maximizing the sum of values of the agents.)
https://web.stanford.edu/~gentzkow/research/BayesianPersuasion.pdf no matter what distribution of posterior beliefs you pick, there is some information a “sender” can give that will make a Bayesian-rational “receiver” update to it, provided that the expected value is the same as the prior’s (conservation of expected evidence).
“Consider the example of a prosecutor trying to convince a judge that a defendant is guilty. When the defendant is indeed guilty, revealing the facts of the case will tend to help the prosecutor’s case. When the defendant is innocent, revealing facts will tend to hurt the prosecutor’s case. Can the prosecutor structure his arguments, selection of evidence, etc. so as to increase the probability of conviction by a rational judge on average? Perhaps surprisingly, the answer to this question is yes. Bayes’s Law restricts the expectation of posterior beliefs but puts no other constraints on their distribution. Therefore, so long as the judge’s action is not linear in her beliefs, the prosecutor may benefit from persuasion.”
a “solver” agent solves a problem, an “auditor” agent evaluates the correctness of the solution, and both are trained with rewards tuned to optimize the accuracy of the entire system. Empirically, varying schedules of rewards performs better than giving a fixed reward based on correctness and much better than giving the agents exactly the default rewards that the “principal” receives (i.e. +1 for correct results that the auditor passes, −1 for false negatives from the auditor, etc). in fact solver + auditor with fixed rewards performs no better at avoiding hallucinations than a solver alone, while variable rewards help a lot! mostly this comes from the adaptive-reward-trained solver-auditor system abstaining more when unsure rather than hallucinating.
links 7/29/26: https://roamresearch.com/#/app/srcpublic/page/07-29-2026
https://en.wikipedia.org/wiki/Brier_score lower is better (I always forget)
mechanism design for rewarding truth-telling:
https://arxiv.org/abs/1605.01021: “Our framework pays every agent a measure of mutual information between her signal and a peer’s signal”—any strategy besides truth telling will decrease every agent’s expected payment, under certain conditions
https://www.lesswrong.com/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making incentivize AIs to tell the truth via a prediction market where each agent is paid the amount they move the market
https://arielrubinstein.tau.ac.il/papers/debates.pdf how do debate rules actually influence the chance that the listener learns the right answer? toy example, experiment with humans
https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism if each agent in an auction gets rewarded a function of the values reported by other agents, plus the sum of the values reported by other agents, then the “winning strategy” to get the most reward is to report your own value function honestly. this implements the utilitarian welfare function (maximizing the sum of values of the agents.)
https://www.kellogg.northwestern.edu/research/math/papers/284.pdf “Incentive Compatibility and the Bargaining Problem”, Myerson 1977.
https://web.stanford.edu/~gentzkow/research/BayesianPersuasion.pdf no matter what distribution of posterior beliefs you pick, there is some information a “sender” can give that will make a Bayesian-rational “receiver” update to it, provided that the expected value is the same as the prior’s (conservation of expected evidence).
“Consider the example of a prosecutor trying to convince a judge that a defendant is guilty. When the defendant is indeed guilty, revealing the facts of the case will tend to help the prosecutor’s case. When the defendant is innocent, revealing facts will tend to hurt the prosecutor’s case. Can the prosecutor structure his arguments, selection of evidence, etc. so as to increase the probability of conviction by a rational judge on average? Perhaps surprisingly, the answer to this question is yes. Bayes’s Law restricts the expectation of posterior beliefs but puts no other constraints on their distribution. Therefore, so long as the judge’s action is not linear in her beliefs, the prosecutor may benefit from persuasion.”
https://arxiv.org/pdf/2605.01643 “AI Alignment via Incentives and Correction”
a “solver” agent solves a problem, an “auditor” agent evaluates the correctness of the solution, and both are trained with rewards tuned to optimize the accuracy of the entire system. Empirically, varying schedules of rewards performs better than giving a fixed reward based on correctness and much better than giving the agents exactly the default rewards that the “principal” receives (i.e. +1 for correct results that the auditor passes, −1 for false negatives from the auditor, etc). in fact solver + auditor with fixed rewards performs no better at avoiding hallucinations than a solver alone, while variable rewards help a lot! mostly this comes from the adaptive-reward-trained solver-auditor system abstaining more when unsure rather than hallucinating.