Of arguments and wagers

Link post

Au­to­mat­i­cally cross­posted from ai-al­ign­ment.com

(In which I ex­plore an un­usual way of com­bin­ing the two.)

Sup­pose that Alice and Bob dis­agree, and both care about Judy’s opinion. Per­haps Alice wants to con­vince Judy that rais­ing the min­i­mum wage is a cost-effec­tive way to fight poverty, and Bob wants to con­vince Judy that it isn’t.

If Judy has the same back­ground knowl­edge as Alice and Bob, and is will­ing to spend as much time think­ing about the is­sue as they have, then she can hear all of their ar­gu­ments and de­cide for her­self whom she be­lieves.

But in many cases Judy will have much less time than Alice or Bob, and is miss­ing a lot of rele­vant back­ground knowl­edge. Often Judy can’t even un­der­stand the key con­sid­er­a­tions in the ar­gu­ment; how can she hope to ar­bi­trate it?


For a warm-up, imag­ine that Judy could eval­u­ate the ar­gu­ments if she spent a long enough think­ing about them.

To save time, she could make Alice and Bob wa­ger on the re­sult. If both of them be­lieve they’ll win the ar­gu­ment, then they should be happy to agree to the deal: “If I win the ar­gu­ment I get $100; if I lose I pay $100.” (Note: by the end of the post, no dol­lars will need to be in­volved.)

If ei­ther side isn’t will­ing to take the bet, then Judy could de­clare the case set­tled with­out wast­ing her time. If they are both will­ing to bet, then Judy can hear them out and de­cide who she agrees with. That per­son “wins” the ar­gu­ment, and the bet: Alice and Bob are bet­ting about what Judy will be­lieve, not about the facts on the ground.

Of course we don’t have to stick with 1:1 bets. Judy wants to know the prob­a­bil­ity that she will be con­vinced, and so wants to know at what odds the two par­ties are both will­ing to bet. Based on that prob­a­bil­ity, she can de­cide if she wants to hear the ar­gu­ments.

It may be that both par­ties are happy to take 2:1 bets, i.e. each be­lieves they have a 23 chance of be­ing right. What should Judy be­lieve? (In fact this should always hap­pen at small stakes: both par­ti­ci­pants are will­ing to pay some pre­mium to try to con­vince Judy. For ex­am­ple, no mat­ter what Alice be­lieves, she would prob­a­bly be will­ing to take a bet of $0.10 against $0.01, if do­ing so would help her con­vince Judy.)

If this hap­pens, there is an ar­bi­trage op­por­tu­nity: Judy can make 2:1 bets with both of them, and end up with a guaran­teed profit. So we can con­tin­u­ously raise the re­quired stakes for each wa­ger, un­til ei­ther (1) the mar­ket ap­prox­i­mately clears, i.e. the two are will­ing to bet at nearly the same odds, or (2) the ar­bi­trage gap is large enough to com­pen­sate Judy for the time of hear­ing the ar­gu­ment. If (2) hap­pens, then Judy im­ple­ments the ar­bi­trage and hears the ar­gu­ments. (In this case Judy gets paid for her time, but the pay is in­de­pen­dent of what she de­cides.)


Bet­ting about the whole claim saved us some time (at best). Bet­ting about parts of the claim might get us much fur­ther.

In the course of ar­gu­ing, Alice and Bob will prob­a­bly rely on in­ter­me­di­ate claims or sum­maries of par­tic­u­lar ev­i­dence. For ex­am­ple, Alice might provide a short re­port de­scribing what we should in­fer from study Z, or Bob might claim “The anal­y­sis in study Z is so prob­le­matic that we should ig­nore it.”

Let’s al­low any­one to make a claim at any time. But if Alice makes a claim, Bob can make a coun­ter­claim that he feels bet­ter rep­re­sents the ev­i­dence. Then we have a re­cur­sive ar­gu­ment to de­cide which ver­sion bet­ter rep­re­sents the ev­i­dence.

The key idea is that this re­cur­sive ar­gu­ment can also be set­tled by bet­ting. So one of two things hap­pens: (1) Judy is told the mar­ket-clear­ing odds, and can use that in­for­ma­tion to help set­tle the origi­nal ar­gu­ment, or (2) there is an ar­bi­trage op­por­tu­nity, so Judy hears out the ar­gu­ment and col­lects the prof­its to com­pen­sate her for the time.

This re­cur­sive ar­gu­ment is made in con­text: that is, Judy eval­u­ates which of the two claims she feels would be a more helpful sum­mary within the origi­nal ar­gu­ment. Some­times this will be a ques­tion of fact about which Alice and Bob dis­agree, but some­times it will be a more com­pli­cated judg­ment call. For ex­am­ple, we could even have a re­cur­sive ar­gu­ment about which word­ing bet­ter re­flects the nu­ances of the situ­a­tion.

When mak­ing this eval­u­a­tion, Judy uses facts she learned over the course of the ar­gu­ment, but she in­ter­prets the claim as she would have in­ter­preted it at the be­gin­ning of the ar­gu­ment. For ex­am­ple, if Bob as­serts “The el­lip­soid al­gorithm is effi­cient” and Alice dis­agrees, Bob can­not win the ar­gu­ment by ex­plain­ing that “effi­cient” is a tech­ni­cal term which in con­text means “polyno­mial time”—un­less that’s how Judy would have un­der­stood the state­ment to start with.

This al­lows Judy to ar­bi­trate dis­agree­ments that are too com­plex for her to eval­u­ate in their en­tirety, by show­ing her what she “would have be­lieved” about a num­ber of in­ter­me­di­ate claims, if she had both­ered to check. Each of these in­ter­me­di­ate claims might it­self be too com­pli­cated for Judy to eval­u­ate di­rectly—if Judy needed to eval­u­ate it, she would use the same trick again.

Bet­ting with attention

If Alice and Bob are bet­ting about many claims over the course of a long ar­gu­ment, we can re­place dol­lars by “at­ten­tion points,” which rep­re­sent Judy’s time think­ing about the ar­gu­ment (per­haps 1 at­ten­tion point = 1 minute of Judy’s time). Judy con­sid­ers an ar­bi­trage op­por­tu­nity “good enough” if the profit is more than the time re­quired to eval­u­ate the ar­gu­ment. The ini­tial al­lo­ca­tion of at­ten­tion points re­flects the to­tal amount of time Judy is will­ing to spend think­ing about the is­sue. If some­one runs out of at­ten­tion points, then they can no longer make any claims or use up any of Judy’s time.

This re­moves some of the prob­lems of us­ing dol­lars, and in­tro­duces a new set of prob­lems. The mod­ified sys­tem works best when the to­tal stock of at­ten­tion points is large com­pared to the num­ber at stake for each claim. In­tu­itively, if there are N com­pa­rable claims to wa­ger about, the stakes of each should not be more than a 1/​sqrt(N) of the to­tal at­ten­tion pool — or else ran­dom chance will be too large a fac­tor. This re­quire­ment still al­lows a large gap be­tween the time ac­tu­ally re­quired to eval­u­ate an ar­gu­ment (i.e. the ini­tial bankroll of at­ten­tion points) and the to­tal time that would have been re­quired to eval­u­ate all of the claims made in the ar­gu­ment (the to­tal stake of all of the bets). If each claim is it­self sup­ported by a re­cur­sive ar­gu­ment, this gap can grow ex­po­nen­tially.

Talk­ing it out

If Alice and Bob dis­agree about a claim (rather, if they dis­agree about Judy’s prob­a­bil­ity of ac­cept­ing the claim) then they can have an in­cen­tive to “talk it out” rather than bring­ing the dis­pute to Judy.

For ex­am­ple, sup­pose that Alice and Bob each think they have a 60% chance of win­ning an ar­gu­ment. If they bring in Judy to ar­bi­trate, both of them will get un­fa­vor­able odds. Be­cause the sur­plus from the dis­agree­ment is go­ing to Judy, both par­ties would be happy enough to see their coun­ter­party wise up (and of course both would be happy to wise up them­selves). This cre­ates room for pos­i­tive sum trades.

Rather than bring­ing in Judy to ar­bi­trate their dis­agree­ment, they could do fur­ther re­search, con­sult an ex­pert, pay Judy at­ten­tion points to hear her opinion on a key is­sue, talk to Judy’s friends—what­ever is the most cost-effec­tive way to re­solve the dis­agree­ment. Once they have this in­for­ma­tion, their bet­ting odds can re­flect it.

An example

Sup­pose that Alice and Bob are ar­gu­ing about how many trees are in North Amer­ica; both are ex­perts on the topic, but Judy knows noth­ing about it.

The eas­iest case is if Alice and Bob know all of the rele­vant facts, but one of them wants to mis­lead Judy. In this case, the truth will quickly pre­vail. Alice and Bob can be­gin by break­ing down the is­sue into “How many trees are in each of Canada, the US, and Mex­ico?” If Alice or Bob lie about any of these es­ti­mates, they will quickly be cor­rected. Nei­ther should be will­ing to bet much for a lie, but if they do, the same thing will hap­pen re­cur­sively — the ques­tion will be bro­ken down into “how many trees are east and west of the Mis­sis­sippi?” and so on, un­til they dis­agree about how many trees are on a par­tic­u­lar hill—a straight­for­ward dis­agree­ment to re­solve.

In re­al­ity, Alice and Bob will have differ­ent in­for­ma­tion about each of these es­ti­mates (and ge­og­ra­phy prob­a­bly won’t be the eas­iest way to break things down — in­stead they might com­bine the differ­ent con­sid­er­a­tions that in­form their views, the best guess sug­gested by differ­ent method­olo­gies, ap­prox­i­mate counts of each type of tree on each type of land, and so on). If Alice and Bob can reach a ra­tio­nal con­sen­sus on a given es­ti­mate, then Judy can use that con­sen­sus to in­form her own view. If Alice and Bob can’t re­solve their dis­agree­ment, then we’re back to the pre­vi­ous case. The only differ­ence is that now Alice and Bob have prob­a­bil­is­tic dis­agree­ments: if Alice dis­agrees with Bob she doesn’t ex­pect to win the en­su­ing ar­gu­ment with 100% prob­a­bil­ity, merely with a high prob­a­bil­ity.

Odds and ends

This writeup leaves many de­tails un­der­speci­fied. In par­tic­u­lar, how does Judy es­ti­mate how long it will take her to ar­bi­trate a dis­agree­ment? This can be han­dled in sev­eral ways: by hav­ing Judy guess, by hav­ing Alice and Bob bet on the length of time un­til Judy reaches a con­clu­sion, by hav­ing them make bets of the form “Alice will agree with me with Z effort,” or so on. I don’t know what would work best.

De­spite my use of the word “re­cur­sion,” the es­ti­mate for “time to set­tle an ar­gu­ment” (which Judy uses to de­cide when the stakes are high enough to step in and re­solve a dis­agree­ment) prob­a­bly shouldn’t in­clude the time re­quired to set­tle sub-ar­gu­ments, since Judy is be­ing paid sep­a­rately for ar­bi­trat­ing each of those. The struc­ture of the ar­gu­ments and sub-ar­gu­ments need not be a tree.

This is a sim­ple enough pro­posal that it can be re­al­is­ti­cally im­ple­mented, so even­tu­ally we’ll hope­fully see how it works and why it fails.

I ex­pect this will work best if Alice and Bob of­ten ar­gue about similar top­ics.

This scheme was mo­ti­vated by a par­tic­u­lar ex­otic ap­pli­ca­tion: del­e­gat­ing de­ci­sion-mak­ing to very in­tel­li­gent ma­chines. In that set­ting the goal is to scale to very com­plex dis­agree­ments, with very in­tel­li­gent ar­guers, while be­ing very effi­cient with the over­seer’s time (and more cav­a­lier with the ar­guers’ time).

Of ar­gu­ments and wa­gers was origi­nally pub­lished in AI Align­ment on Medium, where peo­ple are con­tin­u­ing the con­ver­sa­tion by high­light­ing and re­spond­ing to this story.