# abramdemski(Abram Demski)

Karma: 10,548
• That seems like a sen­si­ble way to set up the no-trade situ­a­tion. Pre­sum­ably the con­nec­tion to trade is via some the­o­rem that trade will re­sult in pareto-op­ti­mal situ­a­tions, there­fore mak­ing com­par­a­tive ad­van­tage ap­pli­ca­ble.

But I still won­der what the ex­act the­o­rem is.

Then if you want to de­scribe the pareto fron­tier that max­i­mizes the amount of goods pro­duced, it in­volves each per­son pro­duc­ing a good where they have a fa­vor­able ra­tio of how much of that good they can pro­duce vs. how much of other goods-be­ing-pro­duced they can pro­duce.

What do you mean by “fa­vor­able”? Is there some thresh­hold?

What do you mean by “in­volves each per­son pro­duc­ing”? Does it mean that they’ll ex­clu­sively pro­duce such goods? Or does it mean they’ll pro­duce at least some of such goods?

• Cor­rec­tion: I now see that my for­mu­la­tion turns the ques­tion of com­plete­ness into a ques­tion of tran­si­tivity of in­differ­ence. An “in­com­plete” prefer­ence re­la­tion should not be un­der­stood as one in which al­lows strict prefer­ences to go in both di­rec­tions (which is what I in­ter­pret them as, above) but rather, a prefer­ence re­la­tion in which the re­la­tion (and hence the re­la­tion) is not tran­si­tive.

In this case, we can dis­t­in­guish be­tween ~ and “gaps”, IE, in­com­pa­rable A and B. ~ might be tran­si­tive, but this doesn’t bridge across the gaps. So we might have a prefer­ence chain A>B>C and a chain X>Y>Z, but not have any way to com­pare be­tween the two chains.

In my for­mu­la­tion, which lumps to­gether in­differ­ence and gaps, we can’t have this two-chain situ­a­tion. If A~X, then we must have A>Y, since X>Y, by tran­si­tivity of .

So what would be a com­plete­ness vi­o­la­tion in the wikipe­dia for­mu­la­tion be­comes a tran­si­tivity vi­o­la­tion in mine.

But no­tice that I never ar­gued for the tran­si­tivity of ~ or in my com­ment; I only ar­gued for the tran­si­tivity of >.

I don’t think a money-pump ar­gu­ment can be offered for tran­si­tivity here.

How­ever, I took a look at the pa­per by Au­mann which you cited, and I’m fairly happy with the gen­er­al­iza­tion of VNM therein! Drop­ping unique­ness does not seem like a big cost. This seems like more of an ex­am­ple of John Went­worth’s “boiler­plate” point, rather than a coun­terex­am­ple.

• This was helpful, but I’m still some­what con­fused. Con­spicu­ously ab­sent from your post is an out­right state­ment of what com­par­a­tive ad­van­tage is—par­tic­u­larly, what the con­cept and the­o­rem is sup­posed to be in the gen­eral case with more than two re­sources and more than two agents.

The ques­tion is: who and where do I or­der to grow ba­nanas, and who and where do I or­der to build things? To max­i­mize con­struc­tion, I will want to or­der peo­ple with the largest com­par­a­tive ad­van­tage in ba­nana-grow­ing to spe­cial­ize in ba­nana-grow­ing, and I will want to or­der those ba­nanas to be grown on the is­lands with the largest com­par­a­tive ad­van­tage in ba­nana-grow­ing. (In fact, this is not just rele­vant to max­i­miza­tion of con­struc­tion—it ap­plies to pareto-op­ti­mal pro­duc­tion in gen­eral.)

Could you elab­o­rate on this by pro­vid­ing the gen­eral state­ment rather than only the ex­am­ple?

• Com­par­a­tive ad­van­tage is of­ten used as an ar­gu­ment for free trade. Dynomight’s post seems to provide a suffi­cient coun­ter­ar­gu­ment to this, in its ex­am­ple illus­trat­ing how with more than 2 play­ers, open­ing up a trade route may not be a Pareto im­prove­ment (may not be a good thing for ev­ery­one).

• Com­par­a­tive ad­van­tage is some­times used in ca­reer ad­vice, EG, “find your com­par­a­tive ad­van­tage”. This is the case I fo­cus on in the com­ment I linked to illus­trat­ing my con­fu­sion. What ad­vice is ac­tu­ally offered? Are agents sup­posed to pro­duce and sell things which they have a com­par­a­tive ad­van­tage in? Not so much. It seems that ad­vice com­ing from the con­cept is ac­tu­ally ex­tremely weak in the case of a mar­ket with more than two goods.

Your post gave me a third po­ten­tial ap­pli­ca­tion, namely, a crite­rion for when trade may oc­cur at all. This ex­panded my un­der­stand­ing of the con­cept con­sid­er­ably. It’s clear that where no com­par­a­tive ad­van­tage ex­ists, no trade makes sense. A coun­try that’s bad at pro­duc­ing ev­ery­thing might want to buy stuff from a coun­try that’s just 10x bet­ter, but to do so they’d at least need a com­par­a­tive ad­van­tage in pro­duc­ing money (which doesn’t re­ally make sense; money isn’t some­thing you pro­duce). (Or putting it a differ­ent way: their money would soon be used up.)

But then you ap­ply the con­cept of com­par­a­tive ad­van­tage to a case where there isn’t any trade at all. What would you give as your gen­eral state­ment of the con­cept and the the­o­rem you’re ap­ply­ing?

• I hap­pened upon this old thread, and found the dis­cus­sion in­trigu­ing. Thanks for post­ing these refer­ences! Un­less I’m mis­taken, it sounds like you’ve dis­cussed this topic a lot on LW but have never made a big post de­tailing your whole per­spec­tive. Maybe that would be use­ful! At least I per­son­ally find dis­cus­sions of ap­pli­ca­bil­ity/​gen­er­al­iz­abil­ity of VNM and other ra­tio­nal­ity ax­ioms quite in­ter­est­ing.

In­deed, I think I re­cently ran into an­other old com­ment of yours in which you made a re­mark about how Dutch Books only hold for re­peated games? I don’t re­call the de­tails now.

I have some com­ments on the pre­ced­ing dis­cus­sion. You said:

It would be rather au­da­cious to claim that this is true for each of the four ax­ioms. For in­stance, do please demon­strate how you would Dutch-book an agent that does not con­form to the com­plete­ness ax­iom!

For me, it seems that tran­si­tivity and com­plete­ness are on an equally jus­tified foot­ing, based on the clas­sic money-pump ar­gu­ment.

Just to keep things clear, here is how I think about the de­tails. There are out­comes. Then there are gam­bles, which we will define re­cur­sively. An out­come counts as a gam­ble for the sake of the base case of our re­cur­sion. For gam­bles A and B, pA+(1-p)B also counts as a gam­ble, where p is a real num­ber in the range [0,1].

Now we have a prefer­ence re­la­tion > on our gam­bles. I un­der­stand its nega­tion to be ; say­ing is the same thing as . The in­differ­ence re­la­tion, is just the same thing as .

This is differ­ent than the de­vel­op­ment on wikipe­dia, where ~ is defined sep­a­rately. But I think it makes more sense to define > and then define ~ from that. A>B can be un­der­stood as “definitely choose A when given the choice be­tween A and B”. ~ then rep­re­sents in­differ­ence as well as un­cer­tainty like the kind you de­scribe when you dis­cuss bounded ra­tio­nal­ity.

From this start­ing point, it’s clear that ei­ther A<B, or B<A, or A~B. This is just a way of say­ing “ei­ther A<B or B<A or nei­ther”. What’s im­por­tant about the com­plete­ness ax­iom is the as­sump­tion that ex­actly one of these hold; this tells us that we can­not have both A<B and B<A.

But this is prac­ti­cally the same as cir­cu­lar prefer­ences A<B<C<A, which tran­si­tivity out­laws. It’s just a cir­cle of length 2.

The clas­sic money-pump against cir­cu­lar­ity is that if we have cir­cu­lar prefer­ences, some­one can charge us for mak­ing a round trip around the cir­cle, swap­ping A for B for C for A again. They leave us in the same po­si­tion we started, less some money. They can then do this again and again, “pump­ing” all the money out of us.

Per­son­ally I find the this ar­gu­ment ex­tremely meta­phys­i­cally weird, for sev­eral rea­sons.

• The money-pumper must be God, to be able to swap ar­bi­trary A for B, and B for C, etc.

• But fur­ther­more, the agent must not un­der­stand the true na­ture of the money-pumper. When God asks about swap­ping A for B, the agent thinks it’ll get B in the end, and makes the de­ci­sion ac­cord­ingly. Yet, God pro­ceeds to then ask a new ques­tion, offer­ing to swap B for C. So God doesn’t ac­tu­ally put the agent in uni­verse B; rather, God puts the agent in “B+God”, a uni­verse with the pos­si­bil­ity of B, but also a new offer from God, namely, to move on to C. So God is ac­tu­ally fool­ing the agent, mak­ing an offer of B but re­ally giv­ing the agent some­thing differ­ent than B. Bad de­ci­sion-mak­ing should not count against the agent if the agent was mis­lead in such a man­ner!

• It’s also pretty weird that we can end up “in the same situ­a­tion, but with less money”. If the out­comes A,B,C were cap­tur­ing ev­ery­thing about the situ­a­tion, they’d in­clude how much money we had!

I have similar (but less se­vere) ob­jec­tions to Dutch-book ar­gu­ments.

How­ever, I also find the ar­gu­ment ex­tremely prac­ti­cally ap­pli­ca­ble, so much so that I can ex­cuse the meta­phys­i­cal weird­ness. I have come to think of Dutch-book and money-pump ar­gu­ments as illus­tra­tive of im­por­tant types of (in)con­sis­tency rather than literal ar­gu­ments.

OK, why do I find money-pumps prac­ti­cal?

Sim­ply put, if I have a loop in my prefer­ences, then I will waste a lot of time de­liber­at­ing. The real money-pump isn’t some­one tak­ing ad­van­tage of me, but rather, time it­self pass­ing.

What I find is that I get stuck de­liber­at­ing un­til I can find a way to get rid of the loop. Or, if I “just choose ran­domly”, I’m stuck with a yucky dis­satis­fied feel­ing (I have re­gret, be­cause I see an­other op­tion as bet­ter than the one I chose).

This is equally true of three-choice loops and two-choice loops. So, tran­si­tivity and com­plete­ness seem equally well-jus­tified to me.

Stu­art Arm­strong ar­gues that there is a weak money pump for the in­de­pen­dence ax­iom. I made a very tech­ni­cal post (not all of which seems to ren­der cor­rectly on LessWrong :/​) jus­tify­ing as much as I could with money-pump/​dutch-book ar­gu­ments, and similarly got ev­ery­thing ex­cept con­ti­nu­ity.

I re­gard con­ti­nu­ity as not very the­o­ret­i­cally im­por­tant, but highly ap­pli­ca­ble in prac­tice. IE, I think the pure the­ory of ra­tio­nal­ity should ex­clude con­ti­nu­ity, but a re­al­is­tic agent will usu­ally have con­tin­u­ous val­ues. The rea­son for this is again be­cause of de­liber­a­tion time.

If we drop con­ti­nu­ity, we get a ver­sion of util­ity the­ory with in­finite and in­finites­i­mal val­ues. This is perfectly fine, has the ad­van­tage of be­ing more gen­eral, and is in some sense more el­e­gant. To refer­ence the OP, con­ti­nu­ity is definitely just boiler­plate; we get a nice gen­er­al­iza­tion if we want to drop it.

How­ever, a real agent will ig­nore its own in­finites­i­mal prefer­ences, be­cause it’s not worth spend­ing time think­ing about that. In­deed, it will al­most always just think about the largest in­finity in its prefer­ences. This is es­pe­cially true if we as­sume that the agent places pos­i­tive prob­a­bil­ity on a re­ally broad class of things, which again seems true of ca­pa­ble agents in prac­tice. (IE, if you have in­fini­ties in your val­ues, and a broad prob­a­bil­ity dis­tri­bu­tion, you’ll be Pas­cal-mugged—you’ll only think of the in­finite pay­offs, ne­glect­ing finite pay­offs.)

So all of the ax­ioms ex­cept in­de­pen­dence have what ap­pear to me to be rather prac­ti­cal jus­tifi­ca­tions, and in­de­pen­dence has a weak money-pump jus­tifi­ca­tion (which may or may not trans­late to any­thing prac­ti­cal).

• I once had a dis­cus­sion with Scott G and Eli Tyre about this. We de­cided that the “real thing” was ba­si­cally where you should end up in the com­pli­cated worker/​job op­ti­miza­tion prob­lem, and there were more or less two ways to try and ap­prox­i­mate it:

1. Sup­pos­ing ev­ery­one else has already cho­sen their op­ti­mal spot, what still needs do­ing? What can I best con­tribute? This is sorta easy, be­cause you just look around at what needs do­ing, com­bine this with what you know about how ca­pa­ble you are at con­tribut­ing, and you get an es­ti­mate of how much you’d con­tribute in each place. Then you go to the place with the high­est num­ber. [mod­ulo gut feel­ings, in­trin­sic mo­ti­va­tion, etc]

2. Sup­pos­ing you choose first, how could ev­ery­one else move around you to cre­ate an op­ti­mal con­figu­ra­tion? You then go do the thing which im­plies the best con­figu­ra­tion. This seems much harder, but might be nec­es­sary for peo­ple who provide a lot of value (and there­fore what they do has a big in­fluence on what other peo­ple should do), par­tic­u­larly in small teams where a near-op­ti­mal re­ac­tion to your choice is fea­si­ble.

• OK. It seems there are re­sults for more than 2 goods, but the re­sults are quite weak:

Thus, if both rel­a­tive prices are be­low the rel­a­tive prices in au­tarky, we can rule out the pos­si­bil­ity that both goods 1 and 2 will be im­ported—but we can­not rule out the pos­si­bil­ity that one of them will be im­ported. In other words, once we leave the two-good case, we can­not es­tab­lish de­tailed pre­dic­tive re­la­tions say­ing that if the rel­a­tive price of a traded good ex­ceeds the rel­a­tive price of that good in au­tarky, then that good will be ex­ported by the coun­try in ques­tion. It fol­lows that any search for a strong the­o­rem along the lines of our first propo­si­tion ear­lier is bound to fail. The most one can hope for is a cor­re­la­tion be­tween the pat­tern of trade and differ­ences in au­tarky prices.

Dixit, Av­inash; Nor­man, Vic­tor (1980). The­ory of In­ter­na­tional Trade: A Dual, Gen­eral Equil­ibrium Ap­proach. Cam­bridge: Cam­bridge Univer­sity Press. p. 8

The im­plied ad­vice, as far as I un­der­stand it, is to check which good you have a com­par­a­tive ad­van­tage in pro­duc­ing, and offer that good to the mar­ket.

But sup­pose that there are a lot more goods and a lot more par­ti­ci­pants in the mar­ket.

For any one in­di­vi­d­ual, given fixed prices and sup­ply of ev­ery­one else, it sounds like we can for­mu­late the pro­duc­tion and trade strat­egy as a lin­ear pro­gram­ming prob­lem:

• We have some max­i­mum amount of time. That’s a lin­ear con­straint.

• We can al­lo­cate time to differ­ent tasks.

• The out­put of the tasks are as­sumed to be lin­ear in time.

• The tasks pro­duce differ­ent goods.

• Th­ese goods all have differ­ent prices on the mar­ket.

• We might have some ba­sic needs, like the 10 ba­nanas and 10 co­conuts. That’s a con­straint.

• We might also have de­sires, like not work­ing, or we might de­sire some goods. That’s our lin­ear pro­gram­ming ob­jec­tive.

OK. So we can solve this as a lin­ear pro­gram.

But… lin­ear pro­grams don’t have some nice closed-form solu­tion. The sim­plex al­gorithm can solve them effi­ciently in prac­tice, but that’s very differ­ent from an easy for­mula like “pro­duce the good with the high­est com­par­a­tive ad­van­tage”.

And that’s just solv­ing the prob­lem for one player, as­sum­ing the other play­ers have fixed strate­gies. More gen­er­ally, we have to an­ti­ci­pate the rest of the mar­ket as well. I don’t even know if that can be solved effi­ciently, via lin­ear pro­gram­ming or some other tech­nique.

Is “pro­duce where you have com­par­a­tive ad­van­tage” re­ally very use­ful ad­vice for more com­plex cases?

Wikipe­dia starts out de­scribing com­par­a­tive ad­van­tage as a law:

The law of com­par­a­tive ad­van­tage de­scribes how, un­der free trade, an agent will pro­duce more of and con­sume less of a good for which they have a com­par­a­tive ad­van­tage.[1]

But no pre­cise math­e­mat­i­cal law is ever stated, and the law is only jus­tified with ex­am­ples (speci­fi­cally, two-player, two-com­mod­ity ex­am­ples). Fur­ther­more, I only ever re­call see­ing com­par­a­tive ad­van­tage ex­plained with ex­am­ples, rather than be­ing stated as a the­o­rem. (Although this may be be­cause I never got past econ 101.)

This makes it hard to know what the claimed law even is, pre­cisely. “pro­duce more and con­sume less”? In com­par­i­son to what?

One spot on Wikipe­dia says:

Skep­tics of com­par­a­tive ad­van­tage have un­der­lined that its the­o­ret­i­cal im­pli­ca­tions hardly hold when ap­plied to in­di­vi­d­ual com­modi­ties or pairs of com­modi­ties in a world of mul­ti­ple com­modi­ties.

Although, with­out cita­tion, so I don’t know where to find the de­tails of these cri­tiques.

• I’m not sure I fol­low that it has to be lin­ear—I sus­pect higher-or­der polyno­mi­als will work just as well. Even if lin­ear, there are a very wide range of trans­for­ma­tion ma­tri­ces that can be rea­son­ably cho­sen, all of which are com­pat­i­ble with not block­ing Pareto im­prove­ments and still not agree­ing on most trade­offs.

Well, I haven’t ac­tu­ally given the ar­gu­ment that it has to be lin­ear. I’ve just as­serted that there is one, refer­enc­ing Harsanyi and com­plete class ar­gu­ments. There are a va­ri­ety of re­lated ar­gu­ments. And these ar­gu­ments have some as­sump­tions which I haven’t been em­pha­siz­ing in our dis­cus­sion.

Here’s a pretty strong ar­gu­ment (with cor­re­spond­ingly strong as­sump­tions).

1. Sup­pose each in­di­vi­d­ual is VNM-ra­tio­nal.

2. Sup­pose the so­cial choice func­tion is VNM-ra­tio­nal.

3. Sup­pose that we also can use mixed ac­tions, ran­dom­iz­ing in a way which is in­de­pen­dent of ev­ery­thing else.

4. Sup­pose that the so­cial choice func­tion has a strict prefer­ence for ev­ery Pareto im­prove­ment.

5. Also sup­pose that the so­cial choice func­tion is in­differ­ent be­tween two differ­ent ac­tions if ev­ery sin­gle in­di­vi­d­ual is in­differ­ent.

6. Also sup­pose the situ­a­tion gives a non­triv­ial choice with re­spect to ev­ery in­di­vi­d­ual; that is, no one is in­differ­ent be­tween all the op­tions.

By VNM, each in­di­vi­d­ual’s prefer­ences can be rep­re­sented by a util­ity func­tion, as can the prefer­ences of the so­cial choice func­tion.

Imag­ine ac­tions as points in prefer­ence-space, an n-di­men­sional space where n is the num­ber of in­di­vi­d­u­als.

By as­sump­tion #5, ac­tions which map to the same point in prefer­ence-space must be treated the same by the so­cial choice func­tion. So we can now imag­ine the so­cial choice func­tion as a map from R^n to R.

VNM on in­di­vi­d­u­als im­plies that the mixed ac­tion p * a1 + (1-p) * a2 is just the point p of the way on a line be­tween a1 and a2.

VNM im­plies that the value the so­cial choice func­tion places on mixed ac­tions is just a lin­ear mix­ture of the val­ues of pure ac­tions. But this means the so­cial choice func­tion can be seen as an af­fine func­tion from R^n to R. Of course since util­ity func­tions don’t mind ad­di­tive con­stants, we can sub­tract the value at the ori­gin to get a lin­ear func­tion.

But re­mem­ber that points in this space are just vec­tors of in­di­vi­d­ual’s util­ities for an ac­tion. So that means the so­cial choice func­tion can be rep­re­sented as a lin­ear func­tion of in­di­vi­d­ual’s util­ities.

So now we’ve got a lin­ear func­tion. But I haven’t used the pareto as­sump­tion yet! That as­sump­tion, to­gether with #6, im­plies that the lin­ear func­tion has to be in­creas­ing in ev­ery in­di­vi­d­ual’s util­ity func­tion.

Now I’m lost again. “you should have a prefer­ence over some­thing where you have no prefer­ence” is non­sense, isn’t it? Either the some­one in ques­tion has a util­ity func­tion which in­cludes terms for (their be­liefs about) other agents’ prefer­ences (that is, they have a so­cial choice func­tion as part of their prefer­ences), in which case the change will ALREADY BE pos­i­tive for their util­ity, or that’s already fac­tored in and that’s why it nets to neu­tral for the agent, and the ar­gu­ment is moot.

[...]

If you’re just say­ing “peo­ple don’t un­der­stand their own util­ity func­tions very well, and this is an in­tu­ition pump to help them see this as­pect”, that’s fine, but “the­o­rem” im­plies some­thing deeper than that.

In­deed, that’s what I’m say­ing. I’m try­ing to sep­a­rately ex­plain the for­mal ar­gu­ment, which as­sumes the so­cial choice func­tion (or in­di­vi­d­ual) is already on board with Pareto im­prove­ments, and the in­for­mal ar­gu­ment to try to get some­one to ac­cept some form of prefer­ence util­i­tar­i­anism, in which you might point out that Pareto im­prove­ments benefit oth­ers at no cost (a con­tra­dic­tory and pointless ar­gu­ment if the per­son already has fully con­sis­tent prefer­ences, but an ar­gu­ment which might re­al­is­ti­cally sway some­body from be­liev­ing that they can be in­differ­ent about a Pareto im­prove­ment to be­liev­ing that they have a strict prefer­ence in fa­vor of them).

But the in­for­mal ar­gu­ment re­lies on the for­mal ar­gu­ment.

• Maybe it’s bet­ter phrased as “a CIRL agent has a pos­i­tive in­cen­tive to al­low shut­down iff it’s un­cer­tain [or the hu­man has a pos­i­tive term for it be­ing shut off]”, in­stead of “a ma­chine” has a pos­i­tive in­cen­tive iff.

I would fur­ther char­i­ta­bly rewrite it as:

“In chap­ter 16, we an­a­lyze an in­cen­tive which a CIRL agent has to al­low it­self to be switched off. This in­cen­tive is pos­i­tive if and only if it is un­cer­tain about the hu­man ob­jec­tive.”

A CIRL agent should be ca­pa­ble of be­liev­ing that hu­mans ter­mi­nally value press­ing but­tons, in which case it might al­low it­self to be shut off de­spite be­ing 100% sure about val­ues. So it’s just the par­tic­u­lar in­cen­tive ex­am­ined that’s iff.

• Sure, but the the­o­rem he proves in the set­ting where he proves it prob­a­bly is if and only if. (I have not read the new edi­tion, so, not re­ally sure.)

It also seems to me like Stu­art Rus­sell en­dorses the if-and-only-if re­sult as what’s de­sir­able? I’ve heard him say things like “you want the AI to pre­vent its own shut­down when it’s suffi­ciently sure that it’s for the best”.

Of course that’s not tech­ni­cally the full if-and-only-if (it needs to both be cer­tain about util­ity and think pre­vent­ing shut­down is for the best), but it sug­gests to me that he doesn’t think we should add more shut­off in­cen­tives such as AUP.

Keep in mind that I have fairly lit­tle in­ter­ac­tion with him, and this is based off of only a few off-the-cuff com­ments dur­ing CHAI meet­ings.

My point here is just that it seems pretty plau­si­ble that he meant “if and only if”.

• No­tice how­ever that for Log­i­cal Coun­ter­fac­tual Mug­ging to be well defined, you need to define what Omega is do­ing when it is mak­ing its pre­dic­tion. In Coun­ter­fac­tu­als for Perfect Pre­dic­tors, I ex­plained that when deal­ing with perfect pre­dic­tors, of­ten the coun­ter­fac­tual would be un­defined. For ex­am­ple, in Parfit’s Hitch­hicker a perfect pre­dic­tor would never give a lift to some­one who never pays in town, so it isn’t im­me­di­ately clear that pre­dict­ing what such a per­son would do in town in­volves pre­dict­ing some­thing co­her­ent.

Another ap­proach is to change the ex­am­ple to re­move the ob­jec­tion.

The poker-like game at the end of De­ci­sion The­ory (I re­ally ti­tled that post sim­ply “de­ci­sion the­ory”? rather vague, past-me...) is iso­mor­phic to coun­ter­fac­tual mug­ging, but re­moves some dis­trac­tions, such as “how does Omega take the coun­ter­fac­tual”.

Alice re­ceives a High or Low card. Alice can re­veal the card to Bob. Bob then states a prob­a­bil­ity for Alice’s card be­ing Low. Bob’s in­cen­tives just en­courage him to re­port hon­est be­liefs. Alice loses .

When Alice gets a Low card, she can just re­veal it to Bob, and get the best pos­si­ble out­come. But this strat­egy means Bob will know if she has a high card, giv­ing her the worst pos­si­ble out­come in that case. In or­der to suc­cess­fully bluff, Alice has to some­times act like she has differ­ent cards than she has. And in­deed, the op­ti­mal strat­egy for Alice in this case is to never show her cards.

This ex­am­ple will get less ob­jec­tions from peo­ple, be­cause it is grounded in a very re­al­is­tic game. Play­ing poker well re­quires this kind of rea­son­ing. The pow­er­ful pre­dic­tor is re­placed with an­other player. We can still tech­ni­cally ask “how is the other player deal with un­defined coun­ter­fac­tu­als?”, but we can skip over that by just rea­son­ing about strate­gies in the usual game-the­o­retic way—if Alice’s strat­egy were to re­veal low cards, then Bob could always call high cards.

We can then in­sert log­i­cal un­cer­tainty by stipu­lat­ing that Alice gets her card pseu­do­ran­domly, but nei­ther Alice nor Bob can pre­dict the ran­dom num­ber gen­er­a­tor.

Not sure yet whether you can pull a similar trick with Coun­ter­fac­tual Pri­soner’s Dilemma.

== nit­picks ==

# Ap­ply­ing the Coun­ter­fac­tual Pri­soner’s Dilemma to Log­i­cal Uncertainty

Why isn’t the ti­tle “ap­ply­ing log­i­cal un­cer­tainty to the coun­ter­fac­tual pris­oner’s dilemma”? Or “A Log­i­cally Uncer­tain Ver­sion of Coun­ter­fac­tual Pri­soner’s Dilemma”? I don’t see how you’re ap­ply­ing CPD to LU.

The Coun­ter­fac­tual Pri­soner’s Dilemma is a sym­met­ric ver­sion of the origi­nal

Sym­met­ric? The origi­nal is already sym­met­ric. But “sym­met­ric” is a con­cept which ap­plies to multi-player games. Coun­ter­fac­tual PD makes PD into a one-player game. Pre­sum­ably you meant “a one-player ver­sion”?

where re­gard­less of whether the coin comes up heads or tails you are asked to pay $100 and you are then paid$10,000 if Omega pre­dicts that you would have paid if the coin had come up the other way. If you de­cide up­date­lesly you will always re­ceived $9900, while if you de­cide up­date­fully, then you will re­ceive$0.

This is only true if you use clas­si­cal CDT, yeah? Whereas EDT can get \$9900 in both cases, pro­vided it be­lieves in a suffi­cient cor­re­la­tion be­tween what it does upon see­ing heads vs tails.

So un­like Coun­ter­fac­tual Mug­ging, pre-com­mit­ting to pay en­sures a bet­ter out­come re­gard­less of how the coin flip turns out, sug­gest­ing that fo­cus­ing only on your par­tic­u­lar prob­a­bil­ity branch is mis­taken.

I don’t get what you meant by the last part of this sen­tence. Coun­ter­fac­tual Mug­ging already sug­gests that fo­cus­ing only on your par­tic­u­lar branch is mis­taken. If some­one bought that you should pay up in this prob­lem but not in coun­ter­fac­tual mug­ging, I ex­pect that per­son to say some­thing like “be­cause in this case that strat­egy is guaran­teed bet­ter even in this branch”—hence, they’re not nec­es­sar­ily con­vinced to look at other branches. So I don’t think this ex­am­ple nec­es­sar­ily ar­gues for look­ing at other branches.

Also, why is this posted as a ques­tion?

• If it’s cer­tain about the hu­man ob­jec­tive, then it would be cer­tain that it knows what’s best, so there would be no rea­son to let a hu­man turn it off. (Un­less hu­mans have a ba­sic prefer­ence to turn it off, in which case it could pre­fer to be shut off.)

• I thought that the end re­sult is that since any change would not be a pareto im­prove­ment the func­tion can’t recom­mend any change so it must be com­pletely am­biva­lent about ev­ery­thing thus is the con­stant func­tion of ev­ery op­tion be­ing of util­ity 0.

Pareto-op­ti­mal­ity says that if there is a mass mur­derer that wants to kill as many peo­ple as pos­si­ble then you should not do a choice that lessens the amount of peo­ple kil­led ie you should not op­pose the mass mur­derer.

Ah, I should have made more clear that it’s a one-way im­pli­ca­tion: if it’s a Pareto im­prove­ment, then the so­cial choice func­tion is sup­posed to pre­fer it. Not the other way around.

A so­cial choice func­tion meet­ing that min­i­mal re­quire­ment can still do lots of other things. So it could still op­pose a mass mur­derer, so long as mass-mur­der is not it­self a Pareto im­prove­ment.

• Me too! I’m hav­ing trou­ble see­ing how that ver­sion of the pareto-prefer­ence as­sump­tion isn’t already as­sum­ing what you’re try­ing to show, that there is a uni­ver­sally-us­able so­cial ag­gre­ga­tion func­tion.

I’m not sure what you meant by “uni­ver­sally us­able”, but I don’t re­ally ar­gue any­thing about ex­is­tence, only what it has to look like if it ex­ists. It’s easy enough to show ex­is­tence, though; just take some ar­bi­trary sum over util­ity func­tions.

Or maybe I mi­s­un­der­stand what you’re try­ing to show—are you claiming that there is a (or a fam­ily of) ag­gre­ga­tion func­tion that are priv­ileged and should be used for Utili­tar­ian/​Altru­is­tic pur­poses?

Yep, at least in some sense. (Not sure how “priv­ileged” they are in your eyes!) What the Harsanyi Utili­tar­i­anism The­o­rem shows is that lin­ear ag­gre­ga­tions are just such a dis­t­in­guished class.

And now we have to spec­ify which agent’s prefer­ences we’re talk­ing about when we say “sup­port”.

[...]

The as­sump­tion I missed was that there are peo­ple who claim that a change is = for them, but also they sup­port it. I think that’s a con­fus­ing use of “prefer­ences”.

That’s why, in the post, I moved to talk­ing about “a so­cial choice func­tion”—to avert that con­fu­sion.

So we have peo­ple, who are what we define Pareto-im­prove­ment over, and then we have the so­cial choice func­tion, which is what we sup­pose must > ev­ery Pareto im­prove­ment.

Then we prove that the so­cial choice func­tion must act like it prefers some weighted sum of the peo­ple’s util­ity func­tions.

But this re­ally is just to avert a con­fu­sion. If we get some­one to as­sent to both VNM and strict prefer­ence of Pareto im­prove­ments, then we can go back and say “by the way, the so­cial choice func­tion was se­cretly you” be­cause that per­son meets the con­di­tions of the ar­gu­ment.

There’s no con­tra­dic­tion be­cause we’re not se­cretly try­ing to sneak in a =/​> shift; the per­son has to already pre­fer for Pareto im­prove­ments to hap­pen.

If it’s > for the agent in ques­tion, they clearly sup­port it. If it’s =, they don’t op­pose it, but don’t nec­es­sar­ily sup­port it.

Right, so, if we’re ap­ply­ing this ar­gu­ment to a per­son rather than just some so­cial choice func­tion, then it has to be > in all cases.

If you imag­ine that you’re try­ing to use this ar­gu­ment to con­vince some­one to be util­i­tar­ian, this is the step where you’re like “if it doesn’t make any differ­ence to you, but it’s bet­ter for them, then wouldn’t you pre­fer it to hap­pen?”

Yes, it’s triv­ially true that if it’s = for them then it must not be >. But hu­mans aren’t perfectly re­flec­tively con­sis­tent. So, what this ar­gu­ment step is try­ing to do is en­gage with the per­son’s in­tu­itions about their prefer­ences. Do they pre­fer to make a move that’s (at worst) costless to them and which is benefi­cial to some­one else? If yes, then they can be en­gaged with the rest of the ar­gu­ment.

To put it a differ­ent way: yes, we can’t just as­sume that an agent strictly prefers for all Pareto-im­prove­ments to hap­pen. But, we also can’t just as­sume that they don’t, and dis­miss the ar­gu­ment on those grounds. That agent should figure out for it­self whether it has a strict prefer­ence in fa­vor of Pareto im­prove­ments.

When I say “Pareto op­ti­mal­ity is min-bar for agree­ment”, I’m mak­ing a dis­tinc­tion be­tween literal con­sen­sus, where all agents ac­tu­ally agree to a change, and as­sumed im­prove­ment, where an agent makes a unilat­eral (or pop­u­la­tion-sub­set) de­ci­sion, and jus­tifies it based on their preferred ag­gre­ga­tion func­tion. Pareto op­ti­mal­ity tells us some­thing about agree­ment. It tells us noth­ing about ap­pli­ca­bil­ity of any pos­si­ble ag­gre­ga­tion func­tion.

Ah, ok. I mean, that makes perfect sense to me and I agree. In this lan­guage, the idea of the Pareto as­sump­tion is that an ag­gre­ga­tion func­tion should at least pre­fer things which ev­ery­one agrees about, what­ever else it may do.

In my mind, we hit the same com­pa­ra­bil­ity prob­lem for Pareto vs non-Pareto changes. Pareto-op­ti­mal im­prove­ments, which re­quire zero in­ter­per­sonal util­ity com­par­i­sons (only the sign mat­ters, not the mag­ni­tude, of each af­fected en­tity’s prefer­ence), teach us noth­ing about ac­tual trade­offs, where a func­tion must weigh the mag­ni­tudes of mul­ti­ple en­tities’ prefer­ences against each other.

The point of the Harsanyi the­o­rem is sort of that they say sur­pris­ingly much. Par­tic­u­larly when cou­pled with a VNM ra­tio­nal­ity as­sump­tion.

# What Does “Sig­nal­ling” Mean?

16 Sep 2020 21:19 UTC
35 points
• You said:

That’s not what Pareto-op­ti­mal­ity as­serts. It only talks about >= for all par­ti­ci­pants in­di­vi­d­u­ally.

From wikipe­dia:

Given an ini­tial situ­a­tion, a Pareto im­prove­ment is a new situ­a­tion where some agents will gain, and no agents will lose.

So a pareto im­prove­ment is a move that is > for at least one agent, and >= for the rest.

If you’re mak­ing as­sump­tions about al­tru­ism, you should be clearer that it’s an ar­bi­trary ag­gre­ga­tion func­tion that is be­ing in­creased.

I stated that the setup is to con­sider a so­cial choice func­tion (a way of mak­ing de­ci­sions which would “re­spect ev­ery­one’s prefer­ences” in the sense of re­gard­ing pareto im­prove­ments as strict prefer­ences, ie, >-type prefer­ences).

Per­haps I didn’t make clear that the so­cial choice func­tion should re­gard Pareto im­prove­ments as strict prefer­ences. But this is the only way to en­sure that you pre­fer the Pareto im­prove­ment and not the op­po­site change (which only makes things worse).

And then, Pareto-op­ti­mal­ity is a red her­ring. I don’t know of any ag­gre­ga­tion func­tions that would change a 0 to a + for a Pareto-op­ti­mal change, and would not give a + to some non-Pareto-op­ti­mal changes, which vi­o­late other agents’ prefer­ences.

Ex­actly. That’s, like, ba­si­cally the point of the Harsanyi the­o­rem right there. If your so­cial choice func­tion re­spects Pareto op­ti­mal­ity and ra­tio­nal­ity, then it’s forced to also make some trade-offs—IE, give a + to some non-Pareto changes.

(Un­less you’re in a de­gen­er­ate case, EG, ev­ery­one already has the same prefer­ences.)

I feel as if you’re deny­ing my ar­gu­ment by… mak­ing my ar­gu­ment.

My pri­mary ob­jec­tion is that any given ag­gre­ga­tion func­tion is it­self merely a prefer­ence held by the eval­u­a­tor. There is no rea­son to be­lieve that there is a jus­tifi­able-to-as­sume-in-oth­ers or au­to­mat­i­cally-agree­able ag­gre­ga­tion func­tion.

I don’t be­lieve I ever said any­thing about jus­tify­ing it to oth­ers.

I think one pos­si­ble view is that ev­ery al­tru­ist could have their own per­sonal ag­gre­ga­tion func­tion.

There’s still a ques­tion of which ag­gre­ga­tion func­tion to choose, what prop­er­ties you might want it to have, etc.

But then, many peo­ple might find the same con­sid­er­a­tions per­sua­sive. So I see noth­ing against peo­ple work­ing to­gether to figure out what “the right ag­gre­ga­tion func­tion” is, ei­ther.

This may be the crux. I do not as­sent to that. I don’t even think it’s com­mon.

OK! So that’s just say­ing that you’re not in­ter­ested in the whole setup. That’s not con­trary to what I’m try­ing to say here—I’m just try­ing to say that if an agent satis­fies the min­i­mal al­tru­ism as­sump­tion of prefer­ring Pareto im­prove­ments, then all the rest.

If you’re not at all in­ter­ested in the util­i­tar­ian pro­ject, that’s fine, other peo­ple can be in­ter­ested.

Pareto im­prove­ments are fine, and some of them ac­tu­ally im­prove my situ­a­tion, so go for it! But in the wider sense, there are lots of non-Pareto changes that I’d pick over a Pareto sub­set of those changes.

Again, though, now it just seems like you’re stat­ing my ar­gu­ment.

Weren’t you just crit­i­ciz­ing the kind of ag­gre­ga­tion I dis­cussed for as­sent­ing to Pareto im­prove­ments but in­evitably as­sent­ing to non-Pareto-im­prove­ments as well?

Pareto is a min-bar for agree­ment, not an op­ti­mum for any ac­tual ag­gre­ga­tion func­tion.

My sec­tion on Pareto is liter­ally ti­tled “Pareto-Op­ti­mal­ity: The Min­i­mal Stan­dard”

I’m feel­ing a bit of “are you trol­ling me” here.

You’ve both de­nied and as­serted both the premises and the con­clu­sion of the ar­gu­ment.

All in the same sin­gle com­ment.

• I agree that this can cre­ate per­verse in­cen­tives in prac­tice, but that seems like the sort of thing that you should be han­dling as part of your de­ci­sion the­ory, not your util­ity func­tion.

I’m mainly wor­ried about the per­verse in­cen­tives part.

I rec­og­nize that there’s some weird level-cross­ing go­ing on here, where I’m do­ing some­thing like mix­ing up the de­ci­sion the­ory and the util­ity func­tion. But it seems to me like that’s just a re­flec­tion of the weird muddy place our val­ues come from?

You can think of hu­mans a lit­tle like self-mod­ify­ing AIs, but where the mod­ifi­ca­tion took place over evolu­tion­ary his­tory. The util­ity func­tion which we even­tu­ally ar­rived at was (sort of) the re­sult of a bar­gain­ing pro­cess be­tween ev­ery­one, and which took some ac­count­ing of things like ex­ploita­bil­ity con­cerns.

In terms of de­ci­sion the­ory, I of­ten think in terms of a gen­er­al­ized NicerBot: ex­tend ev­ery­one else the same cofrence-co­effi­cient they ex­tend to you, plus an ep­silon (to en­sure that two gen­er­al­ized NicerBots end up fully co­op­er­at­ing with each other). This is a pretty de­cent strat­egy for any game, gen­er­al­iz­ing from one of the best strate­gies for Pri­soner’s Dilemma. (Of course there is no “best strat­egy” in an ob­jec­tive sense.)

But a de­ci­sion the­ory like that does mix lev­els be­tween the de­ci­sion the­ory and the util­ity func­tion!

I feel like the solu­tion of hav­ing cofrences not count the other per­son’s cofrences just doesn’t re­spect peo­ple’s prefer­ences—when I care about the prefer­ences of some­body else, that in­cludes car­ing about the prefer­ences of the peo­ple they care about.

I to­tally agree with this point; I just don’t know how to bal­ance it against the other point.

A crux for me is the coal­i­tion metaphor for util­i­tar­i­anism. I think of util­i­tar­i­anism as sort of a nat­u­ral end­point of form­ing benefi­cial coal­i­tions, where you’ve built a coal­i­tion of all life.

If we imag­ine form­ing a coal­i­tion in­cre­men­tally, and imag­ine that the coal­i­tion sim­ply av­er­ages util­ity func­tions with its new mem­bers, then there’s an in­cen­tive to join the coal­i­tion as late as you can, so that your prefer­ences get the largest pos­si­ble rep­re­sen­ta­tion. (I know this isn’t the same prob­lem we’re talk­ing about, but I see it as analo­gous, and so a point in fa­vor of wor­ry­ing about this sort of thing.)

We can cor­rect that by do­ing 1/​n av­er­ag­ing: ev­ery time the coal­i­tion gains mem­bers, we make a fresh av­er­age of all mem­ber util­ity func­tions (us­ing some util­ity-func­tion nor­mal­iza­tion, of course), and ev­ery­body vol­un­tar­ily self-mod­ifies to have the new mixed util­ity func­tion.

But the prob­lem with this is, we end up pun­ish­ing agents for self-mod­ify­ing to care about us be­fore join­ing. (This is more closely analo­gous to the prob­lem we’re dis­cussing.) If they’ve already self-mod­ified to care about us more be­fore join­ing, then their origi­nal val­ues just get washed out even more when we re-av­er­age ev­ery­one.

So re­ally, the im­plicit as­sump­tion I’m mak­ing is that there’s an agent “be­fore” al­tru­ism, who “chose” to add in ev­ery­one’s util­ity func­tions. I’m try­ing to set up the rules to be fair to that agent, in an effort to re­ward agents for mak­ing “the al­tru­is­tic leap”.

• First off, I’m not try­ing to illus­trate the many-player game here. So imag­ine there’s just Alice and Bob. I agree that the many-player ver­sion is rele­vant, but I was just deal­ing with the com­plex­ities that arise from iter­a­tion.

Se­cond, yeah, ab­solutely: strate­gies in iter­ated games can be any func­tion of the his­tory. But that’s a re­ally com­pli­cated strat­egy space to try and draw. Essen­tially I’m show­ing you just a very high-level sum­mary, fo­cus­ing on fre­quency of co­op­er­a­tion as a salient fea­ture.

The idea is that fre­quency is some­thing each player can ob­serve about the other. Alice can im­ple­ment a Grim Trig­ger strat­egy to en­force any given fre­quency of co­op­er­a­tion from Bob. It needs to have some wig­gle room, to al­low chance fluc­tu­a­tions in fre­quency with­out pul­ling the Grim Trig­ger; but Alice can in­clude wig­gle room while en­forc­ing tight enough a guaran­tee that Bob is forced to co­op­er­ate with the de­sired fre­quency in the limit, and Alice runs only a small risk of spu­ri­ously Grim Trig­ger­ing.