Zvi

• Zvi 22 Mar 2019 18:06 UTC

  7 points

  on: More re­al­is­tic tales of doom

  Is this fu­ture AI catas­tro­phe? Or is this just a de­scrip­tion of cur­rent events be­ing a gen­eral grad­ual col­lapse?

  This seems like what is hap­pen­ing now, and has been for a while. Ex­ist­ing ML sys­tems are clearly mak­ing Type-I prob­lems, already quite bad be­fore ML was a thing at all, much worse, to the ex­tent that I don’t see much abil­ity left of our civ­i­liza­tion to get any­thing that can’t be mea­sured in a short term feed­back loop—even in spaces like this, ap­peals to non-mea­surable or non-ex­plicit con­cerns are a near-im­pos­si­ble sell.

  Part II prob­lems are not yet com­ing from ML sys­tems, ex­actly, But we cer­tainly have al­gorithms that are effec­tively op­ti­mized and se­lected for the abil­ity to gain in­fluence; the al­gorithm gains in­fluence, which causes peo­ple to care about it and feed into it, caus­ing it to get more. If we get less di­rect in the metaphor we get the same thing with memet­ics, cul­ture, life strate­gies, cor­po­ra­tions, me­dia prop­er­ties and so on. The em­pha­sis on choos­ing win­ners, be­ing ‘on the right side of his­tory’, sup­port­ing those who are good at get­ting sup­port. OP notes that this hap­pens in non-ML situ­a­tions ex­plic­itly, and there’s no clear di­vid­ing line in any case.

  So if there is an­other the­ory that says, this has already hap­pened, what would one do next?

• Zvi 21 Mar 2019 14:04 UTC

  13 points

  on: Rest Days vs Re­cov­ery Days

  This dis­tinc­tion seems su­per valuable. What I find most in­ter­est­ing is that I would have la­beled what OP calls Rest as Re­cov­ery, and what it calls Re­cov­ery as Rest...

• Zvi 17 Mar 2019 17:20 UTC

  19 points

  in reply to: jessicata's comment on: Privacy

  I will at­tempt to clar­ify which of these things I ac­tu­ally be­lieve, as best I can, but do not ex­pect to be able to en­gage deeper into the thread.

  Im­pli­ca­tion: it’s bad for peo­ple to have much more in­for­ma­tion about other peo­ple (gen­er­ally), be­cause they would re­ward/​pun­ish them based on that info, and such re­ward­ing/​pun­ish­ing would be un­just. We cur­rently have scape­goat­ing, not jus­tice. (Note that a just sys­tem for re­ward­ing/​pun­ish­ing peo­ple will do no worse by hav­ing more in­for­ma­tion, and in par­tic­u­lar will do no worse than the null strat­egy of not re­ward­ing/​pun­ish­ing be­hav­ior based on cer­tain sub­sets of in­for­ma­tion)

  >> What I’m pri­mar­ily think­ing about here is that if one is go­ing to be re­warded/​pun­ished for what one does and thinks, one chooses what one does and thinks largely based upon that—you have a sig­nal­ing equil­ibria, as Wei Dei notes in his top-level com­ment. I be­lieve that this in many situ­a­tions is much worse, and will lead to mas­sive warp­ing of be­hav­ior in var­i­ous ways, even if those re­ward­ing/​pun­ish­ing were at­tempt­ing to be just (or even if they ac­tu­ally were just, if there wasn’t both com­mon knowl­edge of this and agree­ment on what is and isn’t just). The pri­mary con­cern isn’t whether some­one can ex­pect to be on-net pun­ished or re­warded, but on how be­hav­iors are changed.

  We need peo­ple there with us who won’t judge us. Who won’t use in­for­ma­tion against us.

  Im­pli­ca­tion: “judge” means to use in­for­ma­tion against some­one. Lin­guis­tic norms re­lated to the word “judg­ment” are thor­oughly cor­rupt enough that it’s worth ced­ing to these, lin­guis­ti­cally, and us­ing “judge” to mean (usu­ally un­justly!) us­ing in­for­ma­tion against peo­ple.

  >> Judge here means to re­act to in­for­ma­tion about some­one or their ac­tions or thoughts largely by up­dat­ing their view of the per­son—to not have to worry (as much, at least) about how things make you seem. The sec­ond sen­tence is a sec­ond claim, that we also need them not to use the in­for­ma­tion against us. I did not in­tend for the sec­ond to seem to be part of the first.

  A com­plete trans­for­ma­tion of our norms and norm prin­ci­ples, be­yond any­thing I can think of in a healthy his­tor­i­cal so­ciety, would be re­quired to even at­tempt full non-con­tex­tual strong en­force­ment of all re­main­ing norms.

  Im­pli­ca­tion (in the con­text of the over­all ar­gu­ment): a gen­eral re­duc­tion in pri­vacy wouldn’t lead to norms chang­ing or be­ing en­forced less strongly, it would lead to the same norms be­ing en­forced strongly. What­ever or who­ever de­cides which norms to en­force and how to en­force them is re­flex­ive rather than re­spon­sive to in­for­ma­tion. We live in a re­flex-based con­trol sys­tem.

  >> That doesn’t fol­low at all, and I’m con­fused why you think that it does. I’m say­ing that when I try to de­sign a norm sys­tem from scratch in or­der to be com­pat­i­ble with full non-con­tex­tual strong en­force­ment, I don’t see a way to do that. Not that things wouldn’t change—I’m sure they would.

  There are also known dilem­mas where any ac­tion taken would be a norm vi­o­la­tion of a sa­cred value.

  Im­pli­ca­tion: the sys­tem of norms is so cor­rupt that they will reg­u­larly put peo­ple in situ­a­tions where they are guaran­teed to be blamed, re­gard­less of their ac­tions. They won’t ad­just even when this is ob­vi­ous.

  >> The sys­tem of norms is messy, which is differ­ent than cor­rupt. Differ­ent norms con­flict. Yes, the sys­tem is cor­rupt, but that’s not re­quired for this to be a prob­lem. Con­crete ex­am­ple, cho­sen to hope­fully be not con­tro­ver­sial: Either turn away the ex­pen­sive sick child pa­tient, or risk bankrupt­ing the hos­pi­tal.

  Part of the job of mak­ing sausage is to al­low oth­ers not to see it. We still get re­li­ably dis­gusted when we see it.

  Im­pli­ca­tion: peo­ple ex­pect to lose value by know­ing some things. Prob­a­bly, it is be­cause they would ex­pect to be pun­ished due to it be­ing re­vealed they know these things (as in 1984). It is all an act, and it’s bet­ter not to know that in con­crete de­tail.

  >> Con­sider the literal ex­am­ple of sausage be­ing made. The cen­tral prob­lem is not that peo­ple are afraid the sausage mak­ers will strike back at them. The prob­lem is know­ing re­duces one’s abil­ity to en­joy sausage. Alter­na­tively, it might force one to stop en­joy­ing sausage.

  >> Another im­por­tant dy­namic is that we want to en­force a norm that X is bad and should be min­i­mized. But some­times X is nec­es­sary. So we’d rather not be too re­minded of the X that is nec­es­sary in some situ­a­tions where we know X must oc­cur, to avoid weak­en­ing the norm against X el­se­where, and be­cause we don’t want to pe­nal­ize those do­ing X where it is nec­es­sary as we would in­stinc­tively do if we learned too much de­tail.

  We con­stantly must claim ‘ev­ery­thing is go­ing to be all right’ or ‘ev­ery­thing is OK.’ That’s never true. Ever.

  Im­pli­ca­tion: the con­trol sys­tem de­mands op­ti­mistic sto­ries re­gard­less of the facts. There is some­thing or some­one forc­ing ev­ery­one to call the deer a horse un­der threat of pun­ish­ment, to main­tain a lie about how good things are, prob­a­bly to prop up an un­just regime.

  >> OK, this one’s just straight up cor­rect if you re­move the un­just regime part. Also, I am mar­ried with chil­dren.

  But these prob­lems, while im­proved, wouldn’t go away in a bet­ter or less hyp­o­crit­i­cal time. Norms are not a sys­tem that can have full well-speci­fied con­text de­pen­dence and be uni­ver­sally en­forced. That’s not how norms work.

  Im­pli­ca­tion: even in the most just pos­si­ble sys­tem of norms, it would be good to some­times vi­o­late those norms and hide the fact that you vi­o­lated them. (This seems in­cor­rect to me!)

  >> As I noted above, my model of norms is that they are even at their best messy ways of steer­ing be­hav­ior, and gen­er­ally just norms will in some cir­cum­stances push to­wards in­cor­rect ac­tion in ways the norm sys­tem would cause peo­ple to in­stinc­tively pun­ish. In such cases it is some­times cor­rect to vi­o­late the norm sys­tem, even if it is as just a sys­tem as one could hope for. And yes, in some of those cases, it would be good to hide that this was done, to avoid weak­en­ing norms (in­clud­ing by al­low­ing such cases not be pun­ished thus en­abling oth­er­wise stronger pun­ish­ment).

  If oth­ers know ex­actly what re­sources we have, they can and will take all of them.

  Im­pli­ca­tion: the bad guys won; we have rule by gang­sters, who aren’t con­cerned with sus­tain­able pro­duc­tion, and just take as much stuff as pos­si­ble in the short term. (This seems on the right track but par­tially false; the top marginal tax rate isn’t 100% [EDIT: see Ben’s com­ment, the ac­tual rate of ex­trac­tion is higher than the marginal tax rate])

  >> This is not pri­mar­ily a state­ment about The Pow­ers That Be or any par­tic­u­lar bad guys. I think this is in­her­ent in how peo­ple and poli­tics op­er­ate, and what hap­pens when one has many con­flict­ing would-be sa­cred val­ues. Of course, it is also a state­ment that when gang­sters do go af­ter you, it is im­por­tant that they not know, and there is always worry about po­ten­tial gang­sters on many lev­els whether or not they have won. Often the thing tak­ing all your re­sources is not a bad guy—e.g. ex­pen­sive med­i­cal treat­ments, or in-need fam­ily mem­bers, etc etc.

  If it is known how we re­spond to any given ac­tion, oth­ers find best re­sponses. They will re­spond to in­cen­tives. They ex­ploit ex­actly the amount we won’t re­tal­i­ate against. They feel safe.

  Im­pli­ca­tion: more gen­er­ally available in­for­ma­tion about what strate­gies peo­ple are us­ing helps “our” en­e­mies more than it helps “us”. (This seems false to me, for no­tions of “us” that I usu­ally use in strat­egy)

  >> Often on the mar­gin more in­for­ma­tion is helpful. But com­plete in­for­ma­tion is highly dan­ger­ous. And in my ex­pe­rience, most sys­tems in an in­ter­est­ing equil­ibrium where good things hap­pen sus­tain that partly with fuzzi­ness and un­cer­tainty—the idea that obey­ing the spirit of the rules and work­ing to­wards the goals and good things gets re­warded, other ac­tion gets pun­ished, in un­cer­tain ways. There need to be un­knowns in the sys­tem. Com­pe­ti­tions where ev­ery ac­tion by other agents is known are one-player games about op­ti­miza­tion and ex­ploita­tion.

  World peace, and do­ing any­thing at all that in­ter­acts with oth­ers, de­pends upon both strate­gic con­fi­dence in some places, and strate­gic am­bi­guity in oth­ers. We need to choose care­fully where to use which.

  Im­pli­ca­tion (in con­text): strate­gic am­bi­guity isn’t just nec­es­sary for us given our cir­cum­stances, it’s nec­es­sary in gen­eral, even if we lived in a surveillance state. (Huh?)

  >> Strate­gic am­bi­guity is nec­es­sary for the surveillance state so that peo­ple can’t do ev­ery­thing the state didn’t ex­plic­itly pun­ish/​for­bid. It is nec­es­sary for those liv­ing in the state, be­cause the risk of rev­olu­tion, the we’re-not-go­ing-to-take-it-any­more mo­ment, helps keep such places rel­a­tively liv­able ver­sus places where there is no such fear. It is im­por­tant that you don’t know ex­actly what will cause the peo­ple to rise up, or you’ll treat them as bad as won’t do that. And of course I was also talk­ing ex­plic­itly about things like ‘if you cross that bor­der we will be at war’ - there are times when you want to be 100% clear that there will be war (e.g. NATO) and oth­ers where you want to be 100% un­clear (e.g. Taiwan).

  To con­clude: if you think the ar­gu­ments in this post are sound (with the con­clu­sion be­ing that we shouldn’t dras­ti­cally re­duce pri­vacy in gen­eral), you also be­lieve the im­pli­ca­tions I just listed, un­less I (or you) mis­in­ter­preted some­thing.

  >> I hope this cleared things up. And of course, you can dis­agree with many, most or even all my ar­gu­ments and still not think we should rad­i­cally re­duce pri­vacy. Rad­i­cal changes don’t de­fault to be­ing a good idea if some­one gives in­valid ar­gu­ments against them!

• Zvi 17 Mar 2019 16:48 UTC

  2 points

  in reply to: Benquo's comment on: Privacy

  I replied to this com­ment on my blog (https://​thezvi.word­press.com/​2019/​03/​15/​pri­vacy/​#com­ment-3827)

Privacy

Spec­u­la­tions on Duo Standard

New York Res­tau­rants I Love: Pizza

• Zvi 7 Mar 2019 14:01 UTC

  2 points

  in reply to: Raemon's comment on: On The Lon­don Mulligan

  They would not change it back.

On The Lon­don Mulligan

• Zvi 20 Feb 2019 22:31 UTC

  2 points

  in reply to: T3t's comment on: Blackmail

  Yes. Long post is long and I didn’t want to throw out ar­gu­ments about par­tic­u­lar re­veals to show this—in par­tic­u­lar, we all think the cost of that should be zero in that case, and we all know it of­ten very much isn’t. And I didn’t want any­one to think I was rely­ing on that.

• Zvi 20 Feb 2019 22:29 UTC

  3 points

  in reply to: Duncan_Sabien's comment on: Blackmail

  I could have worded it to make this more clear but I think the point stands when clar­ified/​un­der­stood—the prox­i­mate goal of the black­mail re­lease is to be harm­ful, whereas the prox­i­mate goal of the gos­sip might or might not be.

  If oth­ers agree it is mis­lead­ing I will make this more ex­plicit.

• Zvi 20 Feb 2019 22:05 UTC

  4 points

  in reply to: sharps030's comment on: Blackmail

  Yes. It’s do­ing a few things, and that’s a lot of it.

Blackmail

New York Res­tau­rants I Love: Breakfast

Min­i­mize Use of Stan­dard In­ter­net Food Delivery

• Zvi 31 Jan 2019 17:39 UTC

  5 points

  in reply to: Gurkenglas's comment on: “AlphaS­tar: Mas­ter­ing the Real-Time Strat­egy Game StarCraft II”, Deep­Mind [won 10 of 11 games against hu­man pros]

  We’re not out. Cer­tainly we’re not out of games—e.g. Magic: The Gather­ing. Which would be a big leap.

  For ac­tual ba­sic board games, the one I want to see is Strat­ego, ac­tu­ally; the only is­sue is I don’t know if there are hu­mans who have both­ered to mas­ter it.

Book Tril­ogy Re­view: Re­mem­brance of Earth’s Past (The Three Body Prob­lem)

• Zvi 28 Jan 2019 13:52 UTC

  6 points

  in reply to: Vanessa Kosoy's comment on: An­nounce­ment: AI al­ign­ment prize round 4 winners

  Im­por­tant not to let the perfect be the en­emy of the good. There’s al­most cer­tainly a bet­ter way to find men­tors, but this would be far bet­ter than not do­ing any­thing, so I’d say that if you can’t find an ac­tion­able bet­ter op­tion within (let’s say) a month, you should just do it. Or just do it now and re­place with bet­ter method when you find one.

• Zvi 23 Jan 2019 14:17 UTC

  4 points

  in reply to: Raemon's comment on: Less Com­pe­ti­tion, More Mer­i­toc­racy?

  In that par­tic­u­lar case, I would have cho­sen differ­ent names that likely would have res­onated bet­ter, but felt it was im­por­tant not to change the pa­per’s cho­sen la­bels, even though they seemed not great. That might have been an er­ror.

  Their ex­pla­na­tion is that the ques­tion is, will the weaker can­di­dates con­cede that they are weaker than strong ones and let the strong ones all win, or will they challenge the stronger can­di­dates.

  Sugges­tions for other ways to make this more clear are ap­pre­ci­ated. I’d like to be able to write things like this in a way that peo­ple ac­tu­ally read and benefit from.

Game Anal­y­sis Index