rossry

• rossry 17 Mar 2019 1:53 UTC

  1 point

  in reply to: rossry's comment on: How can we re­spond to info-cas­cades? [Info-cas­cade se­ries]

  A re­lated idea in non-pun­ish­ment of “wrong” re­ports that have in­suffi­cient sup­port (again in the com­mon-prior/​pri­vate-info set­ting) comes from this pa­per [pdf] (pre­sented at the same con­fer­ence), which sug­gests col­lect­ing re­ports from all agents and as­sign­ing re­wards/​pun­ish­ments by as­sum­ing that agents’ re­ports rep­re­sent their pri­vate sig­nal, com­put­ing their pos­te­rior, and scor­ing this as­sumed pos­te­rior. Un­der the model as­sump­tions, this makes it an op­ti­mal strat­egy for agents to truly re­veal their pri­vate sig­nal to the mechanism, while al­low­ing the mechanism to col­lect non-cas­caded base data to make a de­ci­sion.

  In gen­eral, I feel like the aca­demic liter­a­ture on mar­ket de­sign /​ mechanism de­sign has a lot to say about ques­tions of this fla­vor.

• rossry 17 Mar 2019 1:34 UTC

  4 points

  on: How can we re­spond to info-cas­cades? [Info-cas­cade se­ries]

  Ab­stract: Con­sid­er­ing in­for­ma­tion cas­cades (both up­wards and down­wards) as a prob­lem of in­cen­tives, bet­ter in­cen­tive de­sign holds some promise. This aca­demic pa­per sug­gests a model in which mak­ing truth-find­ing re­wards con­tin­gent on reach­ing a cer­tain num­ber of votes pre­vents down-cas­cades, and where an in­formed (self-in­ter­ested) choice of pay­out odds and thresh­old can also pre­vent up-cas­cades in the limit of a large pop­u­la­tion of pre­dic­tors.

  1) cf. av­turchin from the ques­tion about dis­tri­bu­tion across fields, point­ing out that up-cas­cades and down-cas­cades are both rele­vant con­cerns, in many con­texts.

  2) Con­sider in­for­ma­tion cas­cades as re­lated to a prob­lem of in­cen­tives—in the com­ments of the Johni­chols post refer­enced in the for­mal­iza­tion ques­tion, mul­ti­ple com­men­ta­tors point out that the model fails if agents seek to ex­press their marginal opinion, rather than their true (pos­te­rior) be­lief. But in­cen­tives to be right do need to be built into a sys­tem that you’re try­ing to pump en­ergy into, so the ques­tion re­mains of whether a differ­ent in­cen­tive struc­ture could do bet­ter, while still en­courag­ing truth-find­ing.

  3) Up-Cas­caded Wis­dom of the Crowd (Cong and Xiao, work­ing pa­per) con­sid­ers the in­for­ma­tion-ag­gre­ga­tion prob­lem in terms of in­cen­tives, and con­sider the in­cen­tives at play in an all-or-noth­ing crowd­fund­ing model, like ven­ture cap­i­tal or Kick­starter (as­sum­ing that a ‘no’ vote is ir­re­vo­ca­ble like a ‘yes’ vote is) -- ‘yes’ vot­ers win if there is a crit­i­cal mass of other ‘yes’ vot­ers and the propo­si­tion re­solves to ‘yes’; they lose if there is a crit­i­cal mass and the propo­si­tion re­solves to ‘no’; they have 0 loss/​gain if ‘yes’ doesn’t reach a crit­i­cal mass; ‘no’ vot­ers are merely ab­stain­ing from vot­ing ‘yes’.

  Their main re­sult is that if the pay­ment of in­cen­tives is con­di­tioned on the propo­si­tion gain­ing a fixed num­ber of ‘yes’ votes, a pop­u­la­tion of sym­met­ric, com­mon-prior/​pri­vate-info agents will avoid down-cas­cades, as a sin­gle ‘yes’ vote that breaks a down-cas­cade will not be pe­nal­ized for be­ing wrong un­less some later agent in­ten­tion­ally votes ‘yes’ to put the vote over the ‘yes’ thresh­old. (An agent i with nega­tive pri­vate info still should vote no, be­cause if a later agent i’ puts the vote over the ‘yes’ thresh­old based in part on i’s false vote, then i ex­pects to lose on the truth-eval­u­a­tion, since they’ve backed ‘yes’ but be­lieve ‘no’.)

  A fur­ther re­sult from the same pa­per is that if the ac­tor pos­ing the propo­si­tion can set the pay­out odds and the thresh­old in re­sponse to the com­mon prior and known info-dis­tri­bu­tion, then a propo­si­tion-poser at­tempt­ing to min­i­mize down-cas­cades (per­haps be­cause they will cast the first ‘yes’ vote, and so can only hope to win if the vote re­solves to ‘yes’) will be in­cen­tivized to set odds and a thresh­old that co­in­ci­den­tally min­i­mize the chance of up-cas­cades. In the large-pop­u­la­tion limit, the num­ber of cas­cades un­der such an in­cen­tive de­sign goes to 0.

  4) I sus­pect (but will not here prove) that aug­ment­ing Cong and Xiao’s all-or-noth­ing “crowd­fund­ing for ‘yes’” de­sign with a par­allel “crowd­fund­ing for ‘no’” de­sign—i.e., ‘no’ vot­ers win (resp. lose) iff there is a crit­i­cal mass of ‘no’ vot­ers and the propo­si­tion re­solves ‘no’ (resp. ‘yes’) -- can fur­ther strengthen the defenses against up-cas­cades (by mak­ing it pos­si­ble to cast a more in­formed ‘no’ vote con­di­tioned on a later, more-in­formed agent de­cid­ing to put ‘no’ over the thresh­old).

• rossry 17 Mar 2019 1:30 UTC

  0 points

  on: How can we re­spond to info-cas­cades? [Info-cas­cade se­ries]

  [this an­swer was du­pli­cated when I mis­tak­enly copied my com­ment into an an­swer and then moved the com­ment to an an­swer.]

• rossry 17 Mar 2019 0:14 UTC

  1 point

  in reply to: avturchin's comment on: Distri­bu­tion of info-cas­cades across fields? [Info-cas­cade se­ries]

  Why stop at 2? Belief-space is large, and many is­sues ad­mit more than one (+/​-) bit of in­for­ma­tion to cas­cade.

• rossry 2 Mar 2019 2:03 UTC

  1 point

  in reply to: avturchin's comment on: Open Thread Fe­bru­ary 2019

  But how make small changes in the tra­jec­tory of a star? One idea is to im­pact the star with large comets. It is not difficult, as re­mote Oort cloud ob­jects (or wan­der­ing small planets, as they are not part of already es­tab­lished or­bital move­ment of the star) need only small per­tur­ba­tions to start fal­ling down on the cen­tral star, which could be done via nu­clear ex­plo­sions or smaller im­pacts by smaller as­treoids.

  I don’t think this works; con­ser­va­tion of mo­men­tum means that the im­pact is al­most fully coun­ter­acted by the grav­i­ta­tional pull that ac­cel­er­ated the comet to such speed (so that, in the end, the delta-v im­parted to the star is pre­cisely what you im­parted with your nu­clear ex­plo­sions or smaller as­ter­oids).

  Maybe a Shkadov thruster is what you want? (It’s slow go­ing, though; this ar­ti­cle sug­gests 60ly/​200My, ac­count­ing for ac­cel­er­a­tion.)

• rossry 23 Feb 2019 10:35 UTC

  1 point

  in reply to: TheWakalix's comment on: Blackmail

  Win­ner’s Curse doesn’t seem like the right effect to me—it seems more like an or­thog­o­nal­ity/​Good­hart effect, where op­ti­miz­ing for out­ra­geous­ness de­creases the fit­ness w/​r/​t so­cial welfare (on the mar­gin). It’s always in the black­mailer’s in­ter­est to make the out­ra­geous­ness greater, so they’re not (self­ishly) sad when they over­shoot.

• rossry 19 Feb 2019 2:11 UTC

  8 points

  in reply to: ozziegooen's comment on: Can We Place Trust in Post-AGI Fore­cast­ing Eval­u­a­tions?

  I’m still think­ing about what quan­ti­ta­tive es­ti­mates I’d stand be­hind. I think I’d be­lieve that a prize-based com­pet­i­tive pre­dic­tion sys­tem with all eval deferred un­til and con­di­tioned on AGI is <4% to add more than $1mln of value to [just pay some smart par­ti­ci­pants for their best-efforts opinions].

  (If I thought harder about cor­ner-cases, I think I could come up with a stronger state­ment.)

• rossry 18 Feb 2019 23:39 UTC

  3 points

  in reply to: ozziegooen's comment on: Can We Place Trust in Post-AGI Fore­cast­ing Eval­u­a­tions?

  If the rea­son your ques­tions won’t re­solve is that you are dead or that none of your money at all will be use­ful, I think things are a bit differ­ent.

  I’m con­fused; to restate the above, I think that a p% chance that your pre­dic­tions don’t mat­ter (for any rea­son: game rained out, you’re dead, your money isn’t use­ful) is (to first or­der) equiv­a­lent to a p% tax on in­vest­ment in mak­ing bet­ter pre­dic­tions. What do you think is differ­ent?

  one ma­jor ask is that the fore­cast­ers be­lieve the AGI will hap­pen in be­tween, which seems to me like an even big­ger issue

  Sure, that’s an is­sue, but I think that re­quiring par­ti­ci­pants to all as­sume short AGI timelines is tractable in a way that the de­layed/​im­prob­a­ble re­s­olu­tion is­sues are not.

  I can imag­ine a mar­ket with­out re­s­olu­tion is­sues that as­sumes par­ti­ci­pants all be­lieve short AGI timelines could sup­port 12 semi-pro­fes­sional traders sub­si­dized by in­ter­ested stake­hold­ers. I don’t be­lieve that a mar­ket with re­s­olu­tion is­sues as above can elicit se­ri­ous in­vest­ment in get­ting its an­swers right from half that many. (I rec­og­nize that I’m elid­ing my defi­ni­tion of “se­ri­ous in­vest­ment” here.)

• rossry 18 Feb 2019 14:39 UTC

  3 points

  in reply to: ozziegooen's comment on: Can We Place Trust in Post-AGI Fore­cast­ing Eval­u­a­tions?

  I un­der­stand Zvi’s points as be­ing rel­a­tively uni­ver­sal to sys­tems where you want to use re­wards to in­cen­tivize par­ti­ci­pants to work hard to get good an­swers.

  No mat­ter how the pay­outs work, a p% chance that your ques­tions don’t re­solve is (to first or­der) equiv­a­lent to a p% tax on in­vest­ment in mak­ing bet­ter pre­dic­tions, and a years-long tie-up kills iter­a­tive growth and se­lec­tion/​am­plifi­ca­tion cy­cles as well limit­ing the re­turn-on-in­vest­ment-in-gen­eral-pre­dic­tion-skill to a one-shot game. I don’t think these is­sues go away if you re­ward pre­dic­tions differ­ently, since they’re gen­eral fea­tures of the re­la­tion be­tween the up-front in­vest­ment in mak­ing bet­ter pre­dic­tions and the to-come po­ten­tial re­ward for do­ing so well.

  (A coun­ter­point I’ll en­ter­tain is Zvi’s caveat to “quick re­s­olu­tion”—which also caveats “prob­a­ble re­s­olu­tion”—that suffi­cient liquidity can sub­sti­tute for re­s­olu­tion. But boot­strap­ping that liquidity it­self seems like a Hard Prob­lem, so I’d need to fur­ther be con­vinced that it’s tractable here.)

• rossry 18 Feb 2019 13:15 UTC

  2 points

  on: Can We Place Trust in Post-AGI Fore­cast­ing Eval­u­a­tions?

  As­sum­ing that your AI timelines are well-ap­prox­i­mated by “likely more than three years”, Zvi’s post on pre­dic­tion mar­ket desider­ata sug­gests that post-AGI eval­u­a­tion is pretty dead-on-ar­rival for cre­at­ing liquid pre­dic­tion mar­kets. Even lay­ing aside the con­di­tional-on-AGI di­men­sion, the failures of “quick re­s­olu­tion” (years) and “prob­a­ble re­s­olu­tion” (~20%, by your num­bers) are crip­pling for the prospect of pro­fes­sion­als or ex­perts in­vest­ing se­ri­ous re­sources in mak­ing prof­itable pre­dic­tions.

• rossry 11 Feb 2019 6:23 UTC

  7 points

  on: Prob­a­bil­ity space has 2 metrics

  The spec­u­la­tive propo­si­tion that hu­mans might only be us­ing one met­ric rings true and is com­pel­lingly pre­sented.

  How­ever, I feel a bit click­baited by the ti­tle, which (to me) im­plies that prob­a­bil­ity-space has only two met­rics (which isn’t true, as the later propo­si­tion de­pends on). Maybe con­sider chang­ing it to “Prob­a­bil­ity space has mul­ti­ple met­rics”, to avoid con­fu­sion?

• rossry 8 Feb 2019 13:23 UTC

  9 points

  in reply to: ryan_b's comment on: Open Thread Fe­bru­ary 2019

  It varies a lot be­tween pa­pers (in my ex­pe­rience) and be­tween fields (I imag­ine), but sev­eral hours for a deep read­ing doesn’t seem out of line. To take an anec­dote, I was re­cently re-read­ing a pa­per (of com­pa­rable length, though in eco­nomics) that I’m plan­ning to pre­sent as a guest of a math­e­mat­ics read­ing group, and I prob­a­bly spent 4 hours on the re-read, be­fore start­ing on my slides and pre­sum­ably re-re-read­ing a bunch more.

  Graz­ing over sev­eral days (and/​or mul­ti­ple sep­a­rate read­ings) is also my usual prac­tice for a close read, fwiw.

• rossry 7 Feb 2019 9:29 UTC

  2 points

  in reply to: T3t's comment on: X-risks are a tragedies of the commons

  I ac­knowl­edge that there’s a dis­tinc­tion, but I fail to see how it’s im­por­tant. When you (shut up and) mul­ti­ply out the prob­a­bil­ities, the ex­pected per­sonal re­ward for putting in dis­pro­por­tionate re­sources is nega­tive, and the per­sonal-welfare-op­ti­miz­ing level of effort is lower than the so­cial-welfare-op­ti­miz­ing level.

  Why is it im­por­tant that that nega­tive ex­pected re­ward is made up of a pos­i­tive term plus a large nega­tive term (X-risk defense), in­stead of a nega­tive term plus a differ­ent nega­tive term (over­graz­ing)?

• rossry 30 Jan 2019 14:24 UTC

  1 point

  in reply to: dvasya's comment on: Book Tril­ogy Re­view: Re­mem­brance of Earth’s Past (The Three Body Prob­lem)

  Huh. I don’t think I ever heard some­one call this se­ries hard sci-fi where I could hear them; the most com­mon recom­men­da­tion was re­lated to its Chi­ne­se­ness, which, as Zvi claims, definitely de­liv­ers.

  And I’m not sure I’d take Niven as the archetype of truly hard sci-fi; have you ever tried Egan? Di­as­pora says sen­si­ble things about philos­o­phy of mind for em­u­lated, branch­ing AIs with a plot arc where the power laws of a 5+1-di­men­sional uni­verse be­come rele­vant, and Clock­work Rocket in­vents al­ter­nate laws of spe­cial rel­a­tivity in­ci­den­tally to a story in­volv­ing truly cre­ative alt-biol­ogy...

• rossry 21 Jan 2019 10:54 UTC

  3 points

  in reply to: Gurkenglas's comment on: How could shares in a megapro­ject re­turn value to share­hold­ers?

  In­vestors would pre­fer to in­vest in moon­shot megapro­jects over, like, in­fras­truc­ture megapro­jects.

  I don’t think this is true; as in cor­po­rate equity mar­kets, the prefer­ence should be re­spon­sive to price, and equity shares should trade over naive book value, but at a price where the marginal in­vestor is in­differ­ent to buy­ing more.

  In fact, in­so­far as the Miller-Modigli­ani as­sump­tions hold, any cap­i­tal­iza­tion of the pro­ject in terms of equity and debt should trade at the same to­tal price (and so, raise the same fund­ing), just with a differ­ent mix of the to­tal com­ing from the debt and the equity.

  should they be in­cen­tivized to abort?

  Clearly that would be ideal, but this scheme doesn’t make this is­sue worse than the sta­tus quo (and pro­vides the ad­van­tage that at least there’s some mar­ket-based met­ric to tell you that the pro­ject is go­ing off the rails).

• rossry 20 Jan 2019 4:57 UTC

  3 points

  in reply to: Gurkenglas's comment on: How could shares in a megapro­ject re­turn value to share­hold­ers?

  Proves too much; this is similarly true of share­hold­ers in any com­pany with a debt+equity cap­i­tal­iza­tion.

  It’s true that share­hold­ers’ in­cen­tives are not perfectly al­igned with cred­i­tors’. In pri­vate cor­po­ra­tions, this is han­dled with gov­er­nance prac­tices; for pub­lic megapro­jects it seems like even less of an is­sue.

• rossry 6 Jan 2019 12:31 UTC

  1 point

  in reply to: paulfchristiano's comment on: Does anti-malaria char­ity de­stroy the lo­cal anti-malaria in­dus­try?

  Does the “typ­i­cal ar­gu­ment for pro­tec­tion­ism” you cite claim that pro­tec­tion­ist policy in­creases the to­tal amount of lo­cal pro­duc­tion (cre­at­ing a steady-state trade sur­plus)? Or does it merely shift lo­cal pro­duc­tion from com­par­a­tively-ad­van­taged-to-pro­duce-lo­cally goods to com­par­a­tively-ad­van­taged-to-im­port ones (hope­fully ones with greater ex­ter­nal­ities to lo­cal pro­duc­tion)?

  There’s a rele­vance to the anti-aid ar­gu­ment: The first-or­der effect of steady-state aid is to cre­ate the effect of a steady-state trade deficit on pro­duc­tion with­out an effect on the cur­rent ac­count. If the to­tal ex­ter­nal­ities to lo­cal pro­duc­tion are larger than the con­sumer value of goods, then this effects a welfare trans­fer from the aid re­cip­i­ent to the aid sender.

  But the gen­eral-equil­ibrium effect is a shift of the re­cip­i­ent’s lo­cal pro­duc­tion away from the re­ceived goods to­wards the marginally-effi­cient goods. If the marginally-effi­cient goods have suffi­ciently greater ex­ter­nal­ities to lo­cal pro­duc­tion than the re­ceived goods, then this might be a net win. (Where “suffi­ciently” here de­pends on elas­tic­i­ties of pro­duc­tion and con­sump­tion.)

• rossry 24 Dec 2018 12:28 UTC

  3 points

  on: State Machines and the Strange Case of Mu­tat­ing API

  I’m not a net­work pro­gram­mer or lan­guage de­signer by trade, so I ex­pect to be miss­ing some­thing here, but I’ll give it a go to learn where I’m wrong.

  If you’re us­ing dis­tinct in­ter­faces for dis­tinct states (as it seems you are in your lat­ter ex­am­ples) and your com­piler is go­ing to en­force them, then shad­ow­ing vari­ables (as a lan­guage fea­ture) lets you unas­sign them as you go along. In a lan­guage I’m more fa­mil­iar with, which uses poly­mor­phic types rather than in­ter­faces:

  let socket = socks5_socket () in
  let socket = con­nect_unau­then­ti­cated socket ~proxy in
  let socket = con­nect_tcp socket ad­dress in
  en­joy socket ();
  

  with sig­na­tures:

  val sock5_socket : unit → [> `Closed of Socket.t]
  val con­nect_unau­then­ti­cated : [< `Closed of Socket.t] → proxy:Ad­dress.t → [> `Authen­ti­cated of Socket.t]
  val con­nect_tcp → [< `Authen­ti­cated of Socket.t] → Ad­dress.t → [> `Tcp_es­tab­lished of Socket.t]
  val en­joy : [< `Tcp_es­tab­lished of Socket.t] → unit → unit
  

  so that if you for­get the con­nect_unau­then­ti­cated line (big dan­ger of shad­ow­ing as you mu­tate), your com­piler will cor­rect you with a friendly but stern:

  This ex­pres­sion has type [> `Closed of Socket.t] but an ex­pres­sion was ex­pected of type [< `Authen­ti­cated of Socket.t].
  

  Of course, shad­ow­ing with­out type-safety sounds like a night­mare, and I’m not claiming that any lan­guage you want to use ac­tu­ally sup­ports it as syn­tax. But I oc­ca­sion­ally ap­pre­ci­ate it (given, of course, that I’ve got a type in­spec­tor read­ily key­bound, to query what state my socket is in at this par­tic­u­lar point).

• rossry 2 Oct 2018 14:34 UTC

  1 point

  on: Crit­i­cism Schedul­ing and Privacy

  I’ve been en­joy­ing this se­quence on pri­vacy, in no small part be­cause I dis­agree with some of its fun­da­men­tal premises so strongly. (Hope­fully, some­day soon I’ll make the time to pull to­gether a se­quence lay­ing out the grounds of my dis­agree­ment.)

  But with­out back­ing up that teaser (sorry), I’ll say that the brief men­tion of pri­vacy as a guard against fal­ling into in­fer­en­tial-dis­tance chasms seems very straight­for­wardly true in hind­sight (even to one skep­ti­cal of the guard-against-co­er­cive-con­trol an­gle), though I hadn’t been able to put it nearly so cleanly in my own thoughts. If you’ve got even a short post’s worth of con­tent on re­lated ideas in de­ploy­ing pri­vacy in deeply co­op­er­a­tive/​al­igned con­texts, I’d be es­pe­cially in­ter­ested to hear more thoughts in that di­rec­tion.

• rossry 19 Sep 2018 11:03 UTC

  2 points

  in reply to: romeostevensit's comment on: Defin­ing by opposites

  I like this fram­ing, es­pe­cially as it grace­fully han­dles the way that com­mu­ni­ca­tion isn’t like Guess Who—you have pri­ors that don’t look like “uniform over the fol­low­ing N pos­si­bil­ities”, your pay­offs for ac­tu­ally find­ing the an­swer might be non­con­stant de­pend on what the an­swer is, some re­source limi­ta­tion might make the maxi-p(win) strat­egy differ­ent from the op­tional dis­crim­i­na­tor—but once you start think­ing about how you’d win a game with those rules, strate­gies for smarter search sug­gest them­selves.