paulfchristiano(Paul Christiano)

• paulfchristiano 8 Jul 2020 0:57 UTC

  LW: 4 AF: 2

  AF

  in reply to: ESRogs’s comment on: Learn­ing the prior

  Yeah, that’s my view.

• paulfchristiano 8 Jul 2020 0:56 UTC

  2 points

  in reply to: Gurkenglas’s comment on: Bet­ter pri­ors as a safety problem

  To the ex­tent that we in­stinci­tively be­lieve or dis­be­lieve this, it’s not for the right rea­sons—nat­u­ral se­lec­tion didn’t have any ev­i­dence to go on. At most, that in­stinct is a use­ful workaround for the ex­is­ten­tial dread glitch.

  To the ex­tent that we be­lieve this cor­rectly, it’s for the same rea­sons that we are able to do math and philos­o­phy cor­rectly (or at least more cor­rectly than chance :) de­spite nat­u­ral se­lec­tion not car­ing about it much. It’s the same rea­son that you can cor­rectly make ar­gu­ments like the one in your com­ment.

• paulfchristiano 8 Jul 2020 0:54 UTC

  4 points

  in reply to: Daniel Kokotajlo’s comment on: Bet­ter pri­ors as a safety problem

  I think that un­der the count­ing mea­sure, the vast ma­jor­ity of peo­ple like us are in simu­la­tions (ig­nor­ing sub­tleties with in­fini­ties that make that state­ment mean­ingless).

  I think that un­der a more re­al­is­tic mea­sure, it’s un­clear whether or not most peo­ple like us are in simu­la­tions.

  Those state­ments are un­re­lated to what I was get­ting at in the post though, which is more like: the simu­la­tion ar­gu­ment rests on us be­ing the kind of peo­ple who are likely to be simu­lated, we don’t think that ev­ery­one should be­lieve they are in a simu­la­tion be­cause the simu­la­tors are more likely to simu­late re­al­is­tic-look­ing wor­lds than re­al­ity is to pro­duce re­al­is­tic-look­ing wor­lds, that seems ab­surd.

  The whole thing is kind of a com­pli­cated mess and I wanted to skip it by brush­ing aside the simu­la­tion ar­gu­ment. Maybe should have just not men­tioned it at all given that the simu­la­tion ar­gu­ment makes such a mess of it. I don’t ex­pect to be able to get clar­ity in this thread ei­ther :)

  I think the rea­son why the hy­poth­e­sis that the world is a dream seems ab­surd has very lit­tle to do with like­li­hood ra­tios and ev­ery­thing to do with heuris­tics like “don’t trust things that sound like what a crazy per­son, drug-ad­dled per­son, or mys­tic would say.”

  It’s not the hy­poth­e­sis that’s ab­surd, it’s this par­tic­u­lar ar­gu­ment.

• paulfchristiano 8 Jul 2020 0:46 UTC

  LW: 6 AF: 3

  AF

  in reply to: Daniel Kokotajlo’s comment on: Learn­ing the prior

  That’s right—you still only get a bound on av­er­age qual­ity, and you need to do some­thing to cope with failures so rare they never ap­pear in train­ing (here’s a post re­view­ing my best guesses).

  But be­fore you weren’t even in the game, it wouldn’t mat­ter how well ad­ver­sar­ial train­ing worked be­cause you didn’t even have the knowl­edge to tell whether a given be­hav­ior is good or bad. You weren’t even get­ting the right be­hav­ior on av­er­age.

  (In the OP I think the claim “the gen­er­al­iza­tion is now com­ing en­tirely from hu­man be­liefs” is fine, I meant gen­er­al­iza­tion from one dis­tri­bu­tion to an­other. “Neu­ral nets are are fine” was sweep­ing these is­sues un­der the rug. Though note that in the real world the dis­tri­bu­tion will change from neu­ral net train­ing to de­ploy­ment, it’s just ex­actly the nor­mal ro­bust­ness prob­lem. The point of this post is just to get it down to only a ro­bust­ness prob­lem that you could solve with some kind of gen­er­al­iza­tion of ad­ver­sar­ial train­ing, the rea­son to set it up as in the OP was to make the is­sue more clear.)

• paulfchristiano 7 Jul 2020 0:59 UTC

  LW: 2 AF: 1

  AF

  in reply to: Charlie Steiner’s comment on: Learn­ing the prior

  So even when you talk about am­plify­ing f, you mean a cer­tain way of ex­tend­ing hu­man pre­dic­tions to more com­pli­cated back­ground in­for­ma­tion (e.g. via break­ing down Z into chunks and then us­ing copies of f that have been trained on smaller Z), not fine-tun­ing f to make bet­ter pre­dic­tions.

  That’s right, f is ei­ther imi­tat­ing a hu­man, or it’s trained by iter­ated am­plifi­ca­tion /​ de­bate—in any case the loss func­tion is defined by the hu­man. In no case is f op­ti­mized to make good pre­dic­tions about the un­der­ly­ing data.

  My im­pres­sion is that your hope is that if Z and f start out hu­man-like, then this is like spec­i­fy­ing the “pro­gram­ming lan­guage” of a uni­ver­sal prior, so that search for highly-pre­dic­tive Z, de­coded through f, will give some­thing that uses hu­man con­cepts in pre­dict­ing the world.

  Z should always be a hu­man-read­able (or am­plified-hu­man-read­able) la­tent; it will nec­es­sar­ily re­main hu­man-read­able be­cause it has no pur­pose other than to help a hu­man make pre­dic­tions. f is go­ing to re­main hu­man-like be­cause it’s pre­dict­ing what the hu­man would say (or what the hu­man-con­sult­ing-f would say etc.).

  The am­plified hu­man is like the pro­gram­ming lan­guage of the uni­ver­sal prior, Z is like the pro­gram that is cho­sen (or slightly more pre­cisely: Z is like a dis­tri­bu­tion over pro­grams, de­scribed in a hu­man-com­pre­hen­si­ble way) and f is an effi­cient dis­til­la­tion of the in­tractable ideal.

• paulfchristiano 6 Jul 2020 3:30 UTC

  LW: 4 AF: 2

  AF

  in reply to: Charlie Steiner’s comment on: Learn­ing the prior

  I’m not to­tally sure what ac­tu­ally dis­t­in­guishes f and Z, es­pe­cially once you start jointly op­ti­miz­ing them. If f in­cor­po­rates back­ground knowl­edge about the world, it can do bet­ter at pre­dic­tion tasks. Nor­mally we imag­ine f hav­ing many more pa­ram­e­ters than Z, and so be­ing more likely to squir­rel away ex­tra facts, but if Z is large then we might imag­ine it con­tain­ing com­pu­ta­tion­ally in­ter­est­ing ar­ti­facts like pat­terns that are de­signed to train a train­able f on back­ground knowl­edge in a way that doesn’t look much like hu­man-writ­ten text.

  f is just pre­dict­ing P(y|x, Z), it’s not try­ing to model D. So you don’t gain any­thing by putting facts about the data dis­tri­bu­tion in f—you have to put them in Z so that it changes P(y|x,Z).

  Now, maybe you can try to en­sure that Z is at least some­what textlike via mak­ing sure it’s not too easy for a dis­crim­i­na­tor to tell from hu­man text, or re­quiring it to play some func­tional role in a pure text gen­er­a­tor, or what­ever.

  The only thing Z does is get handed to the hu­man for com­put­ing P(y|x,Z).

• paulfchristiano 6 Jul 2020 3:27 UTC

  LW: 8 AF: 4

  AF

  in reply to: ESRogs’s comment on: Learn­ing the prior

  The differ­ence is that you can draw as many sam­ples as you want from D* and they are all iid. Neu­ral nets are fine in that regime.

• paulfchristiano 6 Jul 2020 0:10 UTC

  LW: 4 AF: 2

  AF

  in reply to: Donald Hobson’s comment on: AI Un­safety via Non-Zero-Sum Debate

  It seems even worse than any of that. If your AI wanted any­thing at all it might de­bate well in or­der to sur­vive. So if you are bank­ing on it sin­gle-mind­edly want­ing to win the de­bate then you were already in deep trou­ble.

Bet­ter pri­ors as a safety problem

Learn­ing the prior

• paulfchristiano 4 Jul 2020 0:31 UTC

  12 points

  in reply to: mutant-city’s comment on: Se­cond Wave Covid Deaths?

  I don’t un­der­stand how the sec­ond wave can’t be ex­plained by in­crease in test­ing. Be­fore only peo­ple who were sick were al­lowed to be tested, who cor­re­late more with hos­pi­tal vis­its, which cor­re­lates more with deaths, so it more closely fol­lows the death graph.

  US pos­i­tive test rate is up from 4.4% to 7.4%: https://​​coro­n­avirus.jhu.edu/​​test­ing/​​in­di­vi­d­ual-states

  It used to be the case that 4.4% of peo­ple you tested had COVID-19.

  Now you test more peo­ple, who look less risky on av­er­age, and find that 7.4% of peo­ple you test have COVID-19. The peo­ple you would have tested in the old days are the riskiest sub­group, so more than 7.4% of them have COVID-19.

  So it sure seems like the in­fec­tion rate went up by at least (7.4/​4.4) = +70%.

• paulfchristiano 4 Jul 2020 0:18 UTC

  4 points

  in reply to: johnswentworth’s comment on: High Stock Prices Make Sense Right Now

  My im­pres­sion is that most in­di­vi­d­ual in­vestors and pen­sion funds put a sig­nifi­cant part of their port­fo­lio into bonds.

• paulfchristiano 4 Jul 2020 0:09 UTC

  9 points

  in reply to: johnswentworth’s comment on: High Stock Prices Make Sense Right Now

  I’d love to get ev­i­dence on that and it seems im­por­tant.

  Your po­si­tion doesn’t sound right to me. You don’t need many peo­ple chang­ing their al­lo­ca­tions mod­er­ately to to­tally swamp a 1% change in in­flows.

  My guess would be that more than 10% of in­vestors, weighted by to­tal equity hold­ings, ad­just their ex­po­sure de­liber­ately, but I’d love to know the real num­bers.

• paulfchristiano 3 Jul 2020 23:57 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen’s comment on: The “AI De­bate” Debate

  Do you think some­thing like IDA is the only plau­si­ble ap­proach to al­ign­ment? If so, I hadn’t re­al­ized that, and I’d be cu­ri­ous to hear more ar­gu­ments, or just in­tu­itions are fine. The al­igned over­seer you de­scribe is sup­posed to make treach­ery im­pos­si­ble by rec­og­niz­ing it, so it seems your con­cern is equiv­a­lent to the con­cern: “any agent (we make) that learns to act will be treach­er­ous if treach­ery is pos­si­ble.” Are all learn­ing agents fun­da­men­tally out to get you? I sup­pose that’s a live pos­si­bil­ity to me, but it seems to me there is a pos­si­bil­ity we could de­sign an agent that is not in­clined to treach­ery, even if the treach­ery wouldn’t be recognized

  No, but what are the ap­proaches to avoid­ing de­cep­tive al­ign­ment that don’t go through com­pet­i­tive­ness?

  I guess the ob­vi­ous one is “don’t use ML,” and I agree that doesn’t re­quire com­pet­i­tive­ness.

  Edit: even so, hav­ing two in­ter­nal com­po­nents that are com­pet­i­tive with each other (e.g. over­seer and over­seee) does not re­quire com­pet­i­tive­ness with other pro­jects.

  No, but now we are start­ing to play the game of throt­tling the over­seee (to avoid it over­pow­er­ing the over­seer) and it’s not clear how this is go­ing to work and be sta­ble. It cur­rently seems like the only ap­peal­ing ap­proach to get­ting sta­bil­ity there is to en­sure the over­seer is com­pet­i­tive.

• paulfchristiano 3 Jul 2020 23:55 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen’s comment on: The “AI De­bate” Debate

  This ar­gu­ment seems to prove too much. Are you say­ing that if so­ciety has learned how to do ar­tifi­cial in­duc­tion at a su­per­hu­man level, then by the time we give a safe plan­ner that in­duc­tion sub­rou­tine, some­one will have already given that in­duc­tion rou­tine to an un­safe plan­ner? If so, what hope is there as pre­dic­tion al­gorithms re­lentlessly im­prove? In my view, the whole point of AGI Safety re­search is to try to come up with ways to use pow­er­ful-enough-to-kill-you ar­tifi­cial in­duc­tion in a way that it doesn’t kill you (and helps you achieve your other goals). But it seems you’re say­ing that there is a cer­tain level of in­ge­nu­ity where mal­i­cious agents will prob­a­bly act with that level of in­ge­nu­ity be­fore be­nign agents do.

  I’m say­ing that if you can’t pro­tect your­self from an AI in your lab, un­der con­di­tions that you care­fully con­trol, you prob­a­bly couldn’t pro­tect your­self from AI sys­tems out there in the world.

  The hope is that you can pro­tect your­self from an AI in your lab.

• paulfchristiano 3 Jul 2020 23:53 UTC

  LW: 4 AF: 2

  AF

  in reply to: michaelcohen’s comment on: The “AI De­bate” Debate

  So com­pet­i­tive­ness still mat­ters some­what, but here’s a po­ten­tial dis­agree­ment we might have: I think we will prob­a­bly have at least a few months, and maybe more than a year, where the top one or two teams have AGI (pow­er­ful enough to kill ev­ery­one if let loose), and no­body else has any­thing more valuable than an Ama­zon Me­chan­i­cal Turk worker.

  Definitely a dis­agree­ment, I think that be­fore any­one has an AGI that could beat hu­mans in a fist­fight, tons of peo­ple will have sys­tems much much more valuable than a me­chan­i­cal turk worker.

• paulfchristiano 3 Jul 2020 23:50 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen’s comment on: The “AI De­bate” Debate

  The way I map these con­cepts, this feels like an eli­sion to me. I un­der­stand what you’re say­ing, but I would like to have a term for “this AI isn’t try­ing to kill me”, and I think “safe” is a good one. That’s the rele­vant sense of “safe” when I say “if it’s safe, we can try it out and tin­ker”. So maybe we can re­cruit an­other word to de­scribe an AI that is both safe it­self and able to pro­tect us from other agents.

  I mean that we don’t have any pro­cess that looks like de­bate that could pro­duce an agent that wasn’t try­ing to kill you with­out be­ing com­pet­i­tive, be­cause de­bate re­lies on us­ing al­igned agents to guide the train­ing pro­cess (and if they aren’t com­pet­i­tive then the agent-be­ing-trained will, at least in the limit, con­verge to an equil­ibrium where it kills you).

• paulfchristiano 3 Jul 2020 22:14 UTC

  18 points

  on: High Stock Prices Make Sense Right Now

  The main rea­son I’m per­son­ally con­fused is that 2 months ago I thought there was real un­cer­tainty about whether we’d be able to keep the pan­demic un­der con­trol. Over the last 2 months that un­cer­tainty has grad­u­ally been re­solved in the nega­tive, with­out much pos­i­tive news about peo­ple’s will­ing­ness to throw in the towel rather than con­tin­u­ing to panic and do lock­downs, and yet over that pe­riod SPY has con­tinued mov­ing up.

  I’m mak­ing no at­tempt at all to es­ti­mate prices based on fun­da­men­tals and I’m hon­estly not even sure how that ex­er­cise is sup­posed to work. In­ter­est rates are very low and volatility isn’t that high so it seems like you would have ex­tremely high equity prices if e.g. most in­vestors were ra­tio­nal with plau­si­ble util­ity func­tions. But equity prices are never nearly as high as that kind of anal­y­sis would sug­gest.

• paulfchristiano 3 Jul 2020 22:05 UTC

  31 points

  on: High Stock Prices Make Sense Right Now

  I think peo­ple’s an­nual in­come is on av­er­age <20% of their net worth ($100T vs $20T), maybe more like 15%.

  So 2 months of +20% sav­ings amounts to <1% in­crease in to­tal sav­ings, right?

  If that’s right, this doesn’t seem very im­por­tant rel­a­tive to small changes in peo­ple’s av­er­age al­lo­ca­tion be­tween equities/​debt/​cur­rency, which fluc­tu­ate by 10%s dur­ing the nor­mal busi­ness cy­cle.

  Nor­mally I ex­pect higher sav­ings rates to rep­re­sent con­cern about hav­ing money in the fu­ture, which will be ac­com­panied by a move to safer as­sets. And of course volatility is cur­rently way up, so ra­tio­nal in­vestors prob­a­bly couldn’t af­ford to in­vest nearly as much in stocks un­less they were be­ing com­pen­sated with sig­nifi­cantly higher re­turns (which should in­volve prices only re­turn­ing to nor­mal lev­els as volatility falls).

• paulfchristiano 3 Jul 2020 0:24 UTC

  LW: 5 AF: 3

  AF

  on: The “AI De­bate” Debate

  So what if AI De­bate sur­vives this con­cern? That is, sup­pose we can re­li­ably find a hori­zon-length for which run­ning AI De­bate is not ex­is­ten­tially dan­ger­ous. One worry I’ve heard raised is that hu­man judges will be un­able to effec­tively judge ar­gu­ments way above their level. My re­ac­tion is to this is that I don’t know, but it’s not an ex­is­ten­tial failure mode, so we could try it out and tin­ker with eval­u­a­tion pro­to­cols un­til it works, or un­til we give up. If we can run AI De­bate with­out in­cur­ring an ex­is­ten­tial risk, I don’t see why it’s im­por­tant to re­solve ques­tions like this in ad­vance.

  There are two rea­sons to worry about this:

  • The pur­pose of re­search now is to un­der­stand the land­scape of plau­si­ble al­ign­ment ap­proaches, and from that per­spec­tive vi­a­bil­ity is as im­por­tant as safety.

  • I think it is un­likely for a scheme like de­bate to be safe with­out be­ing ap­prox­i­mately com­pet­i­tive—the goal is to get hon­est an­swers which are com­pet­i­tive with a po­ten­tial mal­i­cious agent, and then use those an­swers to en­sure that mal­i­cious agent can’t cause trou­ble and that the over­all sys­tem can be sta­ble to mal­i­cious per­tur­ba­tions. If your hon­est an­swers aren’t com­pet­i­tive, then you can’t do that and your situ­a­tion isn’t qual­i­ta­tively differ­ent from a hu­man try­ing to di­rectly su­per­vise a much smarter AI.

  In prac­tice I doubt the sec­ond con­sid­er­a­tion mat­ters—if your AI could eas­ily kill you in or­der to win a de­bate, prob­a­bly some­one else’s AI has already kil­led you to take your money (and long be­fore that your so­ciety to­tally fell apart). That is, safety sep­a­rate from com­pet­i­tive­ness mostly mat­ters in sce­nar­ios where you have very large leads /​ very rapid take­offs.

  Even if you were the only AI pro­ject on earth, I think com­pet­i­tive­ness is the main thing re­spon­si­ble for in­ter­nal reg­u­la­tion and sta­bil­ity. For ex­am­ple, it seems to me you need com­pet­i­tive­ness for any of the plau­si­ble ap­proaches for avoid­ing de­cep­tive al­ign­ment (since they re­quire hav­ing an al­igned over­seer who can un­der­stand what a treach­er­ous agent is do­ing). More gen­er­ally, try­ing to main­tain a to­tally san­i­tized in­ter­nal en­vi­ron­ment seems a lot harder than try­ing to main­tain a com­pet­i­tive in­ter­nal en­vi­ron­ment where mis­al­igned agents won’t be at a com­pet­i­tive ad­van­tage.