rohinmshah(Rohin Shah)

• rohinmshah 20 Feb 2020 21:06 UTC

  8 points

  on: Goal-di­rected = Model-based RL?

  First off, I’m su­per happy that peo­ple are think­ing about goal-di­rected be­hav­ior :D

  I think model-based RL is typ­i­cally goal-di­rected, in that it typ­i­cally performs a search us­ing a world model for a tra­jec­tory that achieves high re­ward (the goal). How­ever, pow­er­ful model-free RL usu­ally is also goal-di­rected—con­sider AlphaZero (with­out the MCTS), OpenAI Five, AlphaS­tar, etc: these are all model-free, but still seem fairly goal-di­rected. More gen­er­ally, model-free and model-based RL al­gorithms usu­ally get similar perfor­mance on en­vi­ron­ments (of­ten model-free is less sam­ple effi­cient but has a higher fi­nal perfor­mance, though this isn’t always the case).

  Also more broadly, I think there’s a smooth spec­trum be­tween “ha­bit­ual cog­ni­tion” and “goal-di­rected cog­ni­tion”, such that you can’t cleanly carve up the space into a bi­nary “goal-di­rected” or not.

• rohinmshah 20 Feb 2020 20:53 UTC

  LW: 2 AF: 2

  AF

  in reply to: Wei_Dai's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  To the ex­tent that we ex­pect strong warn­ing shots and abil­ity to avoid build­ing AGI upon re­ceiv­ing such warn­ing shots, this seems like an ar­gu­ment for re­searchers/​longter­mists to work on /​ ad­vo­cate for safety prob­lems be­yond the stan­dard of “AGI is not try­ing to de­ceive us or work against us” (be­cause that stan­dard will likely be reached any­way). Do you agree?

  Yes.

  Some types of AI x-risk don’t af­fect ev­ery­one though (e.g., ones that re­duce the long term value of the uni­verse or mul­ti­verse with­out kil­ling ev­ery­one in the near term).

  Agreed, all else equal those seem more likely to me.

• rohinmshah 20 Feb 2020 20:51 UTC

  2 points

  in reply to: ricraz's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  See also this thread.

[AN #87]: What might hap­pen as deep learn­ing scales even fur­ther?

• rohinmshah 19 Feb 2020 11:16 UTC

  2 points

  in reply to: Matthew Barnett's comment on: Pre­face to EAF’s Re­search Agenda on Co­op­er­a­tion, Con­flict, and TAI

  I think this is prob­a­bly false, but it’s be­cause I’m us­ing the strict defi­ni­tion of s-risk.

  Yeah, I should have said some­thing like “the biggest kinds of s-risks where there is wide­spread op­ti­miza­tion for suffer­ing”.

• rohinmshah 15 Feb 2020 13:34 UTC

  LW: 3 AF: 3

  AF

  in reply to: Wei_Dai's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  What do you ex­pect the ML com­mu­nity to do at that point?

  It de­pends a lot on the par­tic­u­lar warn­ing shot that we get. But on the strong ver­sions of warn­ing shots, where there’s com­mon knowl­edge that build­ing an AGI runs a sub­stan­tial risk of de­stroy­ing the world, yes, I ex­pect them to not build AGI un­til safety is solved. (Not to the stan­dard you usu­ally imag­ine, where we must also solve philo­soph­i­cal prob­lems, but to the stan­dard I usu­ally imag­ine, where the AGI is not try­ing to de­ceive us or work against us.)

  This de­pends on other back­ground fac­tors, e.g. how much the var­i­ous ac­tors think they are value-al­igned vs. in zero-sum com­pe­ti­tion. I cur­rently think the ML com­mu­nity thinks they are mostly but not fully value-al­igned, and they will in­fluence com­pa­nies and gov­ern­ments in that di­rec­tion. (I also want more longter­mists to be try­ing to build more com­mon knowl­edge of how much hu­mans are value al­igned, to make this more likely.)

  I worry about a par­allel with the “en­ergy com­mu­nity”

  The ma­jor dis­anal­ogy is that catas­trophic out­comes of cli­mate change do not per­son­ally af­fect the CEOs of en­ergy com­pa­nies very much, whereas AI x-risk af­fects ev­ery­one. (Also, maybe we haven’t got­ten clear and ob­vi­ous warn­ing shots?)

  (com­pared to which, the dis­asters that will have oc­curred by then may well seem tol­er­able by com­par­i­son), and given prob­a­ble dis­agree­ments be­tween differ­ent ex­perts about how se­ri­ous the fu­ture risks are

  I agree that my story re­quires com­mon knowl­edge of the risk of build­ing AGI, in the sense that you need peo­ple to pre­dict “run­ning this code might lead to all hu­mans dy­ing”, and not “run­ning this code might lead to <warn­ing shot effect>”. You also need rel­a­tive agree­ment on the risks.

  I think this is pretty achiev­able. Most of the ML com­mu­nity already agrees that build­ing an AGI is high-risk if not done with some ar­gu­ment for safety. The thing peo­ple tend to dis­agree on is when we will get AGI and how much we should work on safety be­fore then.

[AN #86]: Im­prov­ing de­bate and fac­tored cog­ni­tion through hu­man experiments

• rohinmshah 12 Feb 2020 1:46 UTC

  LW: 4 AF: 3

  AF

  on: At­tain­able util­ity has a sub­agent problem

  Flo’s sum­mary for the Align­ment Newslet­ter:

  This post ar­gues that reg­u­lariz­ing an agent’s im­pact by <@at­tain­able util­ity@>(@Towards a New Im­pact Mea­sure@) can fail when the agent is able to con­struct sub­agents. At­tain­able util­ity reg­u­lariza­tion uses aux­iliary re­wards and pe­nal­izes the agent for chang­ing its abil­ity to get high ex­pected re­wards for these to re­strict the agent’s power-seek­ing. More speci­fi­cally, the penalty for an ac­tion is the ab­solute differ­ence in ex­pected cu­mu­la­tive aux­iliary re­ward be­tween the agent ei­ther do­ing the ac­tion or noth­ing for one time step and then op­ti­miz­ing for the aux­iliary re­ward.

  This can be cir­cum­vented in some cases: If the aux­iliary re­ward does not benefit from two agents in­stead of one op­ti­miz­ing it, the agent can just build a copy of it­self that does not have the penalty, as do­ing this does not change the agent’s abil­ity to get a high aux­iliary re­ward. For more gen­eral aux­iliary re­wards, an agent could build an­other more pow­er­ful agent, as long as the pow­er­ful agent com­mits to bal­anc­ing out the en­su­ing changes in the origi­nal agent’s at­tain­able aux­iliary re­wards.

  Flo’s opinion:

  I am con­fused about how much the com­mit­ment to bal­ance out the origi­nal agent’s at­tain­able util­ity would con­strain the pow­er­ful sub­agent. Also, in the pres­ence of sub­agents, it seems plau­si­ble that at­tain­able util­ity mostly de­pends on the agent’s abil­ity to pro­duce sub­agents of differ­ent gen­er­al­ity with differ­ent goals: If a sub­agent that op­ti­mizes for a sin­gle aux­iliary re­ward was eas­ier to build than a more gen­eral one, build­ing a gen­eral pow­er­ful agent could con­sid­er­ably de­crease at­tain­able util­ity for all aux­iliary re­wards, such that the high penalty rules out this ac­tion.

• rohinmshah 12 Feb 2020 1:44 UTC

  LW: 6 AF: 4

  AF

  on: Wri­teup: Progress on AI Safety via Debate

  Planned sum­mary for the Align­ment Newslet­ter:

  This post re­ports on work done on cre­at­ing a <@de­bate@>(@AI safety via de­bate@) setup that works well with hu­man play­ers. In the game, one player is hon­est (i.e. ar­gu­ing for the cor­rect an­swer) and one is mal­i­cious (i.e. ar­gu­ing for some worse an­swer), and they play a de­bate in some for­mat, af­ter which a judge must de­cide which player won the de­bate. They are us­ing Think­ing Physics ques­tions for these de­bates, be­cause they in­volve ques­tions with clear an­swers that are con­fus­ing to most peo­ple (the judges) but easy for some ex­perts (the play­ers).

  Early freeform text de­bates did not work very well, even with smart, mo­ti­vated judges. The mal­i­cious player could deflect on ques­tions they didn’t want to an­swer, e.g. by claiming that the ques­tion was am­bigu­ous and redi­rect­ing at­ten­tion by ask­ing new ques­tions. In ad­di­tion, when the mal­i­cious player got to go first and give an in­cor­rect “frame­work” for find­ing the an­swer, and then made in­di­vi­d­u­ally true claims to “fill in” the frame­work, it was hard for the hon­est player to re­but it. So, they moved to a frame­work with­out such asym­me­tries: both play­ers gave a claim (si­mul­ta­neously), both gave con­struc­tive ar­gu­ments, and both re­but­ted the other’s ar­gu­ments. In ad­di­tion, part of the ap­peal of de­bate is that the agents can “zoom in” on the par­tic­u­lar dis­agree­ment they have, and the judge need only an­a­lyze the small­est dis­agree­ment in or­der to de­clare an over­all win­ner.

  This sug­gests the fol­low­ing setup: play­ers si­mul­ta­neously provide an an­swer sup­ported with sub­claims. Then, af­ter look­ing at the other player’s an­swer and sub­claims, they can provide ob­jec­tions (per­haps on par­tic­u­lar sub­claims, or the over­all struc­ture of the ar­gu­ment). Play­ers can con­cede or challenge ob­jec­tions left by the other player; the de­bate then re­curses on one of the challenged ob­jec­tions (cur­rently, the mal­i­cious player chooses which claim to re­curse on, in or­der to be con­ser­va­tive). There is a depth limit to the re­cur­sion, once it is reached (or if there are no challenged ob­jec­tions) the judge is given a tran­script of the de­bate and must de­clare a win­ner.

  A new prob­lem arises: the mal­i­cious player can ex­ploit am­bi­guity to their ad­van­tage. For ex­am­ple, sup­pose the mal­i­cious player is ar­gu­ing that there are 5 primes be­tween 0 and 10, and makes the sub­claims “there are 3 primes be­tween 0 and 5” and “there are 2 primes be­tween 5 and 10“. If challenged on the first, they will say “the three primes are 2, 3, and 5”, if challenged on the sec­ond, they will say “the two primes are 5 and 7”. While in this sim­ple case the hon­est player could reg­ister a top-level ob­jec­tion claiming that the word “be­tween” is am­bigu­ous, and ask for clar­ifi­ca­tion, in gen­eral it would be too hard to elimi­nate all am­bi­guity.

  To miti­gate this, the team added a method of _cross ex­am­i­na­tion_, in which each player is al­lowed to ask an ear­lier ver­sion of the other agent a ques­tion, and quote its an­swer as part of the de­bate tran­script. (The post talks about how this can be ap­prox­i­mated with hu­mans, but I won’t go into the de­tails.) In the com­plex­ity the­ory anal­ogy, this boosts de­bate from PSPACE to NEXP. In­tu­itively, cross ex­am­i­na­tion al­lows play­ers to force each other to pre­com­mit to spe­cific claims and ar­gu­ments, be­cause if they don’t stay con­sis­tent in their claims and ar­gu­ments, then their op­po­nent can demon­strate the in­con­sis­tency. Now, if the mal­i­cious player tries to make an ar­gu­ment that de­pends on in­ter­pret­ing an am­bigu­ous con­cept in two differ­ent ways, then the hon­est player can cross ex­am­ine and ask which of the two in­ter­pre­ta­tions they mean. If they are in­con­sis­tent, that can be demon­strated to the judge; if they con­sis­tently an­swer one way, then the hon­est player can challenge the part of the ar­gu­ment that de­pends on the other in­ter­pre­ta­tion.

  They then iden­tify sev­eral open con­cerns with de­bate, of which they high­light the long com­pu­ta­tion prob­lem. This is a prob­lem when you no longer as­sume that the de­baters have op­ti­mal play: in this case, the mal­i­cious player could cre­ate a com­pli­cated ar­gu­ment that nei­ther de­bater un­der­stands well, that sup­ports the mal­i­cious case but that the hon­est player doesn’t know how to re­fute.

  Planned opinion:

  I en­joyed this a lot: the prob­lems found were crisp and the solu­tions had good ar­gu­ments that they ac­tu­ally solved the iden­ti­fied prob­lem. Read­ing through the ac­tual ex­am­ples and ar­gu­ments made me more op­ti­mistic about de­bate in gen­eral, mostly from a felt sense that the ac­tual con­crete re­sults were get­ting closer to match­ing the the­o­ret­i­cal ideal, and that there ac­tu­ally could be rea­son­able solu­tions to “messy” prob­lems like am­bi­guity.

  The full post has for­mal ex­pla­na­tions and ac­tual ex­am­ples, which I highly recom­mend.

• rohinmshah 12 Feb 2020 1:43 UTC

  LW: 3 AF: 2

  AF

  on: Wri­teup: Progress on AI Safety via Debate

  We also rely on the prop­erty that, be­cause the hon­est and dishon­est de­baters are copies of each other, they know ev­ery­thing the other knows.

  I don’t see why be­ing copies of each other im­plies that they know ev­ery­thing the other knows: they could (ra­tio­nally) spend their com­pu­ta­tion on un­der­stand­ing the de­tails of their po­si­tion, rather than their op­po­nent’s po­si­tion.

  For ex­am­ple, if I was play­ing against a copy of my­self, where we were both given sep­a­rate puz­zles and had to solve them faster than the other, I would spend all my time fo­cus­ing on my puz­zle, and wouldn’t know the things that my copy knows about his puz­zle (even if we have the same in­for­ma­tion, i.e. both of us can see both puz­zles and even each other’s in­ner thought pro­cess).

• rohinmshah 12 Feb 2020 0:31 UTC

  LW: 3 AF: 3

  AF

  in reply to: ofer's comment on: Pre­face to EAF’s Re­search Agenda on Co­op­er­a­tion, Con­flict, and TAI

  It seems to me that at least some parts of this re­search agenda are rele­vant for some spe­cial cases of “the failure mode of an amoral AI sys­tem that doesn’t care about you”.

  I still wouldn’t recom­mend work­ing on those parts, be­cause they seem de­cid­edly less im­pact­ful than other op­tions. But as writ­ten it does sound like I’m claiming that the agenda is to­tally use­less for any­thing be­sides s-risks, which I cer­tainly don’t be­lieve. I’ve changed that sec­ond para­graph to:

  How­ever, un­der other eth­i­cal sys­tems (un­der which s-risks are worse than x-risks, but do not com­pletely dwarf x-risks), I ex­pect other tech­ni­cal safety re­search to be more im­pact­ful, be­cause other ap­proaches can more di­rectly tar­get the failure mode of an amoral AI sys­tem that doesn’t care about you, which seems both more likely and more amenable to tech­ni­cal safety ap­proaches (to me at least). I could imag­ine work on this agenda be­ing quite im­por­tant for _strat­egy_ re­search, though I am far from an ex­pert here.

• rohinmshah 10 Feb 2020 23:47 UTC

  LW: 4 AF: 4

  AF

  on: Pre­face to EAF’s Re­search Agenda on Co­op­er­a­tion, Con­flict, and TAI

  Flo’s sum­mary for the Align­ment Newslet­ter:

  This agenda by the Effec­tive Altru­ism Foun­da­tion fo­cuses on risks of as­tro­nom­i­cal suffer­ing (s-risks) posed by <@trans­for­ma­tive AI@ >(@Defin­ing and Un­pack­ing Trans­for­ma­tive AI@) (TAI) and es­pe­cially those re­lated to con­flicts be­tween pow­er­ful AI agents. This is be­cause there is a very clear path from ex­tor­tion and ex­e­cuted threats against al­tru­is­tic val­ues to s-risks. While es­pe­cially im­por­tant in the con­text of s-risks, co­op­er­a­tion be­tween AI sys­tems is also rele­vant from a range of differ­ent view­points. The agenda cov­ers four clusters of top­ics: strat­egy, cred­i­bil­ity and bar­gain­ing, cur­rent AI frame­works, as well as de­ci­sion the­ory.

  The ex­tent of co­op­er­a­tion failures is likely in­fluenced by how power is dis­tributed af­ter the tran­si­tion to TAI. At first glance, it seems like widely dis­tributed sce­nar­ios (as <@CAIS@>(@Refram­ing Su­per­in­tel­li­gence: Com­pre­hen­sive AI Ser­vices as Gen­eral In­tel­li­gence@)) are more prob­le­matic, but re­lated liter­a­ture from in­ter­na­tional re­la­tions paints a more com­pli­cated pic­ture. The agenda seeks a bet­ter un­der­stand­ing of how the dis­tri­bu­tion of power af­fects catas­trophic risk, as well as po­ten­tial lev­ers to in­fluence this dis­tri­bu­tion. Other top­ics in the strat­egy/​gov­er­nance cluster in­clude the iden­ti­fi­ca­tion and anal­y­sis of re­al­is­tic sce­nar­ios for mis­al­ign­ment, as well as case stud­ies on co­op­er­a­tion failures in hu­mans and how they can be af­fected by policy.

  TAI might en­able un­prece­dented cred­i­bil­ity, for ex­am­ple by be­ing very trans­par­ent, which is cru­cial for both con­tracts and threats. The agenda aims at bet­ter mod­els of the effects of cred­i­bil­ity on co­op­er­a­tion failures. One ap­proach to this is open-source game the­ory, where agents can see other agents’ source codes. Promis­ing ap­proaches to pre­vent catas­trophic co­op­er­a­tion failures in­clude the iden­ti­fi­ca­tion of peace­ful bar­gain­ing mechanisms, as well as sur­ro­gate goals. The idea of sur­ro­gate goals is for an agent to com­mit to act as if it had a differ­ent goal, when­ever it is threat­ened, in or­der to pro­tect its ac­tual goal from threats.

  As some as­pects of con­tem­po­rary AI ar­chi­tec­tures might still be pre­sent in TAI, it can be use­ful to study co­op­er­a­tion failure in cur­rent sys­tems. One con­crete ap­proach to en­abling co­op­er­a­tion in so­cial dilem­mas that could be tested with con­tem­po­rary sys­tems is based on bar­gain­ing over poli­cies com­bined with pun­ish­ments for de­vi­a­tions. Re­lat­edly, it is worth in­ves­ti­gat­ing whether or not multi-agent train­ing leads to hu­man-like bar­gain­ing by de­fault. This has im­pli­ca­tions on the suit­abil­ity of be­havi­oural vs clas­si­cal game the­ory to study TAI. The be­havi­oural game the­ory of hu­man-ma­chine in­ter­ac­tions might also be im­por­tant, es­pe­cially in hu­man-in-the-loop sce­nar­ios of TAI.

  The last cluster dis­cusses the im­pli­ca­tions of bounded com­pu­ta­tion on de­ci­sion the­ory as well as the de­ci­sion the­o­ries (im­plic­itly) used by cur­rent agent ar­chi­tec­tures. Another fo­cus lies on acausal rea­son­ing and in par­tic­u­lar the pos­si­bil­ity of acausal trade, where differ­ent cor­re­lated AI’s co­op­er­ate with­out any causal links be­tween them.

  Flo’s opinion:

  I am broadly sym­pa­thetic to the fo­cus on pre­vent­ing the worst out­comes and it seems plau­si­ble that ex­tor­tion could play an im­por­tant role in these, even though I worry more about dis­tri­bu­tional shift plus in­cor­rigi­bil­ity. Still, I am ex­cited about the fo­cus on co­op­er­a­tion, as this seems ro­bustly use­ful for a wide range of sce­nar­ios and most value sys­tems.

  My opinion:

  Un­der a suffer­ing-fo­cused ethics un­der which s-risks far over­whelm x-risks, I think it makes sense to fo­cus on this agenda. There don’t seem to be many plau­si­ble paths to s-risks: by de­fault, we shouldn’t ex­pect them, be­cause it would be quite sur­pris­ing for an amoral AI sys­tem to think it was par­tic­u­larly use­ful or good for hu­mans to _suffer_, as op­posed to not ex­ist at all, and there doesn’t seem to be much rea­son to ex­pect an im­moral AI sys­tem. Con­flict and the pos­si­bil­ity of car­ry­ing out threats are the most plau­si­ble ways by which I could see this hap­pen­ing, and the agenda here fo­cuses on ne­glected prob­lems in this space.

  How­ever, un­der other eth­i­cal sys­tems (un­der which s-risks are worse than x-risks, but do not com­pletely dwarf x-risks), I ex­pect other safety re­search to be more im­pact­ful, be­cause the failure mode of an amoral AI sys­tem that doesn’t care about you seems both more likely and more amenable to tech­ni­cal safety ap­proaches (to me at least).

• rohinmshah 10 Feb 2020 12:12 UTC

  LW: 4 AF: 3

  AF

  in reply to: evhub's comment on: Syn­the­siz­ing am­plifi­ca­tion and debate

  The ba­sic de­bate RL setup is meant to be un­changed here—when I say “the RL re­ward de­rived from win­ner” I mean that in the zero-sum de­bate game sense.

  But the an­swers are gen­er­ated from pieces that in­volve hu­mans, and those hu­mans don’t be­have as though they are in a zero-sum game?

  I sup­pose you could imag­ine that the hu­man is just some func­tion, and the mod­els are pro­duc­ing an­swers that get mapped through the func­tion be­fore they get their zero-sum re­ward… but then the equil­ibrium be­hav­ior could very well be differ­ent. For ex­am­ple, if you’re ad­vis­ing a hu­man on how to play rock-pa­per-scis­sors, but they have a bias against pa­per and when you tell them to play pa­per they have a 33% chance of play­ing rock in­stead, you should now have a 50% chance of recom­mend­ing pa­per, 33% chance of scis­sors, and 17% chance for rock. So I’m not sure that the rea­sons for op­ti­mism for de­bate trans­fer over into this set­ting where you have a hu­man in the mix.

  Maybe you could make an ar­gu­ment that for any $H$ who we trust enough to do am­plifi­ca­tion /​ de­bate in the first place, this isn’t a prob­lem, since $\text{Amp}(H,M)$ is sup­posed to be more ca­pa­ble than $M$. Alter­na­tively you could say that at the very least $M$ is such that $\text{Amp}(H,M)$ gives true and use­ful ar­gu­ments, though that might con­flict with train­ing $M$ to imi­tate $\text{Amp}(H,M)$ (as in the rock-pa­per-scis­sors ex­am­ple above).

• rohinmshah 9 Feb 2020 2:45 UTC

  LW: 2 AF: 2

  AF

  in reply to: rohinmshah's comment on: Cri­tiquing “What failure looks like”

  When I imag­ine WFLL1 that doesn’t turn into WFLL2, I usu­ally imag­ine a world in which all ex­ist­ing hu­mans lead great lives, but don’t have much con­trol over the fu­ture. On a mo­ment-to-mo­ment ba­sis, that world is bet­ter than the cur­rent world, but we don’t get to in­fluence the fu­ture and make use of the cos­mic en­dow­ment, and so from a to­tal view we have lost >99% of the po­ten­tial value of the fu­ture.

  I was un­cer­tain about this, but it seems this is at least what Paul in­tended. From here, about WFLL1:

  The availa­bil­ity of AI still prob­a­bly in­creases hu­mans’ ab­solute wealth. This is a prob­lem for hu­mans be­cause we care about our frac­tion of in­fluence over the fu­ture, not just our ab­solute level of wealth over the short term.

• rohinmshah 9 Feb 2020 2:24 UTC

  LW: 4 AF: 3

  AF

  on: Syn­the­siz­ing am­plifi­ca­tion and debate

  I like the ba­sic idea, but I don’t un­der­stand the de­tails, so by de­fault won’t in­clude it in the newslet­ter. Some con­fu­sions:

  • Are the ar­gu­ments the same thing as an­swers? (I get this im­pres­sion be­cause you say “Is arg_t a suffi­cient an­swer to Q in con­text S?”.) If not, where is the an­swer in the de­bate? More gen­er­ally I would benefit a lot from a con­crete ex­am­ple (e.g. the Bali vs. Alaska ex­am­ple).

  • De­bate sets up a game and ar­gues that the equil­ibrium is truth-tel­ling. It does that by set­ting up a zero-sum game and then us­ing self-play for train­ing; self-play will con­verge to the Nash equil­ibrium, so you are then jus­tified in only an­a­lyz­ing the equil­ibrium, while ig­nor­ing the train­ing pro­cess. How­ever, in your use of de­bate, afaict noth­ing en­forces that you con­verge to the equil­ibrium of the zero-sum game, so I don’t see why you gain the benefits of de­bate.

  • Why do you want to add an aux­iliary RL ob­jec­tive? I nor­mally imag­ine two rea­sons. First, maybe the task you want to solve is well suited to RL, e.g. Atari games, and so you want to train on that RL ob­jec­tive in ad­di­tion to the ques­tion an­swer­ing ob­jec­tive, so that the RL ob­jec­tive lets you learn good rep­re­sen­ta­tions quickly. Se­cond, if your model M is un­able to do perfect imi­ta­tion, there must be er­rors, and in this case the imi­ta­tion ob­jec­tive doesn’t nec­es­sar­ily in­cen­tivize the right thing, whereas the RL ob­jec­tive does. (See Against Mimicry.) I think yours is aiming at the sec­ond and not the first?

  What links here?

  • [AN #86]: Im­prov­ing de­bate and fac­tored cog­ni­tion through hu­man experiments by rohinmshah (12 Feb 2020 18:10 UTC; 16 points)

• rohinmshah 9 Feb 2020 1:50 UTC

  LW: 2 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: Clar­ify­ing “AI Align­ment”

  You made a claim a few com­ments above:

  But ul­ti­mately, you need the­o­ret­i­cal knowl­edge to know what can be safely in­ferred from these ex­per­i­ments. Without the­ory you can­not ex­trap­o­late.

  I’m strug­gling to un­der­stand what you mean by “the­ory” here, and the pro­gram­ming ex­am­ple was try­ing to get at that, but not very suc­cess­fully. So let’s take the sand­wich ex­am­ple:

  I made a sand­wich this morn­ing with­out build­ing math­e­mat­i­cal the­ory :)

  Pre­sum­ably the in­gre­di­ents were in a slightly differ­ent con­figu­ra­tion than you had ever seen them be­fore, but you were still able to “ex­trap­o­late” to figure out how to make a sand­wich any­way. Why didn’t you need the­ory for that ex­trap­o­la­tion?

  Ob­vi­ously this is a silly ex­am­ple, but I don’t cur­rently see any qual­i­ta­tive differ­ence be­tween sand­wich-mak­ing-ex­trap­o­la­tion, and the sort of ex­trap­o­la­tion we do when we make qual­i­ta­tive ar­gu­ments about AI risk. Why trust the former but not the lat­ter? One is an­swer is that the lat­ter is more com­plex, but you seem to be ar­gu­ing some­thing else.

• rohinmshah 7 Feb 2020 18:22 UTC

  LW: 2 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: Clar­ify­ing “AI Align­ment”

  Which of these fea­tures are nec­es­sary for in­tent-al­ign­ment and which are only nec­es­sary for strong al­ign­ment?

  As far as I can tell, 2, 3, 4, and 10 are pro­posed im­ple­men­ta­tions, not fea­tures. (E.g. the fea­ture cor­re­spond­ing to 3 is “doesn’t ma­nipu­late the user” or some­thing like that.) I’m not sure what 9, 11 and 13 are about. For the oth­ers, I’d say they’re all fea­tures that an in­tent-al­igned AI should have; just not in liter­ally all pos­si­ble situ­a­tions. But the im­ple­men­ta­tion you want is some­thing that aims for in­tent al­ign­ment; then be­cause the AI is in­tent al­igned it should have fea­tures 1, 5, 6, 7, 8. Maybe fea­ture 12 is one I think is not cov­ered by in­tent al­ign­ment, but is im­por­tant to have.

  I am not an ex­pert but I ex­pect that bridges are con­structed so that they don’t en­ter high-am­pli­tude res­o­nance in the rele­vant range of fre­quen­cies (which is an ex­am­ple of us­ing as­sump­tions in our mod­els that need in­de­pen­dent val­i­da­tion).

  This is prob­a­bly true now that we know about res­o­nance (be­cause bridges have fallen down due to res­o­nance); I was ask­ing you to take the per­spec­tive where you haven’t yet seen a bridge fall down from res­o­nance, and so you don’t think about it.

  On the other hand, I use math­e­mat­i­cal mod­els to write code for ap­pli­ca­tions all the time, with some suc­cess I dare­say. I guess that differ­ent ex­pe­rience pro­duces differ­ent in­tu­itions.

  Maybe I’m fal­ling prey to the typ­i­cal mind fal­lacy, but I re­ally doubt that you use math­e­mat­i­cal mod­els to write code in the way that I mean, and I sus­pect you in­stead mi­s­un­der­stood what I meant.

  Like, if I asked you to write code to check if an el­e­ment is pre­sent in an ar­ray, do you prove the­o­rems? I cer­tainly ex­pect that you have an in­tu­itive model of how your pro­gram­ming lan­guage of choice works, and that model in­forms the code that you write, but it seems wrong to me to de­scribe what I do, what all of my stu­dents do, and what I ex­pect you do as us­ing a “math­e­mat­i­cal the­ory of how to write code”.

  But, even just un­der­stand­ing whether we live in such a uni­verse re­quires build­ing a math­e­mat­i­cal the­ory.

  I’m cu­ri­ous what you think doesn’t re­quire build­ing a math­e­mat­i­cal the­ory? It seems to me that pre­dict­ing whether or not we are doomed if we don’t have a proof of safety is the sort of thing the AI safety com­mu­nity has done a lot of with­out a math­e­mat­i­cal the­ory. (Like, that’s how I in­ter­pret the rocket al­ign­ment and se­cu­rity mind­set posts.)

• rohinmshah 5 Feb 2020 20:26 UTC

  4 points

  in reply to: Pattern's comment on: [AN #85]: The nor­ma­tive ques­tions we should be ask­ing for AI al­ign­ment, and a sur­pris­ingly good chatbot

  Thanks, fixed.

[AN #85]: The nor­ma­tive ques­tions we should be ask­ing for AI al­ign­ment, and a sur­pris­ingly good chatbot

• rohinmshah 2 Feb 2020 8:55 UTC

  LW: 2 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: Clar­ify­ing “AI Align­ment”

  The rea­sons the risk are great are stan­dard ar­gu­ments, so I am a lit­tle con­fused why you ask about this.

  Sorry, I meant what are the rea­sons that the risk greater than the risk from a failure of in­tent al­ign­ment? The ques­tion was meant to be com­pared to the coun­ter­fac­tual of work on in­tent al­ign­ment, since the un­der­ly­ing dis­agree­ment is about com­par­ing work on in­tent al­ign­ment to other AI safety work. Similarly for the ques­tion about why it might take a long time to solve.

  Then, I don’t un­der­stand why you be­lieve that work on any­thing other than in­tent-al­ign­ment is much less ur­gent?

  I’m claiming that in­tent al­ign­ment cap­tures a large pro­por­tion of pos­si­ble failure modes, that seem par­tic­u­larly amenable to a solu­tion.

  Imag­ine that a fair coin was go­ing to be flipped 21 times, and you need to say whether there were more heads than tails. By de­fault you see noth­ing, but you could try to build two ma­chines:

  1. Ma­chine A is easy to build but not very ro­bust; it re­ports the out­come of each coin flip but has a 1% chance of er­ror for each coin flip.

  2. Ma­chine B is hard to build but very ro­bust; it re­ports the out­come of each coin flip perfectly. How­ever, you only have a 50% chance of build­ing it by the time you need it.

  In this situ­a­tion, ma­chine A is a much bet­ter plan.

  (The ex­am­ple is meant to illus­trate the phe­nomenon by which you might want to choose a riskier but eas­ier-to-cre­ate op­tion; it’s not meant to prop­erly model in­tent al­ign­ment vs. other stuff on other axes.)

  This is ac­tu­ally an im­por­tant les­son about why we need the­ory: to con­struct a use­ful the­o­ret­i­cal model you don’t need to know all pos­si­ble failure modes, you only need a rea­son­able set of as­sump­tions.

  I cer­tainly agree with that. My mo­ti­va­tion in choos­ing this ex­am­ple is that em­piri­cally we should not be able to prove that bridges are safe w.r.t res­o­nance, be­cause in fact they are not safe and do fall when res­o­nance oc­curs. (Maybe to­day bridge-build­ing tech­nol­ogy has ad­vanced such that we are able to do such proofs, I don’t know, but at least in the past that would not have been the case.)

  In this case, we ei­ther fail to prove any­thing, or we make un­re­al­is­tic as­sump­tions that do not hold in re­al­ity and get a proof of safety. Similarly, I think in many cases in­volv­ing prop­er­ties about a com­plex real en­vi­ron­ment, your two op­tions are 1. don’t prove things or 2. prove things with un­re­al­is­tic as­sump­tions that don’t hold.

  But if you’re send­ing a space­ship to Mars (or mak­ing a su­per­in­tel­li­gent AI), trial and er­ror is too ex­pen­sive. [...] Without the­ory you can­not ex­trap­o­late.

  I am not sug­gest­ing that we throw away all logic and make ran­dom ed­its to lines of code and try them out un­til we find a safe AI. I am sim­ply say­ing that our things-that-al­low-us-to-ex­trap­o­late need not be ex­pressed in math with the­o­rems. I don’t build math­e­mat­i­cal the­o­ries of how to write code, and usu­ally don’t prove my code cor­rect; nonethe­less I seem to ex­trap­o­late quite well to new cod­ing prob­lems.

  It also sounds like you’re mak­ing a nor­ma­tive claim for proofs; I’m more in­ter­ested in the em­piri­cal claim. (But I might be mis­read­ing you here.)

  I dis­agree. For ex­am­ple, [...]

  Cer­tainly you can come up with bridg­ing as­sump­tions to bridge be­tween lev­els of ab­strac­tion (in this case the as­sump­tion that “hu­man think­ing for a day” is within F). I would ex­pect that I would find some bridg­ing as­sump­tion im­plau­si­ble in these set­tings.