ricraz

[Question] What ex­plana­tory power does Kah­ne­man’s Sys­tem 2 pos­sess?

• ricraz 9 Aug 2019 15:59 UTC

  3 points

  in reply to: ekr's comment on: Why do hu­mans not have built-in neu­ral i/​o chan­nels?

  But hu­man ner­vous sys­tems do have much higher band­width com­mu­ni­ca­tion chan­nels. We share them with the other mam­mals. It’s the lim­bic sys­tem.

  I’m quite un­cer­tain about how high-band­width this ac­tu­ally is. I agree that in the first sec­ond of meet­ing some­one, it’s much more in­for­ma­tive than lan­guage could be. Once the ini­tial “first im­pres­sion” has oc­curred, though, the rate of com­mu­ni­ca­tion drops off sharply, and I think that lan­guage could over­take it af­ter a few min­utes. For ex­am­ple, it takes half a sec­ond to say “I’m ner­vous”, and you can keep say­ing similarly-in­for­ma­tive things for a long time: do you think you could get a new piece of similar in­for­ma­tion ev­ery half sec­ond for ten min­utes via the lim­bic sys­tem?

  (Note that I’m not nec­es­sar­ily say­ing peo­ple do com­mu­ni­cate in­for­ma­tion about their emo­tions, per­son­al­ity and so­cial sta­tus faster via lan­guage, just that they could).

• ricraz 9 Aug 2019 15:52 UTC

  2 points

  in reply to: Veedrac's comment on: Why do hu­mans not have built-in neu­ral i/​o chan­nels?

  The suc­cess of hu­man so­ciety is a good demon­stra­tion of how very low com­plex­ity sys­tems and be­havi­ours can drive your com­pe­ti­tion ex­tinct, mag­nify available re­sources, and more

  On what ba­sis are you call­ing hu­man so­cieties “very low com­plex­ity sys­tems”? The in­di­vi­d­ual units are hu­mans, whose brains are im­mensely com­plex; and then the in­ter­ac­tions be­tween hu­mans are of­ten com­pli­cated enough that no­body has a good un­der­stand­ing of the sys­tem as a whole.

  Ul­ti­mately there seems to be no im­pe­tus for a half-baked neu­ron ten­ta­cle, and a lot of cost and risk, so that will prob­a­bly never be the path to such or­ganisms.

  This seems some­what in­tu­itive, but note that this state­ment is a uni­ver­sal nega­tive: it’s say­ing there is no plau­si­ble path to this out­come. In gen­eral I think we should be pretty cau­tious about such state­ments, since a lot of evolu­tion­ary in­no­va­tions would have seemed deeply im­plau­si­ble be­fore they hap­pened. For ex­am­ple, the elephant’s trunk is a mas­sive neu­ron-rich ten­ta­cle which is heav­ily used for com­mu­ni­ca­tion.

  There are lots of sim­ple things that or­ganisms could do to make them wildly more suc­cess­ful.

  I guess this is the crux of our dis­agree­ment—could you provide some ex­am­ples?

• ricraz 9 Aug 2019 15:43 UTC

  6 points

  in reply to: Dagon's comment on: Why do hu­mans not have built-in neu­ral i/​o chan­nels?

  Any­one who’s done any in­fosec or net­work pro­to­col work will laugh at the idea that trial and er­ror (evolu­tion) can make a safe high-band­width con­nec­tion.

  There are hun­dreds of things that peo­ple have laughed at the idea of trial and er­ror (evolu­tion) do­ing, which evolu­tion in fact did. Think­ing that evolu­tion is dumb is gen­er­ally not a good heuris­tic.

  Also, I’m not sure what counts as safe in this con­text. Is lan­guage safe? Is sight?

[Question] Why do hu­mans not have built-in neu­ral i/​o chan­nels?

• ricraz 1 Aug 2019 18:33 UTC

  11 points

  on: Mis­take Ver­sus Con­flict The­ory of Against Billion­aire Philanthropy

  Down­voted for be­ing very hy­per­bolic (“less than zero”, “his op­po­nents here are all pure con­flict the­o­rists”), un­char­i­ta­ble/​mak­ing per­sonal at­tacks (“There are peo­ple op­posed to nerds or to think­ing”, “There are peo­ple who are op­posed to ac­tion of any kind”), and not sub­stan­ti­at­ing these ex­treme views (“Be­cause, come on. Read their quotes. Con­sider their ar­gu­ments.”)

  As a more ob­ject-level com­ment, sup­pose I ac­cept the hy­po­thet­i­cal that all at­tacks on billion­aire philan­thropy are en­tirely aimed at re­duc­ing the power of billion­aires. Yet if we build a so­cietal con­sen­sus that this par­tic­u­lar at­tack is very mis­guided and causes un­de­sir­able col­lat­eral dam­age, then even peo­ple who are just as anti-billion­aire as be­fore will be less likely to use it. E.g. it’s much harder to crit­i­cise billion­aires who only spend their money on helping kids with can­cer.

Book re­view: The Tech­nol­ogy Trap

• ricraz 11 Jul 2019 23:48 UTC

  16 points

  on: How can guessti­mates work?

  Very in­ter­est­ing ques­tion—the sort that makes me swing be­tween think­ing it’s brilli­ant and think­ing it’s non­sense. I do think you over­state your premise. In al­most all of the ex­am­ples given in The Se­cret of our Suc­cess, the rele­vant knowl­edge is ei­ther non-ar­bi­trary (e.g. the whole pas­sage about hunt­ing seals makes sense, it’s just difficult to ac­quire all that knowl­edge), or there’s a low cost to failure (try a differ­ent wood for your ar­rows; if they don’t fly well, go back to ba­sics).

  If I en­gage with the ques­tion as posed, though, my pri­mary an­swer is sim­ply that over time we be­came wealthy and tech­nolog­i­cally ca­pa­ble enough that we were able to re­place all the nat­u­ral things that might kill us with what­ever we’re con­fi­dent won’t kill us. Which is why you can im­pro­vise while cook­ing—all of the in­gre­di­ents have been screened very hard for safety. This is closely re­lated to your first hy­poth­e­sis.

  How­ever, this still leaves open a slightly differ­ent ques­tion. The mod­ern world is far too com­pli­cated for any­one to un­der­stand, and so we might won­der why in­com­pre­hen­si­ble emer­gent effects don’t ren­der our daily lives hap­haz­ard and illeg­ible. One par­tial an­swer is that even large-scale com­po­nents of the world (like coun­tries and com­pa­nies) were de­signed by hu­mans. A sec­ond par­tial an­swer, though, is that even in­com­pre­hen­si­ble pat­terns and mechanisms in the mod­ern world still in­ter­act with you via other peo­ple.

  This has a cou­ple of effects. Firstly, other peo­ple try to be leg­ible, it’s just part of hu­man in­ter­ac­tion. (If the man­ioc could bar­gain with you, it’d be much eas­ier to figure out how to pro­cess it prop­erly.)

  Se­condly, there’s an illu­sion of trans­parency be­cause we’re so good at and so used to un­der­stand­ing other peo­ple. So­cial in­ter­ac­tions are ob­jec­tively very com­pli­cated: in fact, they’re “cul­tural norms and pro­cesses which ap­pear ar­bi­trary, yet could have fatal con­se­quences if de­parted from”. Yet it doesn’t feel like the rea­son I re­frain from spit­ting on strangers is ar­bi­trary (even though I couldn’t ex­plain the causal path­way by which peo­ple started con­sid­er­ing it rude). Note also that the space of ideas that star­tups ex­plore is heav­ily con­strained by so­cial norms and laws.

  Thirdly, facts about other hu­mans serve as se­man­tic stop signs. Sup­pose your boss fires you, be­cause you don’t get along. There’s a nearly un­limited amount of com­plex­ity which shaped your per­son­al­ity, and your boss’ per­son­al­ity, and the fact that you ended up in your re­spec­tive po­si­tions. But once you’ve fac­tored it out into “I’m this sort of per­son, they’re that sort of per­son”, it feels pretty leg­ible—much more than “some foods are eat-raw sorts of foods, other foods are eat-cooked sorts of foods”. (Or at least, it feels much more leg­ible to us to­day—maybe peo­ple used to find the lat­ter ex­pla­na­tion just as com­pel­ling). A re­lated stop sign is the idea that “some­body knows” why each step of a com­plex causal chain hap­pened, which nudges us away from think­ing of the chain as a whole as illeg­ible.

  So I’ve given two rea­sons for in­creased leg­i­bil­ity (hu­mans build­ing things, and hu­mans ex­plain­ing things), and two for the illu­sion of leg­i­bil­ity (illu­sion of trans­parency, and se­man­tic stop signs). I think on small scales, the former effects pre­dom­i­nate. But on large scales, the lat­ter pre­dom­i­nate—the world seems more leg­ible than it ac­tu­ally is. For ex­am­ple:

  The world seems leg­ible—I can roughly pre­dict how many planes fly ev­ery day by mul­ti­ply­ing a hand­ful rough num­bers.

  Roughly pre­dict­ing the num­ber of planes which fly ev­ery day is a very low bar! You can also pre­dict the num­ber of trees in a for­est by mul­ti­ply­ing a hand­ful of num­bers. This doesn’t help you sur­vive in that for­est. What helps you sur­vive in the for­est is be­ing able to pre­dict the timing of storms and the lo­cal tiger pop­u­la­tion. In the mod­ern world, what helps you thrive is be­ing able to pre­dict the timing of re­ces­sions and crime rate trends. I don’t think we’re any bet­ter at the lat­ter two than our an­ces­tors were at the former. In fact, the large-scale arcs of our lives are now gov­erned to a much greater ex­tent by very un­pre­dictable and difficult-to-un­der­stand events, such as sci­en­tific dis­cov­er­ies, tech­nolog­i­cal in­no­va­tion and in­ter­na­tional re­la­tions.

  In sum­mary, tech­nol­ogy has helped us re­place in­di­vi­d­ual ob­jects in our en­vi­ron­ments with safer and more leg­ible al­ter­na­tives, and the emer­gent com­plex­ity which per­sists in our mod­ern en­vi­ron­ments is now ei­ther me­di­ated by peo­ple, or still very tricky to pre­dict (or both).

• ricraz 11 Jul 2019 19:30 UTC

  22 points

  in reply to: Vaniver's comment on: The AI Timelines Scam

  But my sim­ple sense is that openly dis­cussing whether or not nu­clear weapons were pos­si­ble (a tech­ni­cal claim on which peo­ple might have pri­vate in­for­ma­tion, in­clud­ing in­tu­itions in­formed by their sci­en­tific ex­pe­rience) would have had costs and it was sen­si­ble to be se­cre­tive about it. If I think that timelines are short be­cause maybe tech­nol­ogy X and tech­nol­ogy Y fit to­gether neatly, then pub­li­cly an­nounc­ing that in­creases the chances that we get short timelines be­cause some­one plugs to­gether tech­nol­ogy X and tech­nol­ogy Y. It does seem like marginal sci­en­tists speed things up here.

  I agree that there are clear costs to mak­ing ex­tra ar­gu­ments of the form “timelines are short be­cause tech­nol­ogy X and tech­nol­ogy Y will fit to­gether neatly”. How­ever, you could still make pub­lic that your timelines are a given prob­a­bil­ity dis­tri­bu­tion D, and the rea­sons which led you to that con­clu­sion are Z% ob­ject-level views which you won’t share, and (100-Z)% base rate rea­son­ing and other out­side-view con­sid­er­a­tions, which you will share.

  I think there are very few costs to declar­ing which types of rea­son­ing you’re most per­suaded by. There are some costs to ac­tu­ally mak­ing the out­side-view rea­son­ing pub­li­cly available—maybe peo­ple who read it will bet­ter un­der­stand the AI land­scape and use that in­for­ma­tion to do ca­pa­bil­ities re­search.

  But hav­ing a lack of high-qual­ity pub­lic timelines dis­cus­sion also im­poses se­ri­ous costs, for a few rea­sons:

  1. It means that safety re­searchers are more likely to be wrong, and there­fore end up do­ing less rele­vant re­search. I am gen­er­ally pretty skep­ti­cal of rea­son­ing that hasn’t been writ­ten down and un­der­gone pub­lic scrutiny.

  2. It means there’s a lot of wasted mo­tion across the safety com­mu­nity, as ev­ery­one tries to red­erive the var­i­ous ar­gu­ments in­volved, and figure out why other peo­ple have the views they do, and who they should trust.

  3. It makes build­ing com­mon knowl­edge (and the co­or­di­na­tion which that knowl­edge can be used for) much harder.

  4. It harms the cred­i­bil­ity of the field of safety from the per­spec­tive of out­side ob­servers, in­clud­ing other AI re­searchers.

  Also, the more of a risk you think 1 is, the lower the costs of dis­clo­sure are, be­cause it be­comes more likely that any in­for­ma­tion gleaned from the dis­clo­sure is wrong any­way. Yet pre­dict­ing the fu­ture is in­cred­ibly hard! So the base rate for cor­rect­ness here is low. And I don’t think that safety re­searchers have a com­pel­ling ad­van­tage when it comes to cor­rectly mod­el­ling how AI will reach hu­man level (com­pared with thought­ful ML re­searchers).

  Con­sider, by anal­ogy, a de­bate two decades ago about whether to make pub­lic the ideas of re­cur­sive self-im­prove­ment and fast take­off. The po­ten­tial cost of that is very similar to the costs of dis­clo­sure now—giv­ing ca­pa­bil­ities re­searchers these ideas might push them to­wards build­ing self-im­prov­ing AIs faster. And yet I think mak­ing those ar­gu­ments pub­lic was clearly the right de­ci­sion. Do you agree that our cur­rent situ­a­tion is fairly analo­gous?

  EDIT: Also, I’m a lit­tle con­fused by

  Sup­pose I have 5 rea­sons for want­ing dis­cus­sions to be pri­vate, and 3 of them I can eas­ily say.

  I un­der­stand that there are good rea­sons for dis­cus­sions to be pri­vate, but can you elab­o­rate on why we’d want dis­cus­sions about pri­vacy to be pri­vate?

[Question] What are some of Robin Han­son’s best posts?

• ricraz 27 Jun 2019 14:53 UTC

  13 points

  on: Embed­ded Agency: Not Just an AI Problem

  We have strong out­side-view rea­sons to ex­pect that the in­for­ma­tion pro­cess­ing in ques­tion prob­a­bly ap­prox­i­mates Bayesian rea­son­ing (for some model of the en­vi­ron­ment), and the de­ci­sion-mak­ing pro­cess ap­prox­i­mately max­i­mizes some ex­pected util­ity func­tion (which it­self ap­prox­i­mates fit­ness within the an­ces­tral en­vi­ron­ment).

  The use of “ap­prox­i­mates” in this sen­tence (and in the post as a whole) is so loose as to be deeply mis­lead­ing—for the same rea­sons that the “blue-min­imis­ing robot” shouldn’t be de­scribed as max­imis­ing some ex­pected util­ity func­tion, and the in­for­ma­tion pro­cess­ing done by a sin­gle neu­ron shouldn’t be de­scribed as Bayesian rea­son­ing (even ap­prox­i­mately!)

  See also: co­her­ent be­havi­our in the real world is an in­co­her­ent con­cept.

• ricraz 26 Jun 2019 20:40 UTC

  4 points

  AF

  on: Let’s talk about “Con­ver­gent Ra­tion­al­ity”

  There is at least one ex­am­ple (I’ve strug­gled to dig up) of a mem­ory-less RL agent learn­ing to en­code mem­ory in­for­ma­tion in the state of the world.

  I re­call an ex­am­ple of a Mu­joco agent whose mem­ory was pe­ri­od­i­cally wiped stor­ing in­for­ma­tion in the po­si­tion of its arms. I’m also hav­ing trou­ble dig­ging it up though.

• ricraz 7 Jun 2019 14:49 UTC

  6 points

  AF

  on: Risks from Learned Op­ti­miza­tion: Introduction

  We will say that a sys­tem is an op­ti­mizer if it is in­ter­nally search­ing through a search space (con­sist­ing of pos­si­ble out­puts, poli­cies, plans, strate­gies, or similar) look­ing for those el­e­ments that score high ac­cord­ing to some ob­jec­tive func­tion that is ex­plic­itly rep­re­sented within the sys­tem.

  I ap­pre­ci­ate the difficulty of ac­tu­ally defin­ing op­ti­miz­ers, and so don’t want to quib­ble with this defi­ni­tion, but am in­ter­ested in whether you think hu­mans are a cen­tral ex­am­ple of op­ti­miz­ers un­der this defi­ni­tion, and if so whether you think that most mesa-op­ti­miz­ers will “ex­plic­itly rep­re­sent” their ob­jec­tive func­tions to a similar de­gree that hu­mans do.

• ricraz 3 Jun 2019 1:19 UTC

  3 points

  in reply to: Samuel Hapák's comment on: On alien science

  Agreed that this points in the right di­rec­tion. I think there’s more to it than that though. Con­sider for ex­am­ple a three-body prob­lem un­der New­to­nian me­chan­ics. Then there’s a sense in which spec­i­fy­ing the ini­tial masses and ve­loc­i­ties of the bod­ies, along with New­ton’s laws of mo­tion, is the best way to com­press the in­for­ma­tion about these chaotic tra­jec­to­ries.

  But there’s still an open ques­tion here, which is why are three-body sys­tems chaotic? Two-body sys­tems aren’t. What makes the differ­ence? Find­ing an ex­pla­na­tion prob­a­bly doesn’t al­low you to com­press any data any more, but it still seems im­por­tant and in­ter­est­ing.

  (This seems re­lated to a po­ten­tial mod­ifi­ca­tion of your data com­pres­sion stan­dard: that good ex­pla­na­tions com­press data in a way that min­imises not just stor­age space, but also the com­pu­ta­tion re­quired to un­pack the data. I’m a lit­tle con­fused about this though.)

On alien science

• ricraz 28 May 2019 16:32 UTC

  4 points

  in reply to: Raemon's comment on: Book re­view: The Sleep­walk­ers by Arthur Koestler

  Thanks for the kind words. I agree that re­fac­tor­ing would be use­ful, but don’t have the time now. I have added some head­ings though.

A shift in ar­gu­ments for AI risk

• ricraz 24 May 2019 14:42 UTC

  4 points

  on: “Other peo­ple are wrong” vs “I am right”

  A rele­vant book recom­men­da­tion: The Enigma of Rea­son ar­gues that think­ing of high-level hu­man rea­son­ing as a tool for at­tack­ing other peo­ple’s be­liefs and defend­ing our own (re­gard­less of their ac­tual ve­rac­ity) helps ex­plain a lot of weird asym­me­tries in cog­ni­tive bi­ases we’re sus­cep­ti­ble to, in­clud­ing this one.

• ricraz 22 May 2019 15:25 UTC

  7 points

  AF

  on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  I’d like to push back on the as­sump­tion that AIs will have ex­plicit util­ity func­tions. Even if you think that suffi­ciently ad­vanced AIs will be­have in a util­ity-max­imis­ing way, their util­ity func­tions may be en­coded in a way that’s difficult to for­mal­ise (e.g. some­where within a neu­ral net­work).

  It may also be the case that co­or­di­na­tion is much harder for AIs than for hu­mans. For ex­am­ple, hu­mans are con­strained by hav­ing bod­ies, which makes it eas­ier to pun­ish defec­tion—hid­ing from the gov­ern­ment is tricky! Our bod­ies also make anonymity much harder. Whereas if you’re a piece of code which can copy it­self any­where in the world, reneg­ing on agree­ments may be­come rel­a­tively easy. Another rea­sion why AI co­op­er­a­tion might be harder is sim­ply that AIs will be ca­pa­ble of a much wider range of goals and cog­ni­tive pro­cesses than hu­mans, and so they may be less pre­dictable to each other and/​or have less com­mon ground with each other.

• ricraz 22 May 2019 15:19 UTC

  6 points

  AF

  on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  This pa­per by Critch is rele­vant: it ar­gues that agents with differ­ent be­liefs will bet their fu­ture share of a merged util­ity func­tion, such that it skews to­wards who­ever’s pre­dic­tions are more cor­rect.