Vaniver

• Vaniver 26 Jun 2019 23:17 UTC

  3 points

  AF

  in reply to: G Gordon Worley III's comment on: Re­search Agenda in re­verse: what *would* a solu­tion look like?

  In my ex­pe­rience, peo­ple mostly haven’t had the view of “we can just do CEV, it’ll be fine” and in­stead have had the view of “be­fore we figure out what our prefer­ences are, which is an in­her­ently poli­ti­cal and messy ques­tion, let’s figure out how to load any prefer­ences at all.”

  It seems like there needs to be some in­ter­play here—”what we can load” in­forms “what shape we should force our prefer­ences into” and “what shape our prefer­ences ac­tu­ally are” in­forms “what load­ing needs to be ca­pa­ble of to count as al­igned.”

• Vaniver 26 Jun 2019 22:10 UTC

  3 points

  in reply to: Dr_Manhattan's comment on: Jor­dan Peter­son on AI-FOOM

  Yeah, it’s sort of awk­ward that there are two differ­ent things one might want to talk about with FOOM: the idea of re­cur­sive self im­prove­ment in the typ­i­cal I.J. Good sense, and the “hu­man thresh­old isn’t spe­cial and can be blown past quickly” idea. AlphaZero be­ing able to hit the su­per­hu­man level at Go af­ter 3 days of train­ing, and do­ing so only a year or two af­ter any pro­fes­sional Go player was defeated by a com­puter, feels rele­vant to the sec­ond thing but not the first (and is con­nected to the ‘fleets of cars will learn very differ­ently’ thing Peter­son is point­ing at).

  [And the two ac­tu­ally are dis­tinct; RSI is an ar­gu­ment for ‘blow­ing past hu­mans is pos­si­ble’ but many ‘slow take­off’ views look more like “RSI pulls hu­mans along with it” than “things look slow to a Mar­tian,” and there’s ways to quickly blow past hu­mans that don’t in­volve RSI.]

• Vaniver 26 Jun 2019 18:23 UTC

  6 points

  on: What does the word “col­lab­o­ra­tive” mean in the phrase “col­lab­o­ra­tive truth­seek­ing”?

  If “col­lab­o­ra­tive” is qual­ify­ing truth-seek­ing, per­haps we can see it more eas­ily by con­trast with non-col­lab­o­ra­tive truth­seek­ing. So what might that look like?

  • I might sim­ply be op­ti­miz­ing for the ac­cu­racy of my be­liefs, in­stead of whether or not you also dis­cover the truth.

  • I might be op­ti­miz­ing com­pet­i­tively, where my be­liefs are sim­ply judged on whether they’re bet­ter than yours.

  • I might be pri­mar­ily con­cerned about learn­ing from the en­vi­ron­ment or from my­self as op­posed to learn­ing from you.

  • I might be fol­low­ing only my in­ter­ests, in­stead of joint in­ter­ests.

  • I might be be­hav­ing in a way that doesn’t in­cen­tivize you to point out things use­ful to me, or dis­card­ing clues you provide, or in a way that fails to provide you clues.

  This sug­gests col­lab­o­ra­tive truth­seek­ing is done 1) for the benefit of both par­ties, 2) in a way that builds trust and mu­tual un­der­stand­ing, and 3) in a way that uses that trust and mu­tual un­der­stand­ing as a foun­da­tion.

  There’s an­other rele­vant con­trast, where we could look at col­lab­o­ra­tive non-truth­seek­ing, or con­trast “col­lab­o­ra­tive truth­seek­ing” as a pro­ce­dure with other pro­ce­dures that could be used (like “al­lo­cat­ing blame”), but this one seems most re­lated to what you’re driv­ing at.

• Vaniver 26 Jun 2019 18:03 UTC

  15 points

  on: Jor­dan Peter­son on AI-FOOM

  YouTube’s tran­script (with sig­nifi­cant edit­ing by me, mostly to clean and for­mat):

  Now the guys that are build­ing the au­tonomous cars, they don’t think they’re build­ing au­tonomous cars. They know perfectly well what they’re do­ing. They’re build­ing fleets of mu­tu­ally in­ter­com­mu­ni­cat­ing au­tonomous robots and each of them will to be able to teach the other be­cause their ner­vous sys­tem will be the same and when there’s ten mil­lion of them, when one of them learns some­thing all ten mil­lion of them will learn it at the same time. They’re not gonna have to be very bright be­fore they’re very very very smart.

  Be­cause us, you know, we’ll learn some­thing. You have to imi­tate it, God that’s hard. Or I have to ex­plain it to you and you have to un­der­stand it and then you have to act it out. We’re not con­nected wire­lessly with the same plat­form, but robots they are and so once those things get a lit­tle bit smart they’re not go­ing to stop at a lit­tle bit smart for very long they’re gonna be un­be­liev­ably smart like overnight.

  And they’re imi­tat­ing the hell out of us right now too be­cause we’re teach­ing them how to un­der­stand us ev­ery sec­ond of ev­ery day the net is learn­ing what we’re like. It’s watch­ing us, it’s com­mu­ni­cat­ing with us, it’s imi­tat­ing us and it’s gonna know. It already knows in some ways more about us than we know about our­selves. There’s lots of re­ports already of peo­ple get­ting preg­nancy ads or ads for in­fants, some­times be­fore they know they’re preg­nant, but of­ten be­fore they’ve told their fam­i­lies. The way that that hap­pens is the net is watch­ing what they’re look­ing at and in­fer­ring with its ar­tifi­cial in­tel­li­gence and so maybe you’re preg­nant that’s just tilt­ing you a lit­tle bit to in­ter­est in things that you might not oth­er­wise be in­ter­ested in. The net tracks that, then it tells you what you’re af­ter it does that by offer­ing an ad­ver­tise­ment. It’s read­ing your un­con­scious mind.

  Well, so that’s what’s hap­pen­ing.

• Vaniver 25 Jun 2019 4:07 UTC

  10 points

  on: How does one get in­vited to the al­ign­ment fo­rum?

  We’ve been in some­thing of a tran­si­tion pe­riod with the al­ign­ment fo­rum, where no one was pay­ing ac­tive at­ten­tion to pro­mot­ing com­ments or posts or adding users, but start­ing soon I should be do­ing that. The pri­mary thing that hap­pens when some­one’s an AF mem­ber is that they can add posts and com­ments with­out ap­proval (and one’s votes also con­vey AF karma); I ex­pect I’ll mostly go through some­one’s com­ments on AF posts and ask “would I re­li­ably pro­mote con­tent like this?” (or, in­deed, “have I re­li­ably pro­moted this per­son’s com­ments on AF posts?”).

  De­tails about what sort of com­ments I’ll think are helpful or in­sight­ful are, un­for­tu­nately, harder to ar­tic­u­late.

• Vaniver 20 Jun 2019 18:35 UTC

  16 points

  AF

  on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  Over­all, I was pretty im­pressed by this; there were sev­eral points where I thought “sure, that would be nice, but ob­sta­cle X,” and then the next sec­tion brought up ob­sta­cle X.

  I re­main sort of un­con­vinced that util­ity func­tions are the right type sig­na­ture for this sort of thing, but I do feel con­vinced that “we need some sort of for­mal syn­the­sis pro­cess, and a pos­si­ble end product of that is a util­ity func­tion.”

  That is, most of the ar­gu­ments I see for ‘how a util­ity func­tion could work’ go through some twisted steps. Sup­pose I’m try­ing to build a robot, and I want it to be cor­rigible, and I have a cor­rigi­bil­ity de­tec­tor whose type is ‘de­ci­sion pro­cess’ to ‘score’. I need to wrap that de­tec­tor with a ‘world state’ to ‘de­ci­sion pro­cess’ func­tion and a ‘score’ to ‘util­ity’ func­tion, and then I can hand it off to a robot that does a ‘de­ci­sion pro­cess’ to ‘world state’ pre­dic­tion and op­ti­mizes util­ity. If the robot’s pre­dic­tive abil­ities are su­per­hu­man, it can trace out what­ever weird de­pen­den­cies I couldn’t see; if they’re im­perfect, then each new trans­for­ma­tion pro­vides an­other op­por­tu­nity for er­rors to creep in. And it may be the case that this is a core part of re­flec­tive sta­bil­ity (be­cause if you map through world-his­to­ries you bring ob­jec­tive re­al­ity into things in a way that will be asymp­tot­i­cally sta­ble with in­creas­ing in­tel­li­gence) that doesn’t have an­other re­place­ment.

  I do find my­self wor­ry­ing that em­bed­ded agency will re­quire drop­ping util­ity func­tions in a deep way that ends up con­nected to whether or not this agenda will work (or which parts of it will work), but re­main op­ti­mistic that you’ll find out some­thing use­ful along the way and have that sort of ob­sta­cle in mind as you’re work­ing on it.

• Vaniver 20 Jun 2019 18:02 UTC

  3 points

  AF

  on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  Fixed a typo.

• Vaniver 17 Jun 2019 5:45 UTC

  9 points

  on: Recom­men­da­tion Fea­tures on LessWrong

  So, I was just recom­mended Plasti­na­tion is Ma­tur­ing and Needs Fund­ing. I con­sid­ered putting some effort into “what’s the state of plas­ti­na­tion in 2019, 7 years later?” and com­ment­ing, but hit a hand­ful of ob­sta­cles, one of which was “is the state of plas­ti­na­tion in 2019 long con­tent?”. Like, the rele­vant fund paid out its prizes at var­i­ous times, and it’d take a bit more dig­ging to figure out if the par­tic­u­lar team in Han­son’s post was the one that won, and it’s not re­ally ob­vi­ous if it mat­ters. (Sup­pose we dis­cover that the prize wasn’t won by that team, af­ter the eval­u­a­tion was paid for; what does that im­ply?)

  This makes me more ex­cited about John’s idea that shows posts with some si­mul­tane­ity be­tween users; like the Se­quences Reruns, for ex­am­ple. It might be worth it to have a com­ment writ­ing up what’s changed for the other peo­ple click­ing on it in 2019 who don’t know where to look or aren’t that com­mit­ted to figur­ing things out, where it doesn’t make sense to push that post into ‘re­cent dis­cus­sion’ on my own (if this was ran­domly picked for me).

[Question] Is there a guide to ‘Prob­lems that are too fast to Google’?

• Vaniver 14 Jun 2019 22:26 UTC

  3 points

  in reply to: Zvi's comment on: Some Ways Co­or­di­na­tion is Hard

  I fixed it more.

Wel­come to LessWrong!

• Vaniver 11 Jun 2019 20:57 UTC

  7 points

  on: [An­swer] Why wasn’t sci­ence in­vented in China?

  Ruby, you might also want to bor­row Why the West Rules—for Now from me; it fo­cuses less on the sci­en­tific ques­tion and more on the eco­nomic and tech­nolog­i­cal one (which ends up be­ing con­nected), but I’m not sure it’ll be all that differ­ent from Huff.

• Vaniver 11 Jun 2019 20:41 UTC

  5 points

  on: [An­swer] Why wasn’t sci­ence in­vented in China?

  It’s been as­serted [source] that hav­ing Latin as a lin­gua franca was im­por­tant for Europe in­te­grated mar­ket for ideas. Makes sense if schol­ars who oth­er­wise speak differ­ent lan­guages are go­ing to be able to com­mu­ni­cate.

  But the Mus­lim world was much bet­ter off in this re­gard, with Ara­bic, and while China has ma­jor lin­guis­tic vari­a­tion I think it also had a ‘shared lan­guage’ in ba­si­cally the same way Latin was a shared lan­guage for Europe.

  It seems to me like the thing that’s im­por­tant is not so much that the mar­ket is in­te­grated, but that there are many buy­ers and sel­l­ers. The best works of Chi­nese philos­o­phy, as far as I can tell, come from the pe­riod when there was ma­jor in­tel­lec­tual and mil­i­tary com­pe­ti­tion be­tween com­pet­ing fac­tions; the con­tention be­tween the Hun­dred Schools of Thought. And then af­ter unifi­ca­tion the pri­mary work available for schol­ars was the unified bu­reau­cracy, which was in­ter­ested in the Con­fu­cian-Le­gal­ist blend that won the unifi­ca­tion war, and noth­ing else.

• Vaniver 8 Jun 2019 19:32 UTC

  4 points

  in reply to: Pattern's comment on: Steel­man­ning Divination

  I would imag­ine so, be­cause it means you learn the cards as op­posed to the se­quence of cards. (“In French, chateau always fol­lows voiture.”)

• Vaniver 8 Jun 2019 19:30 UTC

  4 points

  in reply to: Scott Garrabrant's comment on: Steel­man­ning Divination

  I mean, I think it would be more ac­cu­rate to say some­thing like “the die roll, as it’s un­cor­re­lated with fea­tures of the de­ci­sion, doesn’t give me any new in­for­ma­tion about which ac­tion is best,” but the rea­son why I point to CoEE is be­cause it is ac­tu­ally a valid in­tro­spec­tive tech­nique to imag­ine act­ing by coin­flip or dieroll and then see if you’re hop­ing for a par­tic­u­lar re­sult, which rhymes with the “if you can pre­dictably up­date in di­rec­tion X, then you should already be there.”

• Vaniver 5 Jun 2019 23:04 UTC

  12 points

  on: Steel­man­ning Divination

  The SSC post that mo­ti­vated fi­nally finish­ing this up was Book Re­view: The Se­cret Of Our Suc­cess, which dis­cusses the game-the­o­retic val­idity of ran­dom­iza­tion in com­pet­i­tive en­deav­ors (like hunters vs. prey, or gen­er­als vs. gen­er­als). It seemed im­por­tant to also bring up the other sorts of val­idity, of ran­dom­ness as de­bi­as­ing or de-con­found­ing (like why ran­dom­ized con­trol­led tri­als are good) or ran­dom­ness as mechanism to make salient pre-ex­ist­ing prin­ci­ples. I’m re­minded of some on­line ad­vice-pur­veyor who would of­ten get emails from peo­ple ask­ing if their generic ad­vice ap­plied to their spe­cific situ­a­tion; al­most always, the an­swer was ‘yes,’ and there was some­thing about the per­sonal at­ten­tion that was rele­vant; hav­ing it be the case that this par­tic­u­lar bit of ad­vice was se­lected for the situ­a­tion you’re in makes it feel worth con­sid­er­ing in a way that “yeah, I guess I could throw this whole book of ad­vice at my prob­lem” doesn’t.

Steel­man­ning Divination

• Vaniver 31 May 2019 17:10 UTC

  12 points

  in reply to: Dagon's comment on: Quotes from Mo­ral Mazes

  This was prob­a­bly an ac­cu­rate de­pic­tion of Amer­i­can cor­po­rate man­age­ment when it was writ­ten, in the 80s. Since then, things have changed some­what (in part by tech be­com­ing a larger frac­tion of the econ­omy, and by in­creas­ing mer­i­toc­racy through in­creased com­pet­i­tive­ness), but I think it’s still pre­sent in a ma­jor way.

  It seems like most of these quotes are di­rectly at odds with seek­ing profit (ei­ther long- or short-term), and it would be en­light­en­ing to hear why there’s not a bunch more effi­cient or­ga­ni­za­tions tak­ing over.

  I think this is hap­pen­ing, but it’s slow. Koch In­dus­tries claims that a ma­jor piece of so­cial tech they use is com­pen­sat­ing man­agers based on the net pre­sent value of the thing they’re man­ag­ing, rather than whether they’re hit­ting key tar­gets, and they’re grow­ing at some­thing like 10% faster than the rest of the econ­omy, but that still means a very long time un­til they’ve taken over (and the larger they get, the harder it is to main­tain that rel­a­tive rate).

• Vaniver 20 May 2019 20:45 UTC

  14 points

  in reply to: jessicata's comment on: Com­ment sec­tion from 05/​19/​2019

  If we are go­ing to have the ex­cep­tion to the norm at all, then there has to be a pretty high stan­dard of ev­i­dence to prove that adding ‘Y’ to the dis­course, in fact, has bad con­se­quences.

  I want to note that LW definitely has ex­cep­tions to this norm, if only be­cause of the bor­ing, nor­mal ex­cep­tions. (If we would get in trou­ble with law en­force­ment for host­ing some­thing you might put on LW, don’t put it on LW.) We’ve had in the works (for quite some time) a post ex­plain­ing our po­si­tion on less bor­ing cases more clearly, but it runs into difficulty with the sort of is­sues that you dis­cuss here; gen­er­ally these ques­tions are an­swered in pri­vate in a way that con­nects to the judg­ment calls be­ing made and the par­tic­u­lars of the case, as op­posed to through trans­par­ent prin­ci­ples that can be clearly un­der­stood and pre­dicted in ad­vance (in part be­cause, to ex­tend the anal­ogy, this em­pow­ers the were­wolves as well).

• Vaniver 20 May 2019 0:33 UTC

  44 points

  in reply to: Vanessa Kosoy's comment on: Com­ment sec­tion from 05/​19/​2019

  [Writ­ten as an ad­min]

  First and fore­most, LW is a space for in­tel­lec­tual progress about ra­tio­nal­ity and re­lated top­ics. Cur­rently, we don’t ban peo­ple for be­ing fix­ated on a topic, or ‘darkly hint­ing,’ or posts they make off-site, and I don’t think we should. We do keep a care­ful eye on such peo­ple, and in­ter­pret be­hav­ior in ‘grey ar­eas’ ac­cord­ingly, in a way that I think re­flects both good Bayesi­anism and good mod­er­a­tion prac­tice.

  In my fa­vorite world, peo­ple who dis­agree on ob­ject-level ques­tions (both poli­ti­cal and non-poli­ti­cal) can nev­er­the­less civilly dis­cuss ab­stract is­sues. This fa­vors asym­met­ric weapons and is a core com­po­nent of truth-seek­ing. So, while hurt feel­ings and find­ing things un­pleas­ant are le­gi­t­i­mate and it’s worth spend­ing effort op­ti­miz­ing to pre­vent them, we can’t give them that much weight un­less they differ­en­ti­ate the true and the un­true.

  That said, there are ways to bring up true things that as a whole move peo­ple away from the truth, and you might be wor­ried about agree­ments on ab­strac­tions be­ing twisted to force agree­ment on ob­ject-level is­sues. Th­ese are hard to fight, and frus­trat­ing if you see them and oth­ers don’t. The best re­sponse I know is to cat­a­log the lo­cal truths and lay out how they add up to a lie, or es­tab­lish the case that agree­ment on those ab­strac­tions doesn’t force agree­ment on the ob­ject-level is­sues, and bring up the cat­a­log ev­ery time the lo­cal truth ad­vances a global lie. This is a lot more work than flyswat­ting, but has a much stronger bent to­wards truth. If you be­lieve this is what Zack is do­ing, I en­courage you to write a com­pila­tion post and point peo­ple to it as needed; due to the na­ture of that post, and where it falls on the spec­trum from nam­ing ab­stract dy­nam­ics to call-out post, we might leave it on your per­sonal blog or ask that you pub­lish it out­side of LW (and link to it as nec­es­sary).