There is no set­ting to do this cur­rently, but we’re work­ing on it!

Note to Great­er­Wrong users: GW now has full sup­port for spoiler blocks (they will render cor­rectly—mouse over one, or se­lect its text, to re­veal—and there’s a new but­ton in the ed­itor that will in­sert the cor­rect spoiler syn­tax for you.)

Said, can you make it so that if I make a com­ment on an AF post through GW or re­spond to an AF com­ment, it auto­mat­ic­ally gets pos­ted to AF? (Or let me know if there’s a set­ting to con­trol this.) Right now I have to re­mem­ber to go to that com­ment in LW and then manu­ally move it to AF.

A study by the in­vest­ment-re­search firm Strategas which was cited in The Econom­ist and the Wash­ing­ton Post com­pared the 50 firms that spent the most on lob­by­ing re­l­at­ive to their as­sets, and com­pared their fin­an­cial per­form­ance against that of the S&P 500 in the stock mar­ket; the study con­cluded that spend­ing on lob­by­ing was a “spec­tac­u­lar in­vest­ment” yield­ing “blis­ter­ing” re­turns com­par­able to a high-fly­ing hedge fund, even des­pite the fin­an­cial down­turn of the past few years.

I think I read this re­search while I was a Strategas cli­ent; if I’m re­mem­ber­ing it cor­rectly it was ex­tremely poorly done. Short back test (just a few years), garden of fork­ing paths, etc. Most sell-side re­search is not epi­stem­ic­ally rigour­ous and Strategas is not one of the bet­ter firms. I would not put much weight on this re­search.

There is wide­spread agree­ment that a key in­gredi­ent in ef­fect­ive lob­by­ing is money. This view is shared by play­ers in the lob­by­ing in­dustry.

Well of course lob­by­ists would say they’re worth the money!

I agree with habryka that this is a really good ex­plan­a­tion. I also agree with most of your pros and cons, but for me an­other ma­jor con is that this de­com­pos­i­tion moves some prob­lems that I think are cru­cial and ur­gent out of “AI align­ment” and into the “com­pet­ence” part, with the im­pli­cit or ex­pli­cit im­plic­a­tion that they are not as im­port­ant, for ex­ample the prob­lem of ob­tain­ing or help­ing hu­mans to ob­tain a bet­ter un­der­stand­ing of their val­ues and de­fend­ing their val­ues against ma­nip­u­la­tion from other AIs.

In other words, the mo­tiv­a­tion-com­pet­ence de­com­pos­i­tion seems po­ten­tially very use­ful to me as a way to break down a lar­ger prob­lem into smal­ler parts so it can be solved more eas­ily, but I don’t agree that the ur­gent/​not-ur­gent di­vide lines up neatly with the mo­tiv­a­tion/​com­pet­ence di­vide.

Aside from the prac­tical is­sue of con­fu­sion between dif­fer­ent us­ages of “AI align­ment” (I think oth­ers like MIRI had been us­ing “AI align­ment” in a broader sense be­fore Paul came up with his nar­rower defin­i­tion), even us­ing “AI align­ment” in a con­text where it’s clear that I’m us­ing Paul’s defin­i­tion gives me the feel­ing that I’m im­pli­citly agree­ing to his un­der­stand­ing of how vari­ous sub­prob­lems should be pri­or­it­ized.

Have you any idea about how to test your hy­po­thesis ? How to test for dif­fer­ence between past and present ?

I am en­tirely con­vinced that some people don’t try to go and steal other good idea from the out­group. Even people from «pro­gess­ive» group. I can eas­ily ima­gine that it is a gen­eral tend­ency, and not just some­thing I see in the people in my neigh­bor­hood.

However, I don’t see any­thing con­vin­cing that it is ac­tu­ally get­ting worse. I’m not an his­tor­ian, from what I have heard and un­der­stood of the past cen­tur­ies, open-mind­ness was not gen­er­ally an ad­ject­ive which could de­scribe most of the people who had to live in those time, even if they were some great ex­cep­tions. The time it took to switch from ro­man num­ber to ar­abic num­ber seems to show that “steal­ing good ideas” was not an ap­plied ideal. Thus, I must ad­mit I’m kind of scep­tic about the con­tent of this blog post. Or, to say it in an other way, it’s strangely look­ing sim­ilar to an art­icle in the re­cent trend about “how in­ter­net is cre­at­ing a bubble around you”, but re­writ­ten with ra­tion­al­ist word­ing.

By the way, if there is an easy way to dis­tin­guish good idea from bad idea, I’d love to have a pointer to it. Which would be man­dat­ory to know what idea to ac­tu­ally steal.

The cur­rent im­ple­ment­a­tion of spoiler tags is pretty ex­per­i­mental, and we will prob­ably change how it renders, but the syn­tax should con­tinue work­ing for the in­def­in­ite fu­ture.

Note: In mark­down, the spoiler syn­tax is as fol­lows:

::: spoiler 
This is some spoiler text 
:::

Which should render like this:

This is some spoiler text

text

text

text

text

text

text

text

Thanks! The hope is to write some­thing a bit more com­pre­hens­ive that ex­pands on many of these points, which would be its own post (or se­quence).

Good post! I agree with al­most all of this (I have one ob­jec­tion but it’s not really cent­ral). The bit about how your writ­ing would likely have been more pop­u­lar in the ’60s seems pretty likely.

And when I did try read­ing sci­ence fic­tion from later days, a lot of it struck me as… icky. Neur­oman­cer, bleah, what is wrong with this book, it feels dam­aged, why do people like this, it feels like there’s way too much flash and it ate the sub­stance, it’s show­ing off way too hard.

Amus­ingly, I very re­cently read Neur­oman­cer for the first time… and had al­most ex­actly this re­ac­tion. An un­pop­u­lar opin­ion in our sorts of circles, it seems—which sur­prised me a bit.

I felt some deja vu read­ing this and looked it up. Did any­thing in par­tic­u­lar prompt cross­post­ing it to LW over a year later, or is this just when you got around to it?

+1 on the last para, has re­peatedly been my ex­per­i­ence that the best qual­i­fied can­did­ates for a job were not sure that they were and thought this meant they shouldn’t ap­ply, which is a quite un­for­tu­nate de­fault de­cision.

De­scrip­tion of CHAI is pretty ac­cur­ate. I think it’s a par­tic­u­larly good op­por­tun­ity for people who are con­sid­er­ing grad school as a long-term op­tion: we’re in an ex­cel­lent po­s­i­tion to help people get into top pro­grams, and you’ll also get a sense of what aca­demic re­search cul­ture is like.

We’d like to hire more than one en­gin­eer, and are cur­rently tri­al­ling sev­eral hires. We have a mix­ture of work, some of which is more ML ori­ented and some of which is more in­fra­struc­ture ori­ented. So we’d be will­ing to con­sider ap­plic­ants with lim­ited ML ex­per­i­ence, but they’d need to have strengths in other areas to com­pensate.

If any­one is con­sid­er­ing any of these roles and is un­cer­tain whether they’re a good fit, I’d en­cour­age you to just ap­ply. It doesn’t take much time for you to ap­ply or for the or­gan­isa­tion to do an ini­tial screen­ing. I’ve spoken to sev­eral people who didn’t think they were vi­able can­did­ates for a par­tic­u­lar role, and then turned out to be one of the best ap­plic­ants we’d re­ceived.

I think that LW ra­tion­al­ity has given me a slightly bet­ter un­der­stand­ing of the lim­it­a­tions of ex­perts, so com­pared to the av­er­age smart per­son, I know that I prob­ably shouldn’t rely too much on the new think­ing on the topic, either.

Great. But what should I do in­stead? LW ra­tion­al­ity doesn’t really tell me, so in prac­tice—if I go with the “screw it, I’ll just do some­thing” men­tal­ity, I just fall back into go­ing with the best ex­pert pre­dic­tions any­way.

It seems like LW ra­tion­al­ity would straight­for­wardly tell you that this means you ought to keep your eggs in mul­tiple dif­fer­ent bas­kets rather than in­vest­ing everything in the single top ex­pert opin­ion. (As­sum­ing you’re risk-averse, which it sounds like you are.)

(De­leted sec­tion on why I thought cul­tural gen­eral-in­tel­li­gence soft­ware was not much of the work of AGI:)

...be­cause the soft fi­del­ity of im­pli­cit un­con­scious cul­tural trans­mis­sion can store less seri­ally deep and in­tric­ate al­gorithms than the high-fi­del­ity DNA trans­mis­sion used to store the kind of al­gorithms that ap­pear in com­pu­ta­tional neur­os­cience.

I re­com­mend Ter­rence Deacon’s The Sym­bolic Spe­cies for some good dis­cus­sion of the sur­pris­ing im­port­ance of the shal­low al­gorithms and para­met­ers that can get trans­mit­ted cul­tur­ally. The hu­man-raised chim­pan­zee Kanzi didn’t be­come a hu­man, be­cause that takes deeper and more neural al­gorithms than im­it­at­ing the apes around you can trans­mit, but Kanzi was a lot smarter than other chim­pan­zees in some in­ter­est­ing ways.

But as ne­ces­sary as it may be to avoid feral chil­dren, this kind of shal­low soft-soft­ware doesn’t strike me as some­thing that takes a long time to re­develop, com­pared to hard-soft­ware like the secrets of com­pu­ta­tional neur­os­cience.

Where this be­comes dif­fi­cult is when your con­trol on out­comes is un­cer­tain.

More spe­cific­ally, it’s quite of­ten that it looks like you prob­ably can’t do any­thing to change the situ­ation, but it’s also plaus­ible that if you con­sidered the prob­lem more you would dis­cover some­thing. In that case know­ing what amount of con­sid­er­a­tion is worth it is a hard to de­term­ine thing. Pre­sum­ably there is some sort of di­min­ish­ing re­turns curve you could use, but ob­vi­ously you don’t have ac­cess to that so you have to es­tim­ate the ef­fect­ive­ness of your es­tim­a­tion.

Luck­ily, I think there are huge num­ber of situ­ations where this con­sid­er­a­tion isn’t rel­ev­ant, such as be­ing struck in traffic.

My sub­ject­ive an­ti­cip­a­tion is mol­li­fied by the thought that I’ll prob­ably either never ex­per­i­ence dy­ing or wake up to find that I’ve been in an an­ces­tral sim­u­la­tion.

This feels very close to my moral in­tu­itions, but I cog­nit­ively be­lieve very strongly that things can be bad even if there are no ob­serv­ers left to re­gret that they happened. That fol­lows lo­gic­ally from the moral in­tu­ition that people suf­fer­ing where you can’t see is still bad.

...is that a nat­ural moral in­tu­ition? I don’t think so. That could ex­plain the dis­son­ance.

This is a great com­ment, and maybe it should even be its own post. It cla­ri­fied a bunch of things for me, and I think was the best con­cise ar­gu­ment for “we should try to build some­thing that doesn’t look like an ex­pec­ted util­ity max­im­izer” that I’ve read so far.