1. There is also the uni­ver­sal Girar­dian mimetic failure mode. I don’t know if sum­ma­riz­ing the whole the­ory here makes sense—but it posits that hu­mans want what other hu­mans want, so there is a spiral of ever in­creas­ing de­sire for things and sta­tus. I once wrote an es­say on that in the con­text of in­ter­net dis­cus­sions: https://​​blog.p2pfoun­da­tion.net/​​on­line-con­flict-in-the-light-of-mimetic-the­ory/​​2009/​​11/​​25

2. Another failure mode: the repli­ca­tion crisis in sci­ence—where only new and sur­pris­ing the­ses are be­ing pub­lished, but there is no mechanism for re­in­forc­ing ex­ist­ing the­o­ries. This also hap­pens in so­cial me­dia—peo­ple always want to learn new things. And prob­a­bly more gen­er­ally all the other things from https://​​www.gw­ern.net/​​Littlewood

3. https://​​www.less­wrong.com/​​posts/​​ZdtFBCtixqay5aoWF/​​de­sign-thoughts-for-build­ing-a-bet­ter-kind-of-so­cial-space

We are two AI safety re­searchers (com­bined AF karma: around 200) who have been hap­pily liv­ing as room­mates for the past ~year in Reno, Ne­vada. We have an empty room in our house. The rent for the room is $400/​mo. Some fac­tors to con­sider re: whether you want to come live with us in Reno (ei­ther as a room­mate, or start­ing your own house):

• Reno is about as close to the Bay Area as you can get with­out pay­ing state in­come tax. It costs about $20 to get a bus ticket down to the Bay. The bus has power and wifi, so you can work dur­ing the ~8 hour drive. (I’m not sure how good the wifi is, I try & use it as offline work time, but I tend to get dis­tracted by the beau­tiful scenery.)

• If you have a profoundly gifted child you want to send to school, or want to work with profoundly gifted kids, the nearby David­son Academy is sup­posed to be the best place in the US for this.

• Pre-pan­demic, we had a nice SSC meetup go­ing with a few peo­ple at­tend­ing.

• Short com­mute to Burn­ing Man.

• Po­lice/​com­mu­nity re­la­tions seem OK in Reno (maybe not in the ad­ja­cent city of Sparks though), with cops march­ing as part of anti-po­lice-bru­tal­ity protests. There was some ri­ot­ing down­town the other month but not bad enough to make the na­tional news, and the com­mu­nity came to­gether to clean up af­ter­wards. Our house is far from down­town any­ways.

• There are more gun peo­ple here than in Cal­ifor­nia; al­most 40% of house­holds in Washoe County keep firearms in their homes: http://​​www.city-data.com/​​top2/​​co8.html Un­clear if this is good or bad. Cer­tainly should be eas­ier to get a gun here than in Cal­ifor­nia if you want one. There’s a mix of liber­als and con­ser­va­tives in the area; liber­tar­i­ans seem over­rep­re­sented. There’s a con­fed­er­ate flag bumper sticker or two in our ru­ral neigh­bor­hood :/​​

• There is snow and skiing in the win­ter. Lat­i­tude fairly similar to the Bay Area, so the­o­ret­i­cally sun­light fairly similar as well?

• Reno’s tagline is “The Biggest Lit­tle City in the World”, and it lives up to it in my opinion. At least pre-pan­demic, passerby in Reno seemed way nicer than passerby in the Bay Area.

• There have been some wild­fires this year and we got ad­vi­sory evac­u­a­tions twice. (We’re out in the boon­docks; I don’t be­lieve any­one in the city got an ad­vi­sory evac­u­a­tion.) Air qual­ity has been bet­ter than the Bay Area though.

A lit­tle about our house:

• We’re hop­ing to add AI safety re­searchers to the house, po­ten­tially squeez­ing more than 3 peo­ple into 3 bed­rooms, even­tu­ally split­ting off into a sec­ond house (based on who is form­ing AI safety col­lab­o­ra­tions) and grow­ing the Reno com­mu­nity that way.

• We’re both fairly driven and in­de­pen­dently mo­ti­vated, “no ex­cuses” types as op­posed to the touchy-feely peo­ple who do Cir­cling you meet in the Bay Area.

• Clean­ing up messes is usu­ally not a pri­or­ity at the cur­rent mo­ment but this is sus­cep­ti­ble to change.

• We’re not su­per so­cial. We com­mu­ni­cate us­ing Dis­cord and oc­ca­sion­ally go for hikes, do math/​ma­chine learn­ing pre­sen­ta­tions on a white­board, play board games, drink and watch a movie, cook food and eat it to­gether.

• We have a dish­washer, laun­dry ma­chine, dryer.

Please send a pri­vate mes­sage for more info.

The skin color study con­firms that vi­tamin D doesn’t pro­tect against in­fec­tion. But I don’t see it say­ing any­thing clear about how much harm a per­son suffers if they’re in­fected.

Some­one wanted to know about the out­come of my hair loss re­search so I thought I would quickly write up what I’m plan­ning to try for the next year or so. No word on how well it works yet.

Most of the ideas are from this re­view: https://​​www.karger.com/​​Ar­ti­cle/​​Ful­lText/​​492035

• I’m plan­ning to re­place oral caf­feine con­sump­tion with caf­feinated sham­poo/​con­di­tioner: https://​​www.ama­zon.com/​​gp/​​product/​​B076FHJ3K5/​​ (In my ex­pe­rience caf­feine is ab­sorbed through the skin just fine)

• Gonna start tak­ing this as a source of pro­cyani­din and zinc: https://​​www.ama­zon.com/​​gp/​​product/​​B00K7GIMCU/​​

• I un­der­stand that the largest sin­gle in­gre­di­ent in marine pro­teins is glycine, which also seems to be helpful for im­prov­ing sleep and pre­vent­ing di­a­betes per ex­am­ine.com, so I’m go­ing to start tak­ing 3-4g of glycine be­fore bed ev­ery night: https://​​www.ama­zon.com/​​gp/​​product/​​B01LBILPCG/​​ (Romeo pointed out that tak­ing marine pro­teins har­vested from marine crea­tures might in­volve con­sum­ing ex­cess mer­cury etc. which is an­other rea­son to take syn­thetic glycine.)

• Every­one on hair loss fo­rums seems to love microneedling: https://​​www.ama­zon.com/​​gp/​​product/​​B07JMZXBGR/​​ Here is one guide: https://​​www.red­dit.com/​​r/​​tressless/​​com­ments/​​a660h7/​​un­offi­cial_dermapen­rol­ler_guide/​​

• I thought I might as well try emu oil but I’ve been hav­ing trou­ble sourc­ing stuff that I’m rel­a­tively con­fi­dent is low in trans fats.

• 2% ke­to­cona­zole sham­poo at least un­til my sup­ply runs out.

• Once I’ve been able to figure out how effec­tive all this stuff is, maybe I’ll buy this and see if it seems to make a differ­ence when added to the above in 6 months, oth­er­wise re­turn it. This re­view was op­ti­mistic about lasers, al­though this web­page points out that the pa­ram­e­ters may mat­ter quite a bit; I chose the iRe­store based on low cost rather than op­ti­mism about its pa­ram­e­ters.

I think this should be lower cost, safer, and less sketchy than the big 3, but plau­si­bly less effec­tive on ex­pec­ta­tion; let me know if you dis­agree.

Logic and rea­son in­di­cate the ro­bust­ness of a claim, but you can have lots of ro­bust, mu­tu­ally-con­tra­dic­tory claims. A ro­bust claim is one that con­tra­dicts nei­ther it­self nor other claims it as­so­ci­ates with. The other half is how well it res­onates with peo­ple. Res­o­nance in­di­cates how at­trac­tive a claim is through au­thor­ity, con­sen­sus, scarcity, po­etry, or what­ever else.

Sur­vive and spread through ro­bust­ness and res­o­nance. That’s what a strong claim does. You can state that you’ll only let a claim spread into your mind if it’s true, but the fact that it’s so com­mon for two such peo­ple to hold con­tra­dic­tory claims in­di­cates that their real met­ric is much weaker than truth. I’ll posit that the real met­ric in such sce­nar­ios is ro­bust­ness.

Not all dis­agree­ments will sep­a­rate cleanly into true/​false cat­e­go­riza­tions. Godel proved that one.

There could be ways of mak­ing it le­gal given that we’re a non-profit with some­what aca­demic in­ter­ests. (By “mak­ing” I mean ac­tu­ally chang­ing the law or get­ting a No-Ac­tion Let­ter.) Most peo­ple who do gam­bling on­line do it for profit, which is where things get tricky.

I’ve been kick­ing around the idea of a ‘ra­tio­nal­ist cult’, and am in­ter­ested in the monastery idea.

My at­tempt to sum­ma­rize the al­ign­ment con­cern here. Does this seem a rea­son­able gloss?

It seems plau­si­ble that com­pet­i­tive mod­els will not be trans­par­ent or in­tro­spectable. If you can’t see how the model is mak­ing de­ci­sions, you can’t tell how it will gen­er­al­ize, and so you don’t get very good safety guaran­tees. Or to put it an­other way, if you can’t in­ter­act with the way the model is think­ing, then you can’t give a rich enough re­ward sig­nal to guide it to the re­gion of model space that you want

I have already replied once but my post seems to have been deleted so I will try again.

I am un­able to de­tect any nu­ance in your post. per­haps it would be sim­pler if you were to ex­plic­itly say ex­actly what you mean.

thank you.

Most im­por­tantly, the suc­cess of the scheme re­lies on the cor­rect­ness of the prior over helper mod­els (or else the helper could just be an­other copy of GPT-Klin­gon)

I’m not sure I un­der­stand this. My un­der­stand­ing of the worry: what if there’s some equil­ibrium where the model gives wrong ex­pla­na­tions of mean­ings, but I can’t tell us­ing just the model to give me mean­ings.

But it seems to me that hav­ing the hu­man in the loop do­ing pre­dic­tion helps a lot, even with the same prior. Like, if the mean­ings are wrong, then the user will just not pre­dict the cor­rect word. But maybe this is not enough cor­rec­tive data?

I like to imag­ine a fu­ture LessWrong with 1 mil­lion users, and not one press­ing the but­ton. That would be very in­spiring and a strong sig­nal of be­ing a high trust cul­ture.

Thanks for your explanation

This works great as an an­swer!

Ac­cord­ing to Fedex track­ing, on Thurs­day, I will have a Biovyzr. I plan to im­me­di­ately start test­ing it, and write a re­view.

What tests would peo­ple like me to perform?

Tests that I’m already plan­ning to perform:

To test its pro­tec­tive­ness, the main test I plan to perform is a mod­ified Bit­trex fit test. This is where you cre­ate a bit­ter-tast­ing aerosol, and con­firm that you can’t taste it. The nor­mal test pro­ce­dure won’t work as-is be­cause it’s too large to use a plas­tic hood, so I plan to go into a small room, and have some­one (wear­ing a res­pi­ra­tor them­selves) spray co­pi­ous amounts of Bit­trex at the in­put fan and at any spots that seem high-risk for leaks.

To test that air ex­it­ing the Biovyzr is be­ing filtered, I plan to put on a reg­u­lar N95, and use the in­side-out glove to cre­ate Bit­trex aerosol in­side the Biovyzr, and see whether some­one in the room with­out a mask is able to smell it.

I will ver­ify that the Biovyzr is pos­i­tive-pres­sure by run­ning a straw through an edge, cre­at­ing an ar­tifi­cial leak, and see­ing which way the air flows through the leak.

I will have ev­ery­one in my house try wear­ing it (5 adults of varied sizes), have them all rate its fit and com­fort, and get as many of them to do Bit­trex fit tests as I can.

ah, I definitely agree yes, thank you for point­ing that out. it was a quote I took from old notes back when I had a browser ex­ten­sion that would au­to­mat­i­cally change all pro­nouns to gen­der neu­tral pro­nouns, and I didn’t know at the time I would share that quote pub­li­cly, which is the source of the er­ror. I fixed it.

Great com­ment, nat­u­rally. I ap­pre­ci­ate your epistemic sta­tus quite a bit.

I think I want to re­spond to the idea that it’s con­tra­dic­tory and bad to sig­nal that the Petrov Day but­ton is se­ri­ous and sig­nal that it’s fun. A few ex­am­ples:

• HPMOR is a book about grow­ing up, failure, and death. It’s also hilar­i­ous and riv­et­ing.

• Un­song is 50% puns. It also con­tains a chap­ter de­scribing hell and tor­ture in some de­tail.

• Embed­ded Agency is re­search done down­stream of the po­ten­tial for ad­vanced op­ti­miz­ers to lead to an ex­is­ten­tial catas­tro­phe for hu­mans. It’s also a cute and col­or­ful car­toon.

• Pre­vi­ously war­ring coun­tries of­ten come to­gether and have their sports teams play. It re­ally mat­ters that they don’t cheat and play hon­or­ably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.

• Some an­i­mals sheath their claws for dom­i­nance fights, where the los­ing player loses real sta­tus but isn’t phys­i­cally harmed. Again, it’s a game, and it’s in some ways play. And it’s also se­ri­ous.

• I’m here try­ing to build a com­mu­nity around the art of ra­tio­nal­ity. We also do an April Fools’ joke ev­ery year, like that time we made ev­ery­one’s font size pro­por­tional to their to­tal karma for a day.

• Peo­ple have birth­day par­ties, and their friends show up. And it means some­thing for your friends to show up. It’s a party, and it’s also a true sig­nal of friend­ship and be­ing there for your friend, and you can be dis­ap­pointed in them not show­ing up, even if it’s ‘fun’ and ‘just a party’.

I don’t think it’s con­tra­dic­tory to care about some­thing deeply and to be playful with it. As with Feyn­man and physics, Eliezer and HPMOR, Scott and Un­song, Abram/​Scott and Embed­ded Agency; also, LessWrong and Petrov Day.

Re­lat­edly, (part of) you said that some­times I sig­naled that this was se­ri­ous, and some­times I sig­naled that it was ‘just a game’. I think this is in­cor­rect. I sig­naled it was playful, but never un­se­ri­ous, never ‘just’ or ‘merely’ a game. Every­thing I wrote was about trust, honor, ex­tinc­tion, grat­i­tude, mourn­ing, and be­ing Beyond the Reach of God. I didn’t write any­thing that sug­gested you should con­sider press­ing the but­ton, or was se­cretly wink­ing at the au­di­ence. Tak­ing down the site is sym­bolic, but that doesn’t mean it’s ironic or a joke, they’re to­tally differ­ent. We’re sym­bol­iz­ing world-end­ing de­struc­tive tech­nol­ogy, with a much lesser but still de­struc­tive tech­nol­ogy. I care about this tra­di­tion a lot, and it was part of what was in­volved and in ev­ery­thing I wrote.

I feel like it takes a very cyn­i­cal prior to read at ev­ery­thing I wrote, con­sider that I ac­tu­ally cared, then go “Yeah, he prob­a­bly doesn’t mean this, why would some­one ac­tu­ally care about this, he’s prob­a­bly jok­ing?” I don’t think any­one has that prior… I think they more have a prior that peo­ple rarely ac­tu­ally care about things, and so when they look at some­thing that was meant straight­for­wardly and un-iron­i­cally, the hy­poth­e­sis isn’t even brought up to at­ten­tion.

But a lot of peo­ple got it. Most, I think. I got a bunch of short re­sponse mes­sages to get­ting the codes (and some long ones) say­ing things like “I’m hon­ored to be en­trusted with the launch codes.” and “Roger that, gen­eral. I won’t let you down!” and “I’m hon­ored that you gave me the op­por­tu­nity” and “Awe­some, thanks! I love the warm glow of not burn­ing the com­mons.” I love get­ting these mes­sages. And I love that it worked out last year.

It’s a dis­tinct ar­gu­ment to say that I was un­suc­cess­ful at as­sur­ing that was com­mu­ni­cated be­fore some­one took ac­tion, which this is some ev­i­dence of. Like, last year some­one vis­ited the site, pressed the but­ton, and en­tered a string of ze­ros then hit ‘sub­mit’, be­fore find­ing out what the but­ton even did, and with­out read­ing the an­nounce­ment post. (We re­sponded to that this year by hav­ing two sen­tences right un­der­neath the but­ton say­ing what was hap­pen­ing.)

I take se­ri­ously the charge that users like Chris would’ve got­ten it if I had re-writ­ten the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tra­di­tion; I can also do that, write a post called “Why the Petrov Day Big Red But­ton?”, and link to it from ev­ery­where next year.

But… well, there’s more to say, but I have to go for now. I’ll add that it was my re­spon­si­bil­ity to pick the peo­ple and write the an­nounce­ments. I en­trusted 125 peo­ple with codes last year, and suc­ceeded. I tried to do 270 this year, and I failed. I’m writ­ing a post­mortem, and I will work hard to en­sure the site doesn’t go down next year.

This is only true for triv­ial val­ues, e.g. “I ter­mi­nally value hav­ing this spe­cific world model”.

For most util­ity schemes (In­clud­ing, crit­i­cally, that of hu­mans), the su­per­ma­jor­ity of the pur­pose of mod­els and be­liefs is in­stru­men­tal. For ex­am­ple, mak­ing bet­ter pre­dic­tions, us­ing less com­put­ing power, etc.

In fact, hu­mans who do not rec­og­nize this fact and stick to be­liefs or mod­els be­cause they like them are profoundly ir­ra­tional. If the sky is blue, I wish to be­lieve the sky is blue, and so on. So, as­sum­ing that only pre­dic­tion is valuable is not ques­tion beg­ging- I sus­pect you already agreed with this and just didn’t re­al­ize it.

In the sense that be­liefs (and the mod­els they’re part of) are in­stru­men­tal goals, any spe­cific be­lief is “un­nec­es­sary”. Note the quo­ta­tions around “un­nec­es­sary” in this com­ment and the com­ment you’re re­ply­ing to. By “un­nec­es­sary” I mean the choice of which be­liefs and which model to use is sub­ject to the whims of which is more in­stru­men­tally valuable- in prac­tice, a com­plex trade­off be­tween pre­dic­tive ac­cu­racy and com­pu­ta­tional de­mands.

I have plenty of times heard of vari­ables be­ing in­fini­tive in physics and I have seen peo­ple do calcu­lus with in­finitvely small num­bers.

This fits with the idea that mean­ing comes from plea­sure, and that great plea­sure can be worth a fair amount of pain to achieve. The pain drains mean­ing away, but the re­deem­ing fac­tor is that it can serve as a test of the mag­ni­tude of plea­sure, and gen­er­ate plea­surable sto­ries in the fu­ture.

An im­por­tant counter ar­gu­ment to my hy­poth­e­sis is how we may find a priv­ileged “high road” to suc­cess and plea­sure to be less mean­ingful. This at first might seem to sug­gest that we do in­her­ently value pain.

In fact, though, what frus­trates peo­ple about peo­ple born with a silver spoon in their mouths is that so­ciety seems set up to en­sure their plea­sure at an­other’s ex­pense.

It’s not their suc­cess or plea­sure we dis­like. It’s the bar­ri­ers and pain that we think it’s con­tex­tu­al­ized in. If plea­sure for one means pain for an­other, then of course we find the plea­sure to be less mean­ingful.

So this isn’t about short-term pain avoidance. It’s about long-term, over­all, wise and sys­temic pur­suit of plea­sure.

And that plea­sure must be not only in the phys­i­cal ex­pe­riences we have, but in the sto­ries we tell about it—the way we in­ter­pret life. We should look at it, and see that it is good.

If peo­ple are wire­head­ing, and we look at that ten­dency and it causes us great dis­plea­sure, that is in­deed an ar­gu­ment against wire­head­ing.

We need to un­der­stand that there’s no sin­gle bucket where plea­sure can ac­cu­mu­late. There is a psy­cholog­i­cal re­ward sys­tem where plea­sure is eval­u­ated ac­cord­ing to the sen­sory in­put and brain state.

Utili­tar­ian he­do­nism isn’t just about nerve end­ings. It’s about how we in­ter­pret them. If we have a ma­jor aes­thetic ob­jec­tion to wire­head­ing, that counts from where we’re stand­ing, no mat­ter how much you rachet up the pre­sumed plea­sure of wire­head­ing.

The same goes re­cur­sively for any “hack” that could jus­tify wire­head­ing. For ex­am­ple, say you posited that wire­head­ing would be seen as morally good, if only we could find a catchy moral jus­tifi­ca­tion for it.

So we let our finest AI su­per­in­tel­li­gences get to work pro­duc­ing one. In­deed, it’s so catchy that the en­tire hu­man pop­u­la­tion ac­quiesces to wire­head­ing.

Well, if we take offense to the prospect of let­ting the AI su­per­in­tel­li­gence in­fect us with a catchy pro-wire­head­ing meme, then that’s a ma­jor point against do­ing so.

In gen­eral “It pleases or dis­pleases me to find ac­tion X moral” is a valid moral ar­gu­ment—in­deed, the only one there is.

The way moral change hap­pens is by mak­ing moral ar­gu­ments or hav­ing moral ex­pe­riences that in them­selves are pleas­ing or dis­pleas­ing.

What’s needed, then, for moral change to hap­pen, is to find a pleas­ing way to spread an idea that is it­self pleas­ing to adopt—or un­pleas­ant to aban­don. To re­main, that idea needs to gen­er­ate plea­sure for the sub­scriber, or to gen­er­ate dis­plea­sure at the prospect of aban­don­ing it in fa­vor of a com­pet­ing moral scheme.

To be­lieve in some no­tion of moral truth or progress re­quires be­liev­ing that the psy­cholog­i­cal re­ward mechanism we have at­tached to moral­ity cor­re­sponds best with moral schemes that ac­cord with moral truth.

An ar­gu­ment for that is that true ideas are eas­iest to fash­ion into a co­her­ent, sim­ple ar­gu­ment. And true ideas best al­low us to in­ter­face with re­al­ity to ad­van­tage. Be­ing good tends to make you get along with oth­ers bet­ter than be­ing bad, and that’s a more pleas­ant way to ex­ist.

Hence, even though strong cases can be con­structed for im­moral be­hav­ior, truth and good­ness will tend to win in the arms race for the most pleas­ing pre­sen­ta­tion. So we can en­joy the idea that there is moral progress and ob­jec­tive moral truth, even though we make our moral de­ci­sions merely by pur­su­ing plea­sure and avoid­ing pain.

See also: The Best Text­books on Every Subject