• There is a the­ory of “the path of least re­sis­tance” that im­plies the one should go with the flow. With that in mind, how do you con­tinue to nur­ture the growth re­sult­ing from challenges? Does the ra­tio­nale of the path of least re­sis­tance con­flict with the challenges of life that are re­quired for change?

• Does the ra­tio­nale of the path of least re­sis­tance con­flict with the challenges of life that are re­quired for change?

No:

“Life pro­vides enough prob­lems with­out us mak­ing more.”

Yes:

• If you’re up to your neck in wa­ter, (maybe) it’s time to stop walk­ing, and start swim­ming.

• If you don’t have enough challenges you can make what you’re do­ing more difficult—or go find some­thing bet­ter (harder) to do.

• Imag­ine you are at the beach. If you swim out far enough, you can surf back in. If you get caught in a rip tide/​rip cur­rent, the cur­rent may be too strong to fight.*

• Me­taphor­i­cally, just be­cause you’re go­ing in the di­rec­tion of the cur­rent doesn’t mean you have to just stay afloat** - you can swim.

* - swim per­pen­dicu­lar to the beach, then go back in.

** This be­hav­ior also seems char­ac­ter­is­tic of some­thing hav­ing gone wrong. (If this is the case, iden­ti­fy­ing and ad­dress­ing the prob­lem may be as im­por­tant as try­ing to change tack—which is not a 1d move in the literal world.)

• Yes. There­fore you should not do that. “Least re­sis­tance” and “go­ing with the flow” are for those who want to re­main asleep, to do noth­ing, to be noth­ing.

• Would you say the same of “the path” in emo­tions and re­la­tion­ships?

• Hilbert’s Mo­tel improvement

This ho­tel is 2 star at best, imag­ine hav­ing to pack up your stuff ev­ery time the ho­tel re­ceives a new guest? I’ve de­cided to fix that. The ho­tel still has in­finite rooms and guests but this time ev­ery other room is un­oc­cu­pied which pre­pares the ho­tel for an in­finite amount of new vis­i­tors with­out in­con­ve­nienc­ing the cur­rent res­i­dence.

• But isn’t the whole point that the ho­tel is full ini­tially, and yet can ac­cept more guests?

• Yeah, the ho­tel be­ing always half full no mat­ter how many guests it has doesn’t seem as cool.

• As soon as one more guest shows up it’s more than half full.

• But if an in­finite num­ber of guests show up, then they might have to travel an in­finite dis­tance to their ho­tel room.

• If the num­ber of guests is countable (which is the usual as­sump­tion in Hilbert’s setup), then ev­ery guest will only have to travel a finite (albeit un­bound­edly long) dis­tance be­fore they reach their room.

• With cli­mate change get­ting worse by the day we need to switch to sus­tain­able en­ergy sources sooner rather than later. The new Molten salt re­ac­tors are small, clean and safe, but still carry the stigma of nu­clear en­ergy. Since these re­ac­tors (like oth­ers) can use old nu­clear waste as a fuel source, I sug­gest we re­brand them to “Nu­clear Waste Eaters” and give them (or a com­pany that makes them) a logo in the vein of this quick sketch I made: https://​​pos­timg.cc/​​jWy3PtjJ

Hope­fully a re­brand­ing to “thing get­ting rid of the thing you hate, also did you know it’s clean and safe” will get peo­ple more mo­ti­vated for these kinds of en­ergy sources.

• Any­thing suffi­ciently far enough away from you is causally iso­lated from you. Be­cause of the fun­da­men­tal con­straints of physics, in­for­ma­tion from there can never reach here, and vice versa. you may as well be in sep­a­rate uni­verses.

The perfor­mance of AlphaGo got me think­ing about al­gorithms we can’t ac­cess. In the case of AlphaGo, we im­ple­mented the al­gorithm (AlphaGo) which dis­cov­ered some strate­gies we could never have cre­ated. (Go Master Ke Jie fa­mously said “I would go as far as to say not a sin­gle hu­man has touched the edge of the truth of Go.”)

Per­haps we can imag­ine a sort of “log­i­cal causal iso­la­tion.” An al­gorithm is log­i­cally causally iso­lated from us if we can­not dis­cover it (e.g. in the case of the Go strate­gies that AlphaGo used) and we can­not spec­ify an al­gorithm to dis­cover it (ex­cept by ran­dom ac­ci­dent) given finite com­pu­ta­tion over a finite time hori­zon (i.e. in the life­time of the ob­serv­able uni­verse).

Im­por­tantly, we can de­vise al­gorithms which search the en­tire space of al­gorithms (e.g. gen­er­ate all per­mu­ta­tions all pos­si­ble strings of bits less than length n as n ap­proaches in­finity), but there’s lit­tle rea­son to ex­pect that such a strat­egy will re­sult in any use­ful out­puts of some finite length (there ap­pear to be enough atoms in the uni­verse () to rep­re­sent all pos­si­ble al­gorithms of length .

There’s one im­por­tant weak­ness in LCI (that doesn’t ex­ist in Phys­i­cal Causal Iso­la­tion). We can ran­domly jump to al­gorithms of ar­bi­trary lengths. This stipu­la­tion gives us the weird abil­ity to pull stuff from out­side our LCI-cone into it. Un­for­tu­nately, we can­not do so with the ex­pec­ta­tion of ar­riv­ing at a use­ful al­gorithm. (There’s an in­ter­est­ing ques­tion about which I haven’t yet thought about the dis­tri­bu­tion of use­ful al­gorithms of a given length.) Hence we must add the caveat to our defi­ni­tion of LCI “ex­cept by ran­dom ac­ci­dent.”

We aren’t LCI’d from the strate­gies AlphaGo used, be­cause we cre­ated AlphaGo and AlphaGo dis­cov­ered those strate­gies (even if hu­man Go mas­ters may never have dis­cov­ered them in­de­pen­dently). I won­der what al­gorithms ex­ist be­yond not just our hori­zons, but the hori­zons of all the al­gorithms which de­scend from ev­ery­thing we are able to com­pute.

• 2 things nec­es­sary for an al­gorithm to be use­ful:

• If it’s not fast enough, it doesn’t mat­ter how good it is

• If we don’t know what it’s good for, it doesn’t mat­ter how good it is (un­til we figure that out)

Un­for­tu­nately, we can­not do so with the ex­pec­ta­tion of ar­riv­ing at a use­ful al­gorithm.
• Part of the is­sue with this might be pro­grams that don’t work or do any­thing (Beyond the triv­ial, it’s not clear how to se­lect for this, out­side of some­thing like AlphaGo.)

• If it’s not fast enough, it doesn’t mat­ter how good it is

Sure! My brute-force bit­wise al­gorithm gen­er­a­tor won’t be fast enough to gen­er­ate any al­gorithm of length 300 bits, and our uni­verse prob­a­bly can’t sup­port any rep­re­sen­ta­tion of any al­gorithm of length greater than (the num­ber of atoms in the ob­serv­able uni­verse) ~ 10^82 bits. (I don’t know much about physics, so this could be very wrong, but think of it as a use­ful bound. If there’s a bet­ter one (e.g. num­ber of Planck vol­umes in the ob­serv­able uni­verse), sub­sti­tute that and carry on, and also please let me know!)

Part of the is­sue with this might be pro­grams that don’t work or do any­thing (Beyond the triv­ial, it’s not clear how to se­lect for this, out­side of some­thing like AlphaGo.)

Another class of al­gorithms that cause prob­lems are those that don’t do any­thing use­ful for some num­ber of com­pu­ta­tions, af­ter which they be­gin to out­put some­thing use­ful. We don’t re­ally get to know if they will halt, so if the use­ful struc­ture emerges af­ter some num­ber of steps, we may not be com­mit­ted to or able to run it that long.

• I’m not a physi­cist ei­ther, but quan­tum me­chan­ics might change the limits. (If it scales, though this might leave in­put and out­put limits; if the quan­tum com­puter can’t store the out­put in clas­si­cal mode, then it’s abil­ity to run the pro­gram prob­a­bly doesn’t mat­ter. This might make less effi­cient crypto sys­tems more se­cure, by virtue of size.*)

Want your key more se­cure? Length.

• You would hope that peo­ple ac­tu­ally saw steel­man­ning as an ideal to fol­low. If that was ever true, the corona pan­demic and the policy re­sponse seem to have kil­led the de­mand for this. It seems to be­come ac­cept­able to at­tribute just any kind of seem­ingly-wrong be­hav­ior to ei­ther in­cred­ible stu­pidity or in­cred­ible mal­ice, both prov­ing that all in­sti­tu­tions are com­pletely bro­ken.

• I like the word “in­sti­tu­ri­ons”. Some mix of in­sti­tu­tions, in­tu­itions, and cen­tu­ri­ons, and I agree that they’re com­pletely bro­ken.

• :-) Thanks. But I cor­rected it.

• For the fore­see­able fu­ture, it seems that any­thing I might try to say to my UK friends about any­thing to do with LW-style think­ing is go­ing to be met with “but Do­minic Cum­mings”. Three sep­a­rate in­stances of this in just the last few days.

• Can you give some ex­am­ples of “LW-style think­ing” that they now as­so­ci­ate with Cum­mings?

• On Twit­ter I linked to this saying

Ba­sic skills of de­ci­sion mak­ing un­der un­cer­tainty have been sorely lack­ing in this crisis. Oxford Univer­sity’s Fu­ture of Hu­man­ity In­sti­tute is build­ing up its Epi­demic Fore­cast­ing pro­ject, and needs a pro­ject man­ager.

Re­sponse:

I’m hon­estly strug­gling with a po­lite re­sponse to this. Here in the UK, Do­minic Cum­mings has tried a Less Wrong ap­proach to policy mak­ing, and our death rate is ter­rible. This idea that a solu­tion will some­how spring from left-field mav­er­ick think­ing is ac­tu­ally lethal.
• Huh. Wow.

• Seems like a good dis­cus­sion could be had about long-term pre­dic­tions and how much ev­i­dence there is to be had in short-term poli­ti­cal fluc­tu­a­tions. The Cum­mings silli­ness vs un­prece­dented im­mi­gra­tion re­stric­tions—which is likely to have im­pact 5 years from now?

• # [Book Re­view] Sur­fing Uncertainty

Sur­fing Uncer­tainty is about pre­dic­tive cod­ing, the the­ory in neu­ro­science that each part of your brain at­tempts to pre­dict its own in­puts. Pre­dic­tive cod­ing has lots of po­ten­tial con­se­quences. It could re­solve the prob­lem of top-down vs bot­tom-up pro­cess­ing. It cleanly unifies lots of ideas in psy­chol­ogy. It even has im­pli­ca­tions for the con­tinuum with autism on one end and schizophre­nia on the other.

The most promis­ing thing about pre­dic­tive cod­ing is how it could provide a math­e­mat­i­cal for­mu­la­tion for how the hu­man brain works. Math­e­mat­i­cal for­mu­la­tions are great be­cause once they let you do things like falsify ex­per­i­ments and simu­late things on com­put­ers. But while Sur­fing Uncer­tainty goes into many of the po­ten­tial im­pli­ca­tions of pre­dic­tive cod­ings, the au­thor never ham­mers out ex­actly what “pre­dic­tion er­ror” means in quan­tifi­able ma­te­rial terms on the neu­ronal level.

This book is a re­it­er­a­tion of the sci­en­tific con­sen­sus[1]. Judg­ing by the to­tal ab­sense of math­e­mat­i­cal equa­tions on the Wikipe­dia page for pre­dic­tive cod­ing, I sus­pect the book never defines “pre­dic­tion er­ror” in math­e­mat­i­cally pre­cise terms be­cause no such defi­ni­tion ex­ists. There is no sci­en­tific con­sen­sus.

Per­haps I was dis­ap­pointed with this book be­cause my ex­pec­ta­tions were too high. If we could write equa­tions for how the hu­man brain performs pre­dic­tive pro­cess­ing then we would be sig­nifi­cantly closer to build­ing an AGI than where we are right now.

1. The book con­tains 47 pages of sci­en­tific cita­tions. ↩︎

• Has any­one tried to work with neu­ral net­works pre­dict­ing the weights of other neu­ral net­works? I’m think­ing about that in the con­text of some­thing like sub­sys­tem al­ign­ment, e.g. in an RL set­ting where an agent first learns about the en­vi­ron­ment, and then cre­ates the sub­agent (by out­putting the weights or some em­bed­ding of its policy) who ac­tu­ally ob­tains some reward

• Ob­ser­va­tion: It should gen­er­ally be safe to for­bid non-ter­mi­na­tion when search­ing for pro­grams/​al­gorithms.

In prac­tice, all use­ful al­gorithms ter­mi­nate: If you know that you’re deal­ing with a semi-de­cid­able thing and do­ing se­ri­ous work, you’ll ei­ther (a) add a hard cut­off, or (b) struc­ture the al­gorithm into a bounded step func­tion and a con­trol­ler that de­cides whether or not to run for an­other step. That trans­for­ma­tion is not adding sig­nifi­cant over­head size-wise, so you’re bound to find a ter­mi­nat­ing al­gorithm “near” a non-ter­mi­nat­ing one!

Sure, that slightly changes the in­ter­face – it’s now al­lowed to abort with “don’t know”, but that’s a trans­for­ma­tion that you likely would have ap­plied any­way. Even if you con­sider that a draw­back, not hav­ing to deal with po­ten­tially non-ter­mi­nat­ing pro­grams /​ be­ing able to use a de­scrip­tion for­mat that can­not rep­re­sent non-ter­mi­nat­ing forms should more than make up for that.

(I just no­ticed this while think­ing about how to best write some­thing in Coq (and de­cid­ing on ter­mi­na­tion by “fuel limit”), af­ter AABoyles’ short­form on log­i­cal causal iso­la­tion with its trag­i­cally sim­ple bit-flip search had re­cently made me think about pro­gram enu­mer­a­tion again…)

• I had a very use­ful con­ver­sa­tion with some­one about how and why I am ram­bly. (I ram­bled a lot in the con­ver­sa­tion!).

Dis­claimer: I am not mak­ing much effort to not ram­ble in this post.

A cou­ple take­aways:

1. Work­ing Me­mory Limits

One key prob­lem is that I in­tro­duce so many points, sub­points, and sub­threads, that I over­whelm peo­ple’s work­ing mem­ory (where hu­man work­ing mem­ory limits is roughly “4-7 chunks”).

It’s sort of em­bar­rass­ing that I didn’t con­cretely think about this be­fore, be­cause I’ve spent the past year SPECIFICALLY think­ing about work­ing mem­ory limits, and how they are the key bot­tle­neck on in­tel­lec­tual progress.

So, one new habit I have is “when­ever I’ve in­tro­duced more than 6 points to keep track of, stop and and figure out how to con­dense the work­ing tree of points down to <4.

(Ideally, I also keep track of this in ad­vance and word things more sim­ply, or give bet­ter sign­post­ing for what over­all point I’m go­ing to make, or why I’m talk­ing about the things I’m talk­ing about)

...

2. I just don’t finish sente

I fre­quently don’t finish sen­tences, whether in per­son voice or in text (like emails). I’ve known this for awhile, al­though I kinda for­got re­cently. I switch abruptly to a new sen­tence when I re­al­ize the cur­rent sen­tence isn’t go­ing to ac­com­plish the thing I want, and I have a Much Shinier Sen­tence Over Here that seems much more promis­ing.

But, peo­ple don’t un­der­stand why I’m mak­ing the leap from one half-finished thought to an­other.

So, an­other sim­ple habit is “make sure to finish my god damn sen­tences, even if I be­come dis­ap­pointed in them halfway through”

...

3. Use Mind­ful Cog­ni­tion Tun­ing to train on *what is easy for peo­ple to fol­low*, as well as to im­prove the cre­ativity/​use­ful­ness of my thoughts.

I’ve always been ram­bly. But a thing that I think has made me EVEN MORE ram­bly in the past 2 years is a mind­ful-think­ing-tech­nique, where you no­tice all of your thoughts on the less-than-a-sec­ond level, so that you can no­tice which thought pat­terns are use­ful or anti-use­ful.

This has been re­ally pow­er­ful for im­prov­ing my thought-qual­ity. I’m fairly con­fi­dent that I’ve be­come a bet­ter pro­gram­mer and bet­ter thinker be­cause of it.

But, it in­tro­duces even more meta-thoughts for me to no­tice while I’m ar­tic­u­lat­ing a sen­tence, which dis­tract me from the sen­tence it­self.

What I re­al­ized last week­end was: I can use Mind­ful Cog­ni­tion to no­tice what types of thoughts/​sen­tences are use­ful for *other peo­ple’s com­pre­hen­sion of me*, not just how use­ful m origi­nal thought pro­cesses are.

The whole point of the tech­nique is to im­prove your feed­back loop (both speed and aware­ness), which makes it eas­ier to de­liber­ate prac­tice. I think if I just ap­ply that to­wards Be­ing More Com­pre­hen­si­ble, it’ll change from be­ing a li­a­bil­ity in ram­bli­ness to an as­set.

• re work­ing mem­ory: never thought of it dur­ing con­ver­sa­tions, in­ter­est­ing. it seems that we some­time hold the nodes of the con­ver­sa­tion tree to go back to them af­ter­ward. and maybe if you’re in­tro­duc­ing new con­cepts while you’re talk­ing peo­ple need to hold those defi­ni­tions in work­ing mem­ory as well.

• Thoughts on Ryan Carey’s In­cor­rigi­bil­ity in the CIRL Frame­work (I am go­ing to try to post these semi-reg­u­larly).

• This spe­cific situ­a­tion looks un­re­al­is­tic. But it’s not re­ally try­ing to be too re­al­is­tic, it’s try­ing to be a coun­terex­am­ple. In that spirit, you could also just use , which is a re­ward func­tion parametrized by that gives the same be­hav­ior but stops me from say­ing “Why Not Just set ”, which isn’t the point.

• How some­thing like this might ac­tu­ally hap­pen: you try to have your be a com­pli­cated neu­ral net­work that can ap­prox­i­mate any func­tion. But you butcher the im­ple­men­ta­tion and get some­thing ba­si­cally ran­dom in­stead, and this can­not ap­prox­i­mate the real hu­man re­ward.

• An im­por­tant in­sight this high­lights well: An off-switch is some­thing that you press only when you’ve pro­grammed the AI badly enough that you need to press the off-switch. But if you’ve pro­grammed it wrong, you don’t know what it’s go­ing to do, in­clud­ing, pos­si­bly, its off-switch be­hav­ior. Make sure you know un­der which as­sump­tions your off-switch will still work!

• As­sign­ing high value to shut­ting down is in­cor­rigible, be­cause the AI shuts it­self down. What about as­sign­ing high value to be­ing in a but­ton state?

• The pa­per con­sid­ers a situ­a­tion where the shut­down but­ton is hard­coded, which isn’t enough by it­self. What’s re­ally hap­pen­ing is that the hu­man ei­ther wants or doesn’t want the AI to shut down, which sounds like a term in the hu­man re­ward that the AI can learn.

• One way to do this is for the AI to do max­i­mum like­li­hood with a prior that as­signs 0 prob­a­bil­ity to the hu­man er­ro­neously giv­ing the shut­down com­mand. I sus­pect there’s some­thing less hacky re­lated to set­ting an ap­pro­pri­ate prior over the re­ward as­signed to shut­ting down.

• The foot­note on page 7 con­fuses me a bit—don’t you want the AI to always defer to the hu­man in but­ton states? The an­swer feels like it will be clearer to me if I look into how “ex­pected re­ward if the but­ton state isn’t avoided” is calcu­lated.

• Also I did just jump into this pa­per. There are prob­a­bly lots of in­ter­est­ing things that peo­ple have said about MDPs and CIRLs and Q-val­ues that would be use­ful.

• Thoughts on Dy­lan Had­field-Menell et al.’s The Off-Switch Game.

• I don’t think it’s quite right to call this an off-switch—the model is fully gen­eral to the situ­a­tion where the AI is choos­ing be­tween two al­ter­na­tives A and B (nor­mal­ized in the pa­per so that U(B) = 0), and to me an off-switch is a hard­ware over­ride that the AI need not want you to press.

• The wis­dom to take away from the pa­per: An AI will vol­un­tar­ily defer to a hu­man—in the sense that the AI thinks that it can get a bet­ter out­come by its own stan­dards if it does what the hu­man says—if it’s un­cer­tain about the util­ities, or if the hu­man is ra­tio­nal.

• This whole setup seems to be some­what su­per­seded by CIRL, which has the AI, uh, causally find by learn­ing its value from the hu­man ac­tions, in­stead of ev­i­den­tially(?) do­ing it by tak­ing de­ci­sions that hap­pen to land it on ac­tion A when is high be­cause it’s act­ing in a weird en­vi­ron­ment where a hu­man is pre­sent as a side-con­straint.

• Could some wis­dom to gain be that the high-var­i­ance high-hu­man-ra­tio­nal­ity is some­thing of an ex­pla­na­tion as to why CIRL works? I should read more about CIRL to see if this is needed or helpful and to com­pare and con­trast etc.

• Why does the re­ward gained drop when un­cer­tainty is too high? Be­cause the prior that the AI gets from es­ti­mat­ing the hu­man re­ward is more ac­cu­rate than the hu­man de­ci­sions, so in too-high-un­cer­tainty situ­a­tions it keeps mis­tak­enly defer­ring to the flawed hu­man who tells it to take the worse ac­tion more of­ten?

• The ver­bal de­scrip­tion, that the hu­man just types in a nois­ily sam­pled value of , is some­what strange—if the hu­man has ex­plicit ac­cess to their own util­ity func­tion, they can just take the best ac­tions di­rectly! In prac­tice, though, the AI would learn this by look­ing at many past hu­man ac­tions (there’s some CIRL!) which does seem like it plau­si­bly gives a more ac­cu­rate policy than the hu­man’s (ht Should Robots Be Obe­di­ent).

• The hu­man is Boltz­mann-ra­tio­nal in the two-ac­tion situ­a­tion (hence the sig­moid). I as­sume that it’s the same for the multi-ac­tion situ­a­tion, though this isn’t stated. How much does the ex­act way in which the hu­man is ir­ra­tional mat­ter for their re­sults?

• Thoughts on Abram Dem­ski’s Par­tial Agency:

When I read Par­tial Agency, I was struck with a de­sire to try for­mal­iz­ing this par­tial agency thing. Defin­ing My­opia seems like it might have a defi­ni­tion of my­opia; one day I might look at it. Any­way,

For­mal­iza­tion of Par­tial Agency: Try One

A my­opic agent is op­ti­miz­ing a re­ward func­tion where is the vec­tor of pa­ram­e­ters it’s think­ing about and is the vec­tor of pa­ram­e­ters it isn’t think­ing about. The gra­di­ent de­scent step picks the in the di­rec­tion that max­i­mizes (it is my­opic so it can’t con­sider the effects on ), and then moves the agent to the point .

This is dual to a stop-gra­di­ent agent, which picks the in the di­rec­tion that max­i­mizes but then moves the agent to the point (the gra­di­ent through is stopped).

For ex­am­ple,

• Nash equil­ibria - are the pa­ram­e­ters defin­ing the agent’s be­hav­ior. are the pa­ram­e­ters of the other agents if they go up against the agent parametrized by . is the re­ward given for an agent go­ing up against a set of agents .

• Image recog­ni­tion with a neu­ral net­work - is the pa­ram­e­ters defin­ing the net­work, are the image clas­sifi­ca­tions for ev­ery image in the dataset for the net­work with pa­ram­e­ters , and is the loss func­tion plus the loss of the net­work de­scribed by on clas­sify­ing the cur­rent train­ing ex­am­ple.

• Epi­sodic agent - are pa­ram­e­ters de­scribing the agents be­hav­ior. are the perfor­mances of the agent in fu­ture epi­sodes. is the sum of , plus the re­ward ob­tained in the cur­rent epi­sode.

Par­tial Agency due to Uncer­tainty?

Is it pos­si­ble to cast par­tial agency in terms of un­cer­tainty over re­ward func­tions? One rea­son I’d be my­opic is if I didn’t be­lieve that I could, in ex­pec­ta­tion, im­prove some part of the re­ward, per­haps be­cause it’s in­tractable to calcu­late (be­hav­ior of other agents) or some­thing I’m not pro­grammed to care about (re­ward in other epi­sodes).

Let be drawn from a prob­a­bil­ity dis­tri­bu­tion over re­ward func­tions. Then one could de­com­pose the true, un­cer­tain, re­ward into defined in such a way that for any ? Then this is would be my­opia where the agent ei­ther doesn’t know or doesn’t care about , or at least doesn’t know or care what its out­put does to . This seems suffi­cient, but not nec­es­sary.

Now I have two things that might de­scribe my­opia, so let’s use both of them at once! Since you only end up do­ing gra­di­ent de­scent on , it would make sense to say , , and hence that .

Since for small , this means that , so sub­sti­tut­ing in my ex­pres­sion for gives , so . Uncer­tainly is only over , so this is just the claim that the agent will be my­opic with re­spect to if . So it won’t want to in­clude in its gra­di­ent calcu­la­tion if it thinks the gra­di­ents with re­spect to are, on av­er­age, 0. Well, at least I didn’t de­rive some­thing ob­vi­ously false!

But Wait There’s More

When writ­ing the ex­am­ples for the gra­di­ent de­scenty for­mal­i­sa­tion, some­thing struck me: it seems there’s a struc­ture to a lot of them, where is the re­ward on the cur­rent epi­sode, and are re­wards ob­tained on fu­ture epi­sodes.

You could maybe even use this to have soft epi­sode bound­aries, like say the agent re­ceives a re­ward on each timestep so , and say­ing that so that for , which is ba­si­cally the crite­rion for my­opia up above.

Un­re­lated Note

On a com­pletely un­re­lated note, I read the Parable of Pre­dict-O-Matic in the past, but fool­ishly ne­glected to read Par­tial Agency be­fore­hand. The only thing that I took away from PoPOM the first time around was the bit about in­ner op­ti­misers, co­in­ci­den­tally the only con­cept in­tro­duced that I had been think­ing about be­fore­hand. I should have read the manga be­fore I watched the anime.

• So the defi­ni­tion of my­opia given in Defin­ing My­opia was quite similar to my ex­pan­sion in the But Wait There’s More sec­tion; you can roughly match them up by say­ing and , where is a real num­ber cor­re­spond­ing to the amount that the agent cares about re­wards ob­tained in epi­sode and is the re­ward ob­tained in epi­sode . Put­ting both of these into the sum gives , the undis­counted, non-my­opic re­ward that the agent even­tu­ally ob­tains.

In terms of the defi­ni­tion that I give in the un­cer­tainty fram­ing, this is , and .

So if you let be a vec­tor of the re­ward ob­tained on each step and be a vec­tor of how much the agent cares about each step then , and thus the change to the over­all re­ward is , which can be nega­tive if the two sums have differ­ent signs.

I was hop­ing that a point would re­veal it­self to me about now but I’ll have to get back to you on that one.

• Some other peo­ple who play to win

It’s a crowd I’d come into con­tact with as a man­ager of an on­line book­shop (and most of the rea­son I quit­ted). Usu­ally, I can pre­tend they don’t ex­ist, but… we all know how it goes… and now that they don’t make my blood boil ev­ery week­end, I can af­ford to speak about them.

“Some other peo­ple” will play to win—say, a face­book lot­tery with a book for a prize, and they will mean it. If they don’t win, they will say the lot­tery was rigged. Public righ­teous in­dig­na­tion on ev­ery player’s be­half is a weapon (and for the man­ager, a po­tent vac­cine against righ­teously in­dig­nant polemics of many other kinds). Pri­vate ap­peals to the man­ager’s pity; com­ment­ing the rules’ ex­ploitable/​ex­ploited loop­holes—af­ter the win­ner is an­nounced; re­peat­ing ac­tions which have already been an­swered el­se­where in the thread. I don’t in­clude ‘filing a com­plaint’ here, be­cause it’s frankly too straight­for­ward for most of them, most of the time; the book­shop would likely send them a book with an elo­quent bless­ing/​apol­ogy, just to get them to shut up and earn good PR points for “own­ing up to mis­takes”. But in prac­tice, it still mat­ters too much to be the ac­tual win­ner, and the brain of the tro­phy-gath­erer works like other brains don’t. At least not for a while.

I’m not un­usu­ally out-of-touch with cus­tomers; I was recom­mended for the job af­ter two years in an offline shop. And this was… en­tirely differ­ent. I’d never en­coun­tered peo­ple with whole pro­files ded­i­cated to re­post­ing on­line lot­ter­ies—liv­ing peo­ple I had to call on the phone. It is an­other world.

When I read about (sim­ple) “pure” game the­o­ret­i­cal prob­lems, in which the play­ers “care only about win­ning”, I can­not rec­on­cile the image of Wor­thy Ri­vals the au­thor has in mind with the ac­tual Really-Want-This-What­ever Whin­ers who seek out such con­tests. Get it, not the pas­sively al­low­ing them­selves to be drawn into a strate­gic game kind of play­ers, but the self-sort­ing to ex­ploit as many offers as pos­si­ble kind. They will be few, yes. No­body of them might force their way through ev­ery sin­gle time.

But they will define the mean­ing of the rules you think you write.

• I can­not rec­on­cile the image of Wor­thy Ri­vals the au­thor has in mind

What’s the book?

• Not any par­tic­u­lar book, but rather some fre­quent con­di­tions of game the­ory prob­lems I have seen here and el­se­where (my fb friend keeps post­ing such pieces). “The play­ers care only about win­ning” etc. Well, some peo­ple ac­tu­ally do.

• There are two kinds of plea­surable feel­ings. The first one is a self-re­in­forc­ing loop, where the in-the-mo­ment plea­sure leads to crav­ing for more plea­sure, such as mind­lessly scrol­ling through so­cial me­dia, or eat­ing highly-pro­cessed, highly-palat­able food. The sec­ond is plea­sure gained through ei­ther thought­fully con­sum­ing good con­tent, like listen­ing to good mu­sic or read­ing good books, or the fulfill­ment of a task that’s mean­ingful, such as get­ting good grades or get­ting a pro­mo­tion for sus­tained con­scien­tious effort.

The first is plea­sure for its own sake, with­out any “real world re­wards” that come with it, ie, dis­trac­tions. The sec­ond isn’t as “ad­dic­tive” as the first, nor does it cause the same spikes in plea­sure, but it comes with real world tan­gible re­wards.

There is no way to com­pletely elimi­nate the hu­man need for the first plea­sure. But the need can be re­duced. The ra­tio of sec­ond:first plea­sure, is the de­gree to which a per­son is able to achieve his goals, the de­gree to which a per­son is suc­cess­ful.

• There’s a large piece miss­ing from your model—why seek plea­sure at all? Wouldn’t it be bet­ter to mea­sure suc­cess di­rectly based on meet­ing long-term goals (clippy in­ter­jects from the bal­cony: like num­ber of pa­per­clips cre­ated!).

• As an add-on, I found this LW ar­ti­cle to­day that cap­tures the essence of “first plea­sure”

clippy in­ter­jects from the bal­cony: like num­ber of pa­per­clips cre­ated!

I’m not quite sure I got this part, could you please elab­o­rate on it?

why seek plea­sure at all? Wouldn’t it be bet­ter to mea­sure suc­cess di­rectly based on meet­ing long-term goals

Here, I would ar­gue that the feel­ing of the sec­ond plea­sure is es­sen­tial to meet­ing long-term goals. Feel­ing good about ac­com­plish­ing sub-tasks will keep some­one work­ing to­wards an im­por­tant goal, es­pe­cially if it re­quires a long pe­riod of sus­tained effort.

Thus, meet­ing short-term goal → suc­cess → sec­ond plea­sure → work­ing on next short-term goal; with enough iter­a­tions, that will lead to the meet­ing of long-term goals and thus suc­cess.

• https://​​wiki.less­wrong.com/​​wiki/​​Paper­clip_max­i­mizer is the canon­i­cal ex­am­ple of over-sim­plified goal op­ti­miza­tion. I bring it up mostly as a re­minder that get­ting your mo­ti­va­tional model wrong can lead to un­de­sir­able ac­tions and re­sults.

Which leads to my main point. You’re recom­mend­ing one type of plea­sure over an­other, based on it be­ing more al­igned with your non-plea­sure-mea­sured goals. I’m won­der­ing why you are ar­gu­ing for this, as op­posed to just pur­su­ing the goals di­rectly, with­out con­sid­er­a­tion of plea­sure.

• Ah, now I’ve got what you mean. Thanks for refer­ring me to that thought ex­per­i­ment, I don’t have much prior knowl­edge on the field of AI so that was definitely a new in­sight for me.

I see now that my origi­nal short­form did not ex­plic­itly state that my ter­mi­nal value was in­deed the fulfill­ment of im­por­tant goals. I was re­flect­ing more on the dis­tinc­tion be­tween plea­surable feel­ings that led to dis­trac­tion & bad habits, vs ones that led to the ac­tual fulfill­ment of goals. It was a per­sonal re­minder to ex­pe­rience the lat­ter in place of the former, as much as I can.

Now, I hold the view that plea­sure can be a use­ful tool in the pur­suit of my goals. It is a means to that end. An im­por­tant caveat that your re­sponse re­minded me of, though, is that some­times pur­su­ing goals might not be im­me­di­ately plea­surable, so it might be wise not to naïvely ex­pect plea­sure from ev­ery part of that pro­cess.

• Sen­tences spo­ken aloud are a la­tent space em­bed­ding of our thoughts; when try­ing to move a thought from our mind to an­other’s, our thoughts are en­coded with the aim of min­i­miz­ing the other per­son’s de­coder er­ror.

• There’s a prob­lem at par­ties where there’ll be a good, high-con­text con­ver­sa­tion hap­pen­ing, and then one-too-many-peo­ple join, and then the con­ver­sa­tion sud­denly dies.

Some­times this is fine, but other times it’s quite sad.

Things I think might help:

• If you’re an ex­ist­ing con­ver­sa­tion par­ti­ci­pant:

• Ac­tively try to keep the con­ver­sa­tion small. The up­per limit is 5, 3-4 is bet­ter. If some­one looks like they want to join, smile warmly and say “hey, sorry we’re kinda in a high con­text con­ver­sa­tion right now. Listen­ing is fine but prob­a­bly don’t join.”

• If you do want to let a new­comer join in, don’t try to get them up to speed (I don’t know if I’ve ever seen that ac­tu­ally work). In­stead, say “this is high con­text so we’re not gonna re­peat the ear­lier bits, maybe wait to join in un­til you’ve listened enough to un­der­stand the over­all con­text”, and then quickly get back to the con­ver­sa­tion be­fore you lose the Flow.

• If you want to join a con­ver­sa­tion:

• If there are already 5 peo­ple, sorry, it’s prob­a­bly too late. Listen if you find it in­ter­est­ing, but if you ac­tively join you’ll prob­a­bly just kill the con­ver­sa­tion.

• Give them the op­por­tu­nity to grace­fully keep the con­ver­sa­tion small if they choose. (say some­thing like “hey can I join? It sounds like maybe a high con­text con­ver­sa­tion, no wor­ries if you wanna keep it small.”)

• Listen for longer be­fore join­ing. Don’t just wait till you un­der­stand the cur­rent topic – try to un­der­stand the over­all vibe, and what pre­vi­ous top­ics might be in­form­ing the cur­rent one. Try to get a sense of what each cur­rent par­ti­ci­pant is get­ting out the con­ver­sa­tion. When you do join, do so in a small way that gives them af­for­dance to shift back to an ear­lier topic if your new con­tri­bu­tion turned out to be not-ac­tu­ally-on-topic.

• I hosted an on­line-party us­ing zoom break­out rooms a few weeks ago and ran into similar prob­lems.

Half-way through the party I no­ticed peo­ple were clus­ter­ing in sub­op­ti­mal size con­ver­sa­tions and bring­ing high-con­text con­ver­sa­tions to a stop, so I ac­tu­ally brought ev­ery­body backed to the lobby then ran­domly as­signed them to groups of 2 or 3 - and when I checked 10 min­utes later ev­ery­one was in the same two rooms again with groups of 8 − 10 peo­ple.

AFAICT this was sta­tus/​feel­ings driven—there were a few peo­ple at the party who were ei­ther ex­ist­ing high-sta­tus to the par­ti­ci­pants, or who were very charis­matic, and ev­ery­one wanted to be in the same con­ver­sa­tion as them.

I think norm-set­ting around this is very hard, be­cause it’s nat­u­ral to want to be around high-sta­tus and charis­matic peo­ple, and it’s also nat­u­ral to want to par­ti­ci­pate in a con­ver­sa­tion you’re listen­ing to.

I’m go­ing to try to add your sug­ges­tions to the top of the shared google doc next time I host one of these and see how it goes.

• Agreed with the sta­tus/​feel­ings cause. And I’m not 100% sure the solu­tion is “pre­vent peo­ple from do­ing the thing they in­stinc­tively want to do” (es­pe­cially “all the time.”)

My cur­rent guess is “let peo­ple crowd around the charis­matic/​and/​or/​in­ter­est­ing peo­ple, but treat it more like a panel dis­cus­sion or fireside chat, like you might have at a con­fer­ence, where mostly 2-3 peo­ple are talk­ing and ev­ery­one else is more for­mally ‘au­di­ence.’”

But do­ing that all the time would also be kinda bad in differ­ent ways.

In this case… you might ac­tu­ally be able to fix this with tech­nol­ogy? Can you liter­ally put room-caps on the rooms, so if some­one wants to be the 4th or 6th per­son in a room they… just… can’t?

• +lots. Some tech­niques:

• phys­i­cally sep­a­rate the group. Go into an­other room or at least cor­ner. Sig­nal that you’re not seek­ing ad­di­tional par­ti­ci­pants.

• When you no­tice this, make it ex­plicit—“I’m re­ally en­joy­ing the depth of this con­ver­sa­tion, should we move into the lounge for a brandy and a lit­tle more quiet?”

• Ad­mit (to your­self) that oth­ers may feel ex­cluded, be­cause they are. At many gath­er­ings, such dis­cus­sions/​situ­a­tions are time-bound and re­ally can’t last more than 10-45 min­utes. The only solu­tion is to have more fre­quent, smaller gath­er­ings.

• Get good at in­volved listen­ing—it’s differ­ent than 1:1 ac­tive listen­ing, but has similar goals: don’t in­ject any ideas, but do give sig­nals that you’re fol­low­ing and sup­port­ing. This is at least 80% as en­joy­able as ac­tive par­ti­ci­pa­tion, and doesn’t break the flow when you join a clique in progress.

I won­der what analogs there are to on­line con­ver­sa­tions. I sus­pect there’s a lot of similar­ity for syn­chronous chats—too many peo­ple make it im­pos­si­ble to fol­low. For threaded, async dis­cus­sions, the limits are prob­a­bly much larger.

• I won­der what analogs there are to on­line con­ver­sa­tions. I sus­pect there’s a lot of similar­ity for syn­chronous chats—too many peo­ple make it im­pos­si­ble to fol­low. For threaded, async dis­cus­sions, the limits are prob­a­bly much larger.

FYI, the ac­tual mo­ti­vat­ing ex­am­ple here was at a party in gather.town, (formerly on­line.town, formerly town.siem­pre), which has much more typ­i­cal “party” dy­nam­ics. (i.e peo­ple can wan­der around an on­line world and video chat with peo­ple nearby).

In this case there were ac­tu­ally some ad­di­tional com­plex­ities – I had joined a con­ver­sa­tion rel­a­tively late, I did lurk for quite awhile, and wait for the cur­rent set of top­ics to die down com­pletely be­fore in­tro­duc­ing a new one. And then the con­ver­sa­tion took a turn that I was re­ally ex­cited by, and at least 1-2 other peo­ple were in­ter­ested in, but it wasn’t ob­vi­ous to me that it was in­ter­est­ing to ev­ery­one else (I think ~5 peo­ple in­volved to­tal?)

And then a new per­son came in, and asked what we were talk­ing about and some­one filled them in… …and then im­me­di­ately the con­ver­sa­tion ended. And in this case I don’t know if the is­sue was more like “the new­comer kil­led the con­ver­sa­tion” or “the convo ac­tu­ally had roughly reached it’s nat­u­ral end, and/​or other peo­ple weren’t that in­ter­ested in the first place.”

But, from my own per­spec­tive, the con­ver­sa­tion had just finished cov­er­ing all the ob­vi­ous back­ground con­cepts that would be re­quired for the “real” con­ver­sa­tion to be­gin, and I was hop­ing to ac­tu­ally Make Real Progress on a com­plex con­cept.

So, I dunno if this counted as “an in­ter­est­ing con­ver­sa­tion” yet, and un­for­tu­nately the act of ask­ing the ques­tion “hey, do we want to con­tinue div­ing deep into this, or wrap up and tran­si­tion into some other convo?” also kinda kills the con­ver­sa­tion. Con­ver­sa­tions are so god damn frag­ile.

What I re­ally wished was that ev­ery­one already had com­mon knowl­edge of the meta-con­cept, wherein:

• Party con­ver­sa­tions are par­tic­u­larly fragile

• Bring­ing a new­comer up to speed is usu­ally costly if the con­ver­sa­tion is do­ing any­thing deep

• We might or might not want to con­tinue delv­ing into the cur­rent convo (but we don’t cur­rently have com­mon knowl­edge of this in ei­ther di­rec­tion)

And if ev­ery­one (new­comer in­cluded) had those con­cepts, and new ev­ery­one had those con­cepts, then I feel like I could have asked more grace­fully “hey, I’m kinda in­ter­ested in con­tin­u­ing to hash out some ideas here. Are peo­ple up for tak­ing this high con­text”, and had peo­ple give their hon­est an­swer.

• [EDIT, was in­tended as a re­sponse to Rae­mon, not Dagon.]

Maybe it’s the way you phrase the re­sponses. But as de­scribed, I get the im­pres­sion that this norm would mainly work for rel­a­tively ex­tro­verted per­sons with low re­jec­tion sen­si­tivity.

I’d be much less likely to ever try to join a dis­cus­sion (and would tend to not at­tend events with such a norm). But maybe there’s a way to avoid this, both from “my side” and “yours”.

• Hmm, seems like im­por­tant feed­back. I had speci­fi­cally been try­ing to phrase the re­sponses in a way that ad­dressed this spe­cific prob­lem. Sounds like it didn’t work.

There is some in­trin­sic re­jec­tion go­ing on here, which prob­a­bly no amount of kind word­ing can alle­vi­ate for a re­jec­tion-sen­si­tive per­son.

For my “sorry, we’re keep­ing the convo small” bit, I sug­gested:

smile warmly and say “hey, sorry we’re kinda in a high con­text con­ver­sa­tion right now. Listen­ing is fine but prob­a­bly don’t join.”

The Smile Warmly part was meant to be a pretty ac­tive in­gre­di­ent, helping to re­as­sure them it isn’t per­sonal.

Another thing that seems pretty im­por­tant, is that this ap­plies to all new­com­ers, even your friends and High Sta­tus Peo­ple. (i.e. hope­fully if Anx­ious Alex gets turned away, but later sees High Sta­tus Bob also get turned away, they get re­as­sured a bit that this wasn’t about them)

• Virtue ethics seems like model-free con­se­quen­tial­ism to me.

• I’ve was think­ing along similar lines!

From my notes from 2019-11-24: “Deon­tol­ogy is like the learned policy of bounded ra­tio­nal­ity of con­se­quen­tial­ism”

• I’m look­ing for an old post where Eliezer makes the ba­sic point that we should be able to do bet­ter than in­tel­lec­tual figures of the past, be­cause we have the “un­fair” ad­van­tage of know­ing all the sci­en­tific re­sults that have been dis­cov­ered since then.

I think he cites in par­tic­u­lar the heuris­tics and bi­ases liter­a­ture as some­thing that thinkers wouldn’t have known about 100 years ago.

I don’t re­mem­ber if this was the main point of the post it was in, or just an aside, but I’m pretty con­fi­dent he made a point like this at least once, and in par­tic­u­lar com­mented on how the ad­van­tage we have is “un­fair” or some­thing like that, so that we shouldn’t feel at all sheep­ish about declar­ing old thinkers wrong.

Any­body know what post I’m think­ing of?

• Max Gluck­man once said: “A sci­ence is any dis­ci­pline in which the fool of this gen­er­a­tion can go be­yond the point reached by the ge­nius of the last gen­er­a­tion.” Science moves for­ward by slay­ing its heroes, as New­ton fell to Ein­stein. Every young physi­cist dreams of be­ing the new cham­pion that fu­ture physi­cists will dream of de­thron­ing.
Ayn Rand’s philo­soph­i­cal idol was Aris­to­tle. Now maybe Aris­to­tle was a hot young math tal­ent 2350 years ago, but math has made no­tice­able progress since his day. Bayesian prob­a­bil­ity the­ory is the quan­ti­ta­tive logic of which Aris­to­tle’s qual­i­ta­tive logic is a spe­cial case; but there’s no sign that Ayn Rand knew about Bayesian prob­a­bil­ity the­ory when she wrote her mag­num opus, At­las Shrugged. Rand wrote about “ra­tio­nal­ity”, yet failed to fa­mil­iarize her­self with the mod­ern re­search in heuris­tics and bi­ases. How can any­one claim to be a mas­ter ra­tio­nal­ist, yet know noth­ing of such el­e­men­tary sub­jects?

(Not sure if the ma­te­rial is ac­cu­rate, but I think it’s the post you’re look­ing for. There could have been more than one on that though.)

Refer­ences 1.

• Thanks!

• Weird thing I wish ex­isted: I wish there were more videos of what I think of as ‘math/​pro­gram­ming speedruns’. For those fa­mil­iar with speedrun­ning video games, this would be similar ex­cept the idea would be to do the same thing for a math proof or pro­gram­ming prob­lem. While it might seem like this would be quite bor­ing since the solu­tion to the prob­lem/​proof is known, I still think there’s an el­e­ment of skill to and would en­joy watch­ing some­one do ev­ery­thing they can to get to a solu­tion, proof, etc. as quickly as pos­si­ble (in an ed­i­tor, on pa­per, LaTex, etc.).

This is kind of similar to stream­ing ACM/​math olympiad com­pe­ti­tion solv­ing ex­cept I’m equally more in peo­ple do­ing this for known prob­lems/​proofs than I am for tricky but ob­scure prob­lems. E.g., speed-run­ning the SVD deriva­tion.

While I’m post­ing this in the hope that oth­ers are also re­ally in­ter­ested, my sense is that this would be in­cred­ibly niche even amongst peo­ple who like math so I’m not sur­prised it doesn’t ex­ist...

• Some­what re­lated:

Last Thurs­day on the Dis­cord we had peo­ple any% speedrun­ning and rac­ing the Lean tu­to­rial pro­ject . This fits very well into my gen­eral wor­ld­view: I think that do­ing math­e­mat­ics in Lean is like solv­ing lev­els in a com­puter puz­zle game, the ex­cit­ing thing be­ing that math­e­mat­ics is so rich that there are many many kinds of puz­zles which you can solve.

https://​​xe­napro­ject.word­press.com/​​2020/​​05/​​23/​​the-com­plex-num­ber-game/​​

• This is awe­some! I’ve been think­ing I should try out the nat­u­ral num­ber game for a while be­cause I feel like for­mal the­o­rem prov­ing will scratch my cod­ing /​ video game itch in a way nor­mal math doesn’t.

• I have a friend who might be into pro­gram­ming speedrun­ning https://​​merveilles.town/​​@can­cel/​​104005117320841920

• Seems like the post you linked is a joke. Were you se­ri­ous about the friend?

• Se­ri­ous in that I mean he might, I’d say, 0.1 that he’d be in­ter­ested, but if that’s not neg­ligible, I think if he took it up he’d be very good at it. I’ll ask him.

• Cool!

• I had a similar idea which was also based on an anal­ogy with video games (where the anal­ogy came from let’s play videos rather than speedruns), and called it a live math video.

• Cool, I hadn’t seen your page pre­vi­ously but our ideas do in fact seem very similar. I think you were right to not fo­cus on the speed el­e­ment and in­stead analo­gize to ‘let’s play’ videos.

• Re­lated: here, DJB lays out the pri­mary re­sults of a sin­gle-vari­able calcu­lus course in 11 LaTex-ed pages.

• I’m not su­per fa­mil­iar with the com­pet­i­tive math cir­cuit, but my un­der­stand­ing is that this is part of it? Peo­ple are given a hard prob­lem and ei­ther in­di­vi­d­u­ally or as a team solve it as quickly as pos­si­ble.

• Yep, I touched on this above. Per­son­ally, I’m less in­ter­ested in this type of prob­lem solv­ing than I am in see­ing some­one build to a well-known but po­ten­tially eas­ier to prove the­o­rem, but I sus­pect peo­ple solv­ing IMO prob­lems would ap­peal to a wider au­di­ence.

• Do you know of any videos on this? Ideally while the per­son is nar­rat­ing their thoughts out loud.

• 3Blue1Brown has a video where he sort of does this for a hard Put­nam prob­lem. I say “sort of” be­cause he’s not solv­ing the prob­lem in real time so much as ret­ro­spec­tively de­scribing how one might solve it.

• Yeah, that is one of my fa­vorite videos by 3Blue1Brown and more like it would be pretty good.

• The prob­lem with this is that it is very difficult to figure out what counts as a le­gi­t­i­mate proof. What level of rigor is re­quired, ex­actly? Are they al­lowed to mem­o­rize a proof be­fore­hand? If not, how much are they al­lowed to know?

• Solu­tions might be bet­ter to go with than proofs—if the an­swer is wrong, that’s more straight­for­ward to show that whether or not a proof is wrong.

• Yeah what would be ideal is if the­o­rem provers were more us­able and then this wouldn’t be an is­sue (al­though of course there’s still the is­sue of library code vs. from scratch code but this seems eas­ier to deal with).

Me­moriz­ing a proof seems fine (in the same way that I as­sume you end up ba­si­cally mem­o­riz­ing the game map if you do a speedrun).

• When I was in high school, I once had a con­ver­sa­tion with a class­mate that went some­thing like this (ex­cept that it was longer and I was less elo­quent):

Him: “Ger­man is a Scan­d­i­na­vian lan­guage.”

Me: “No, it’s not. Ger­man and the Scan­d­i­na­vian lan­guages both fall un­der the um­brella of Ger­manic lan­guages, but ‘Scan­d­i­na­vian lan­guages’ refers to a nar­rower cat­e­gory that doesn’t in­clude Ger­man.”

Me: “No??? That’s not what an opinion is???”

Him: “Look, it’s your opinion that Ger­man isn’t a Scan­d­i­na­vian lan­guage, and it’s my opinion that it is. We can agree to dis­agree.”

Me: ??????????????????!!!!!!!!!????!?!?!?!?! *punches self in face*

----

When I was tak­ing a re­quired in­tro biol­ogy course in col­lege, I had already read a bunch of LW and SSC, no­tably in­clud­ing That Cho­co­late Study. So when the pro­fes­sor put Bo­han­non’s re­sults and method­ol­ogy up on the pro­jec­tor, I was ready as heck to talk about all of the atroc­i­ties therein. The pro­fes­sor asked us to pair up with the per­son next to us to dis­cuss whether we be­lieved Bo­han­non’s re­sults, and I de­cided to give the fresh­man next to me the chance to speak first be­fore I ab­solutely de­mol­ished ev­ery­thing. The girl turned to me with wide eyes and a con­fi­dent, creaky-voice drawl, and said, ver­ba­tim: “I think it’s true, be­cause choco­late is known to be a su­perfood.”

I was floored. How could this be hap­pen­ing in real life? I was at an elite col­lege with a sub-10% ac­cep­tance rate, and this per­son next to me had just said “known to be” and “su­perfood” like they ex­plained any­thing – like they meant any­thing. I will never for­get those words. Look­ing back, that may have been the day I de­cided to move to the Bay af­ter grad­u­at­ing. No re­grets.

• I’ve been think­ing about peo­ple’s mind­set as it re­lates to spend­ing their free time. Speci­fi­cally, when you go to do some­thing ‘pro­duc­tive’ like learn about a new topic, work through ex­er­cises in a text­book, go through an on­line course, etc...do you feel that you have to in­ten­tion­ally de­cide not to play video games, watch Net­flix, etc and forego short-term hap­piness? Or do you feel that this de­ci­sion is straight­for­ward be­cause that’s what you would pre­fer to be do­ing and you don’t feel like you sac­ri­fice any­thing?