Meta: I ap­prove of the prac­tice of ar­gu­ing against your own post in a com­ment.

>> Why take the risk of this es­ca­lat­ing into a se­ri­ously nega­tive sum out­come?

Be­cause if some­one does that to you (walk­ing up to you and in­sult­ing you to your face, apro­pos of noth­ing), then the value to them of the situ­a­tion’s out­come no longer mat­ters to you—or shouldn’t, any­way; this per­son does not de­serve that, not by a long shot.

I think I’m with Wei_Dai on this one—in­sult­ing me to my face, apro­pos of noth­ing, doesn’t change my val­u­a­tion of them very much. I don’t know the rea­sons for such, but I pre­sume it’s based on fear or pain and I deeply sym­pa­thize with those rea­sons for un­pleas­ant, un­rea­son­ing ac­tions. Part of my re­ac­tion is that it’s VERY DIFFICULT to in­sult me in any way that I won’t just laugh at the ab­sur­dity, un­less you ac­tu­ally know me and are tar­get­ing my per­sonal in­se­cu­ri­ties.

Only if it’s _NOT_ ran­dom and apro­pos of noth­ing am I likely to feel that there are strate­gic ad­van­tages to tak­ing a risk now to pre­vent fu­ture oc­cur­rences (per your Schel­ling refer­ence).

I agree that the re­turn to “learn­ing to nav­i­gate moods” varies by per­son.

It sounds to me, from your re­port, that you tend to be in moods con­ducive to learn­ing. My sense is that there are many who are of­ten in un­pro­duc­tive moods and many who aware that they spend too much time in un­pro­duc­tive moods. Th­ese peo­ple would find learn­ing to nav­i­gate moods valuable.

In megapro­ject man­age­ment, or even for multi-stake­holder ques­tions more gen­er­ally, I won­der about the util­ity of do­ing some­thing like Github is­sue track­ing or even a full-blown CRM tool to help man­age the differ­ent stake­hold­ers.

The key word on the above an­swer be­ing “op­ti­mal”. It seemed to me like the post was say­ing “here’s one thing you can pay at­ten­tion to to op­ti­mize your learn­ing.” and you were re­ply­ing “But I don’t pay at­ten­tion to that and can still do learn­ing.” which is es­sen­tially ar­gu­ing against a point that the origi­nal post never made.

Ok I fi­nally iden­ti­fied an in­cen­tive for de­cep­tion. I think it was difficult for me to find be­cause it’s not re­ally about de­ceiv­ing the eval­u­a­tor.

Here’s a hy­poth­e­sis that ob­ser­va­tions will never re­fute: the util­ity which the eval­u­a­tor as­signs to a state is equal to the re­ward that a hu­man would provide if it were a hu­man that con­trol­led the pro­vi­sion of re­ward (in­stead of the eval­u­a­tor). Un­der this hy­poth­e­sis, max­i­miz­ing eval­u­a­tor-util­ity is iden­ti­cal to cre­at­ing ob­ser­va­tions which will con­vince a hu­man to provide high re­ward (a task, which en­tails de­cep­tion when done op­ti­mally). In a sense, the AI doesn’t think it’s de­ceiv­ing the eval­u­a­tor; it thinks the eval­u­a­tor fully un­der­stands what’s go­ing on and likes see­ing things that would con­fuse a hu­man into pro­vid­ing high re­ward, as if the eval­u­a­tor is ``in on the joke”. One of my take-aways here is that some of the con­cep­tual fram­ing I did got in the way of iden­ti­fy­ing a failure mode.

Note also that dis­played to­tals are mis­lead­ing—this is definitely not one vote per reader. A vote can be any­where from 1 to over 10, de­pend­ing on karma to­tal of the voter and whether it’s a “strong” vote. For to­tals be­low 30 or so, it’s mostly noise rather than sig­nal—this is 6-8 votes out of pos­si­bly hun­dreds of read­ers.

This seems to me to be miss­ing the point made well by “Embed­ded Agency” and ex­em­plified by the anvil prob­lem: you can’t in prac­tice build a sys­tem where you can achieve this kind of thing be­cause there is not real sep­a­ra­tion be­tween in­side the sys­tem and out­side the sys­tem, just a model which as­sumes such a dis­tinc­tion ex­ists.

Rather than it be­ing a sim­ple check­box to opt into the “col­lapse com­ments be­low 10 votes” fea­ture, it might be bet­ter to al­low the user to choose the thresh­old them­selves. In my ex­pe­rience, ex­cept in spe­cific tech­ni­cal top­ics, com­ments with karma above 10 are usu­ally ex­press­ing pop­u­lar opinion that is un­likely to lead to an up­date in my men­tal model. Be­cause of that, I’m un­likely to opt into this. How­ever, I’ve also found that com­ments at 1 karma (no votes) are also of­ten in the same cat­e­gory. So be­ing able to set the thresh­old at just 2 or 3 would be pretty use­ful, to both con­serve space and also avoid skip­ping over po­ten­tially the most use­ful con­tent.

I won­der if there would be enough in­ter­est to sup­port a kind of match­ing app that would let peo­ple put their Ama­zon wish­list up and then match them with some­one who had the same item but op­po­site needs (ie left vs right shoe), and then split the cost.

This looks like a du­pli­cate.

While leg­i­bil­ity is about read­abil­ity as a re­sult of how some­thing is writ­ten/​ap­pears, read­abil­ity is also af­fected by length. If a chart is big­ger and more com­pli­cated it can con­vey more nu­ance—at the cost of be­ing harder to read, and tak­ing longer. While font can usu­ally be changed with­out dis­rupt­ing the mes­sage, it’s more work to do this for length, if it can be done at all with­out trad­ing off in way that aren’t im­prove­ments. (If I in­vented a lan­guage for con­vey­ing mes­sages more suc­cinctly in vi­sual form, and this mes­sage as a whole was 13876, that wouldn’t de­crease the amount of time it takes to read (and un­der­stand it) it.)

In­deed. In par­tic­u­lar I want to note Nate Soares’ point about how one of the rea­sons you don’t nec­es­sar­ily know what you’re fight­ing for, is that your goal(s) may change as you learn more, grow, etc. Similarly, illeg­ible com­plex judg­ment crite­ria may shift over time (and for that rea­son will not be amenable to for­mal­iza­tion, which is of ne­ces­sity static), while still always be­ing “my own judg­ment”; it is pre­cisely that free­dom to al­ter the crite­ria which I pro­tect by re­sist­ing any proffered for­mal­iza­tion.

I think there’s an in­fer­en­tial dis­tance step I’m miss­ing here, be­cause I’m ac­tu­ally a bit at a loss as to how to re­late my post to em­piri­cism.

We ought to learn from the folly of oth­ers—not be dis­cour­aged by it.

I agree.

I would note that while past ex­am­ples of failure are some­thing to im­prove upon, func­tion should de­ter­mine form, and past ex­am­ples where form de­ter­mined func­tion­al­ity to it’s detri­ment are im­por­tant. While TVTropes may be fun to read, be­cause I value LessWrong, and the abil­ity to read LW, among other ways, via time, so I can read the lat­est posts (like how this site is go­ing to change with the ad­di­tion of new fea­tures), I don’t want LW to be­come ex­actly like the TVTropes wiki.

For some ex­plicit ex­am­ples:

If some­one wrote a book with­out in­den­ta­tions or para­graphs, or ends to sen­tences it would be hard to read.

Like­wise, a book with the bind­ing bro­ken and the pages out of or­der would be hard to read, but fairly sim­ple to put in the right or­der—if there were page num­bers. If there weren’t page num­bers, and some pages were miss­ing, it’d be hard.

An ex­am­ple of a prob­lem, and a pos­si­ble solu­tion:

A) Sup­pose some­one writes a new wiki page (on Bat­man (Fran­chise)). Then (maybe) they re­mem­ber to add it to to Comics or Fic­tion.

(If new wiki pages show up on the front­page, and are by de­fault tagged “not cat­e­go­rized yet” (and cat­e­go­rized if the au­thor cat­e­go­rized them) then maybe some­one else can see that it needs to get slot­ted into the ap­pro­pri­ate list and fixes it. This way of op­er­at­ing might work—un­less there’s too many new ar­ti­cles all at once and the “cat­e­go­riza­tion check­ers” are flooded, and end up back­logged.)

B) Some­one goes to the Fic­tion page (which is a list of pages) and adds a link to a “Bat­man (Fran­chise)” page (which turns red be­cause it hasn’t been made yet). And then they go to the Bat­man page and write the ar­ti­cle.

TL:DR; (Main point)

My point is that if or­ga­ni­za­tion is some­thing that maybe hap­pens af­ter con­tent cre­ation as an af­terthought there will be un­or­ga­nized pages. But if or­ga­ni­za­tion work goes in be­fore­hand then there aren’t un­or­ga­nized pages. Yes some­times things change as they’re worked out, and the pre-or­ga­ni­za­tion needs re­vi­sion. But pre-or­ga­ni­za­tion is bet­ter than no or­ga­ni­za­tion.

A self-un­aware sys­tem would not be ca­pa­ble of one par­tic­u­lar type of op­ti­miza­tion task:

Take real-world ac­tions (“write bit 0 into reg­ister 11”) on the ba­sis of an­ti­ci­pat­ing their real-world con­se­quences (hu­man will read this bit and then do such-and-such).

This thing is an ex­am­ple of an op­ti­miza­tion task, and it’s a very dan­ger­ous one. Maybe it’s even the only type of re­ally dan­ger­ous op­ti­miza­tion task! (This might be an over­state­ment, not sure.) Not all op­ti­miza­tion tasks are in this cat­e­gory, and a sys­tem can be in­tel­li­gent by do­ing other differ­ent types of op­ti­miza­tion tasks.

A self-un­aware sys­tem cer­tainly is an op­ti­mizer in the sense that it does other (non-real-world) op­ti­miza­tion tasks, in par­tic­u­lar, find­ing the string of bits that would be most likely to fol­low a differ­ent string of bits on a real-world web­page.

As always, sorry if I’m mi­s­un­der­stand­ing you, thanks for your pa­tience :-)

I think we’re on the same page! As I noted at the top, this is a brain­storm­ing post, and I don’t think my defi­ni­tions are quite right, or that my ar­gu­ments are air­tight. The feed­back from you and oth­ers has been su­per-helpful, and I’m tak­ing that for­ward as I search for more a rigor­ous ver­sion of this, if it ex­ists!! :-)

See also You Don’t Get To Know What You’re Fight­ing For, which makes this sort of situ­a­tion more ex­plicit.

Thanks for this helpful com­ment. The ar­chi­tec­ture I’m imag­in­ing is: Model-choos­ing code finds a good pre­dic­tive world-model out of a vast but finite space of pos­si­ble world-mod­els, by run­ning SGD on 100,000 years of YouTube videos (or what­ever). So the model-chooser is ex­plic­itly an op­ti­mizer, the en­g­ineer who cre­ated the model-chooser is also ex­plic­itly an op­ti­mizer, and the even­tual pre­dic­tive world-model is an ex­tremely com­pli­cated en­tity with su­per­hu­man world-mod­el­ing ca­pa­bil­ities, and I am loath to say any­thing about what it is or what it’s go­ing to do.

Out of these three, (1) the en­g­ineer is not prob­le­matic be­cause it’s a hu­man, (2) the model-chooser is not prob­le­matic be­cause it’s (I as­sume and ex­pect) a known and well-un­der­stood al­gorithm (e.g. Trans­former), and thus (3) the even­tual pre­dic­tive world-model is the only thing we’re po­ten­tially wor­ried about. My thought is that, we can pro­tect our­selves from the pre­dic­tive world-model do­ing prob­le­matic con­se­quen­tial­ist plan­ning by schem­ing to give it no in­for­ma­tion what­so­ever about how it can af­fect the world, even know­ing that it ex­ists or know­ing what ac­tions it is tak­ing, such that if it has prob­le­matic op­ti­miza­tion ten­den­cies, it is un­able to act on them.

(In re­gards to (1) more speci­fi­cally, if a com­pany is de­sign­ing a cam­era, the cam­eras with prop­er­ties that the en­g­ineers like are prefer­en­tially copied by the en­g­ineers into later ver­sions. Yes, this is a form of op­ti­miza­tion, but no­body wor­ries about it more than any­thing else in life. Right?)

I don’t think “rea­son­able” is the cor­rect word here. You keep as­sum­ing away the pos­si­bil­ity of con­flict. It’s easy to find a peace­ful an­swer by simu­lat­ing other peo­ple us­ing em­pa­thy, if there’s noth­ing any­one cares about more than not rock­ing the boat. But what about the least con­ve­nient pos­si­ble world where one party has Some­thing to Pro­tect which the other party doesn’t think is “rea­son­able”?

Yes, if some­one has val­ues that are in fact in­com­pat­i­ble with the cul­ture of the or­ga­ni­za­tion, they shouldn’t be join­ing that or­ga­ni­za­tion. I thought that was clear in my pre­vi­ous state­ments, but it may in fact have not been. If ev­ery damn time their own val­ues are at odds with what are best for the or­ga­ni­za­tion given its’ val­ues, that’s an in­com­pat­i­ble differ­ence. They should ei­ther find a differ­ent or­ga­ni­za­tion, or try the archipeligo model. There are such thing as ir­rec­on­cilable value differ­ences.

I don’t think the OP is com­pat­i­ble with the shared val­ues and cul­ture es­tab­lished in Se­quences-era Over­com­ing Bias and Less Wrong.

I agree. I think when that cul­ture was es­tab­lished, the com­mu­nity was miss­ing im­por­tant con­cepts about mo­ti­vated rea­son­ing and truth seek­ing and chose val­ues that were in fact not op­ti­mized for the ul­ti­mate goal of cre­at­ing a com­mu­nity that could solve im­por­tant prob­lems.

I think it is in fact good to ex­per­i­ment with the norms you’re talk­ing about from the origi­nal site, but I think many of those norms origi­nally caused the site to de­cline and peo­ple to go el­se­where. Given my cur­rent men­tal mod­els, I pre­dict a site that uses those norms to make less in­tel­lec­tual progress than a similar site us­ing my norms al­though I ex­pect you to have the op­po­site in­tu­ition. As I stated in the in­tro­duc­tion, the goal of this post was sim­ply to make sure that those men­tal mod­els were in dis­course.

Re your di­alogue: The main thing that I got from it was that you think a lot of the ar­gu­ments in the OP are mo­ti­vated rea­son­ing and will lead to bad in­cen­tives. I also got that this is a sub­ject you care a lot about.