Back­wards, thanks!

There’s an in­ter­est­ing re­la­tion­ship with math­e­ma­tiz­ing of de­ci­sion prob­lems here, which I think is re­flec­tive of nor­mal philos­o­phy prac­tice.

For ex­am­ple, in the Smok­ing Le­sion prob­lem, and in similar cases where you con­sider an agent to have “urges” or “dis­po­si­tions” et c., it’s im­por­tant to note that these are pre-math­e­mat­i­cal de­scrip­tions of some­thing we’d like our de­ci­sion the­ory to con­sider, and that to try to di­rectly ap­ply them to a math­e­mat­i­cal the­ory is to com­mit a sort of type er­ror.

Speci­fi­cally, a de­ci­sion-mak­ing pro­ce­dure that “has a dis­po­si­tion to smoke” is not FDT. It is some other de­ci­sion the­ory that has the ca­pa­bil­ity to op­er­ate in un­cer­tainty about its own dis­po­si­tions.

I think it’s to­tally rea­son­able to say that we want to re­search de­ci­sion the­o­ries that are ca­pa­ble of this, be­cause this epistemic state of not be­ing quite sure of your own mind is some­thing hu­mans have to deal with all the time. But one can­not start with a math­e­mat­i­cally speci­fied de­ci­sion the­ory like proof-based UDT or causal-graph-based CDT and then ask “what it would do if it had the smok­ing le­sion.” It’s a ques­tion that seems in­tu­itively rea­son­able but, when made pre­cise, is non­sense.

I think what this feels like to philoso­phers is giv­ing the ver­bal con­cepts pri­macy over the math. (With pos­i­tive as­so­ci­a­tions to “con­cepts” and nega­tive as­so­ci­a­tions to “math” im­plied). But what it leads to in prac­tice is peo­ple say­ing “but what about the tickle defense?” or “but what about differ­ent for­mu­la­tions of CDT” as if they were talk­ing about differ­ent facets of unified con­cepts (the things that are sup­posed to have pri­macy), when these facets have to­tally dis­tinct math­ema­ti­za­tions.

At some point, if you know that a tree fal­ling in the for­est makes the air vibrate but doesn’t lead to au­di­tory ex­pe­riences, it’s time to stop wor­ry­ing about whether it makes a sound.

So ob­vi­ously I (and LW or­tho­doxy) are on the pro-math side, and I think most philoso­phers are on the pro-con­cepts side (I’d say “pro-essences,” but that’s a bit too on the nose). But, im­por­tantly, if we agree that this de­scrip­tive differ­ence ex­ists, then we can at least work to bridge it by be­ing clear about whether were’s us­ing the math per­spec­tive or the con­cept per­spec­tive. Then we can keep differ­ent math­ema­ti­za­tions strictly sep­a­rate when us­ing the math per­spec­tive, but work to amalga­mate them when talk­ing about con­cepts.

They’re definitely not suffi­cient, al­most cer­tainly. A full fledged di­a­mond max­i­mizer would need far more ma­chin­ery, if only to do the max­i­miza­tion and prop­erly learn the rep­re­sen­ta­tion.

Clar­ifi­ca­tion: I meant (but in­ad­e­quately ex­pressed) “do you think any rea­son­able ex­ten­sion of these kinds of ideas could get what we want?” Ob­vi­ously, it would be a quite un­fair de­mand for rigor to de­mand whether we can do the thing right now.

Thanks for the great re­ply. I think the re­main­ing dis­agree­ment might boil down to the ex­pected difficulty of avoid­ing Good­hart here. I do agree that us­ing rep­re­sen­ta­tions is a way around this is­sue, and it isn’t the rep­re­sen­ta­tion learn­ing ap­proach’s job to si­mul­ta­neously deal with Good­hart­ing.

Is this se­quence go­ing to be­come a se­quence in the less­wrong con­tent library at some point? I kind of like hav­ing things in the library page so I can go back and read the whole thing later but I no­ticed it’s not there yet.

Ah, I see. Then yes, I agree with you that bet­ter In­box man­age­ment for no­tifi­ca­tions in gen­eral seems quite valuable. As I men­tioned, we have an up­com­ing re­work of the no­tifi­ca­tion sys­tem in gen­eral, so af­ter that is done, I will look into do­ing that.

Also agree with back­links. We are about to roll out hover link-pre­views, which is a re­lated fea­ture, so I think we can do that soon.

Re the last linked fea­ture: We also have some drafts and plans for a re­act-sys­tem, that I am pretty ex­cited about im­ple­ment­ing, but held off on for the last three months be­cause we de­cided to fo­cus on some stuff that would con­flict with that, but I ex­pect us to get around to that soon.

Con­vert­ing this from a Face­book com­ment to LW Short­form.

A friend com­plains about re­cruiters who send re­peated emails say­ing things like “just bump­ing this to the top of your in­box” when they have no right to be try­ing to pri­ori­tize their emails over ev­ery­thing else my friend might be re­ceiv­ing from friends, travel plans, etc. The truth is they’re sim­ply paid to spam.

Some dis­cus­sion of re­peated mes­sag­ing be­hav­ior en­sued. Th­ese are my thoughts:

I feel con­flicted about re­peat­edly mes­sag­ing peo­ple. All the fol­low­ing be­ing fac­tors in this con­flict:

• Re­peat­edly mes­sag­ing can be mak­ing your­self an ass­hole that gets through some­one’s un­for­tu­nate ass­hole filter.

• There’s an an­gle from which re­peat­edly, man­u­ally mes­sag­ing peo­ple is a costly sig­nal bid that their re­sponse would be valuable to you. Ad­mit­tedly this might not filter in the de­sired ways.

• I know that many peo­ple are in fact di­s­or­ga­nized and lose emails or oth­er­wise don’t have sys­tems for get­ting back to you such that failure to get back to you doesn’t mean they didn’t want to.

• There are other peo­ple have ex­tremely good sys­tems I’m always im­pressed by the su­per busy, su­per well-known peo­ple who get back to you re­li­ably af­ter three weeks. Sys­tems. I don’t always know where some­one falls be­tween “has no sys­tems, re­lies on other peo­ple to mes­sage re­peat­edly” vs “has im­pec­ca­ble sys­tems but due to vol­ume of emails will take two weeks.”

  • The over­all in­cen­tives are such that most peo­ple prob­a­bly shouldn’t gen­er­ally re­veal which they are.

• Some­times the only way to get things done is to bug peo­ple. And I hate it. I hate nag­ging, but given other peo­ple’s un­re­li­a­bil­ity, it’s ei­ther you bug­ging them or a good chance of not get­ting some im­por­tant thing.

  • A wise, well-re­spected, busi­ness-ex­pe­rienced ra­tio­nal­ist told me many years ago that if you want some­thing from some­one you, you should just email them ev­ery day un­til they do it. It feels like this is the wis­dom of the busi­ness world. Yet . . .

• Some­times I sign up for a free trial of an en­ter­prise product and, my god, if you give them your email af­ter hav­ing ex­pressed the tiniest in­ter­est, they will keep emailing you for­ever with es­ca­lat­ingly at­ten­tion-grabby and en­ti­tled sub­ject ti­tles. (Like re­cruiters but much worse.) If I was be­ing smart, I’d have a sys­tem which filters those emails, but I don’t, and so they are an­noy­ing. I don’t want to pat­tern match to that kind that of be­hav­ior.

  • Some­times I think I won’t pat­tern match to that kind of spam be­cause I’m differ­ent and my mes­sage is differ­ent, but then the rest of the LW team cau­tions me that such differ­ences are in my mind but not nec­es­sar­ily the re­cip­i­ent tho whom I’m an­noy­ing.

  • I sus­pect as a whole they lean too far in the di­rec­tion of avoid­ing be­ing ass­holes while at the risk of not get­ting things done while I’m bi­ased in the re­verse di­rec­tion. I sus­pect this comes from my pre­vi­ous most re­cent work ex­pe­rience be­ing in the “busi­ness world” where ruth­less, self­ish, ass­hole norms pre­vail. It may be I dial it back from that but still end up seem­ing brazen to peo­ple with less im­mer­sion in that world; prob­a­bly, over­all, cul­tural pri­ors and in­di­vi­d­ual differ­ences heav­ily shape how mes­sag­ing be­hav­ior is in­ter­preted.

So it’s hard. I try to judge on a case by case ba­sis, but I’m usu­ally erring in one di­rec­tion or an­other with a fear in one di­rec­tion or the other.

A heuris­tic I heard in this space is to mes­sage re­peat­edly but with an ex­po­nen­tial de­lay fac­tor each time you don’t get a re­sponse, e.g. mes­sage again af­ter one week, if you don’t get a re­ply, mes­sage again af­ter an­other two weeks, then four weeks, etc. Even­tu­ally, you won’t be bug­ging who­ever it is.

Re­lated:

• Dis­cus­sion of whether pay­ing peo­ple to be able to send them emails or pay­ing if they re­ply can solv­ing the var­i­ous bid-for-at­ten­tion prob­lems in­volved with emails.

By soon I mean 5 years. In­ter­est­ingly, I have a slightly higher prob­a­bil­ity that it will be solved within 20 years, which high­lights the difficulty of say­ing am­bigu­ous things like “soon.”

Foiled! I was go­ing to write a ‘scathing’ com­ment about this not be­ing ap­pro­pri­ate for cross-post­ing here.

(I loved the post. I think it does a great job at ges­tur­ing at the kinds of things I’d ex­pect you to in­clude in an even­tual “fun­da­men­tals-level guide on sports”.)

Can you quan­tify soon :) ? For ex­am­ple, I’d be will­ing to bet at ¹⁄₃ odds that this will be solved in the next 10 years con­di­tional on a cer­tain amount of effort be­ing put in and more like ¹⁄₁ odds for the next 20 years. It’s hard to quan­tify the con­di­tional piece but I’d cash it out as some­thing like if re­searchers put in the same amount of effort into this that they put into NLP/​image recog­ni­tion bench­marks. I don’t think that’ll hap­pen, so this is purely a coun­ter­fac­tual claim, but maybe it will help ground any sub­se­quent dis­cus­sion with some sort of con­crete claim?

Sure, but let me clar­ify that I’m prob­a­bly not draw­ing as hard a bound­ary be­tween “or­di­nary para­noia” and “deep se­cu­rity” as I should be. I think Bruce Sch­neier’s and Eliezer’s buck­ets for “se­cu­rity mind­set” blended to­gether in the months since I read both posts. Also, re-read­ing the lo­gis­tic suc­cess curve post re­minded me that Eliezer calls into ques­tion whether some­one who lacks se­cu­rity mind­set can iden­tify peo­ple who have it. So it’s worth not­ing that my abil­ity to iden­tify peo­ple with se­cu­rity mind­set is it­self sus­pect by this crite­ria (there’s no pub­lic ev­i­dence that I have se­cu­rity mind­set and I wouldn’t claim that I have a con­sis­tent abil­ity to do “deep se­cu­rity”-style anal­y­sis.)

With that out of the way, here are some of the ex­am­ples I was think­ing of.

First of all, at a high level, I’ve no­ticed that you seem to con­sis­tently ques­tion as­sump­tions other posters are mak­ing and clar­ify ter­minol­ogy when ap­pro­pri­ate. This seems like a pre­req­ui­site for se­cu­rity mind­set, since it’s a nec­es­sary first step to­wards con­struct­ing sys­tems.

Se­cond and more sub­stan­tively, I’ve seen you con­sis­tently raise con­cerns about hu­man safety prob­lems (also here. I see this as an ex­am­ple of se­cu­rity mind­set be­cause it re­quires ques­tion­ing the as­sump­tions im­plicit in a lot of pro­pos­als. The anal­ogy to Eliezer’s post here would be that or­di­nary para­noia is try­ing to come up with more ways to pre­vent the AI from cor­rupt­ing the hu­man (or some­thing similar) whereas I think a deep se­cu­rity solu­tion would look more like avoid­ing the as­sump­tion that hu­mans are safe al­to­gether and in­stead seek­ing clear guaran­tees that our AIs will be safe even if we our­selves aren’t.

Last, you seem to be un­usu­ally will­ing to point out flaws in your own pro­pos­als, the prime ex­am­ple be­ing UDT. The most re­cent ex­am­ple of this is your com­ment about the bomb ar­gu­ment, but I’ve seen you do this quite a bit and could find more ex­am­ples if prompted. On re­flec­tion, this may be more of an ex­am­ple of “or­di­nary para­noia” than “deep se­cu­rity”, but it’s still quite im­por­tant in my opinion.

Let me know if that clar­ifies things at all. I can prob­a­bly come up with more ex­am­ples of each type if re­quested, but it will take me some time to keep dig­ging through posts and com­ments so figured I’d check in to see if what I’m say­ing makes sense be­fore con­tin­u­ing to dig.

http://​​www.over­com­ing­bias.com/​​2007/​​05/​​policy_tu­gowar.html

Made me re­al­ize that the things that are op­ti­mized for at­ten­tion aren’t the things op­ti­mized for ac­com­plish­ing any­thing use­ful. I’ve since pur­sued many con­trar­ian rab­bit holes, some of which were of enor­mous benefit.

Some­how you’re ex­pect­ing to get a lot of in­for­ma­tion about task B from perfor­mance on task A

Are “A” and “B” back­wards here, or am I not fol­low­ing?

Oh okay, then yours is definitely su­pe­rior. Good thing it’s mine now.

though not that many users use the LessWrong PM sys­tem, so I am kind of hes­i­tant to op­ti­mize that part of the site su­per much.

I’m ac­tu­ally think­ing more about com­ments than PMs. There are com­ments (ei­ther ad­dressed to me or just gen­eral com­ments) that I want to make sure to even­tu­ally read/​di­gest/​re­spond to, and right now there’s no good way to keep track of that.

Is there any­thing else in this thread that you think we should pri­ori­tize?

Not sure what you mean by “this thread”, but I still want to see the back­links fea­ture, and this.

I added an “archive” but­ton to the PM sys­tem a few months ago, which I was hop­ing would serve some of the func­tion you were re­quest­ing. Ad­ding some kind of tags or flags to it also seems like a rea­son­able im­prove­ment, though not that many users use the LessWrong PM sys­tem, so I am kind of hes­i­tant to op­ti­mize that part of the site su­per much.

Re de­vel­op­ment speed: I think it’s mostly that we’ve been pri­ori­tiz­ing other stuff. I.e. I’ve been re­vamp­ing a bunch of the map in­fras­truc­ture and meetup in­fras­truc­ture re­cently, which I think is some­thing you are not su­per in­ter­ested in, but I think is pretty im­por­tant for other rea­sons. Most of our fea­ture pri­ori­ti­za­tion is a re­sult of some long-term vi­sions, and user in­ter­views we make. The cur­rently two most re­quested fea­tures we are work­ing on is adding Google Docs-like shar­ing func­tion­al­ity to posts, which is a rel­a­tively ma­jor pro­ject, as well as a re­vamp of the no­tifi­ca­tion sys­tem that also sup­ports more gran­u­lar email no­tifi­ca­tions, nei­ther of which are men­tioned in this thread, but are definitely the most re­quested fea­tures via all the other chan­nels we have for peo­ple mak­ing fea­ture re­quests.

(And ac­tu­ally, check­ing the rest of this thread, there isn’t a sin­gle fea­ture re­quest on it that I feel like I want to pri­ori­tize, that isn’t already im­ple­mented, be­sides the two com­ments you made to­day. Is there any­thing else in this thread that you think we should pri­ori­tize?)

I would like to see some con­crete ex­am­ples of this DDDis­course. And with fewer Zs :)

We’ve been work­ing on a re­vamp of the no­tifi­ca­tion sys­tem, which is at least re­lated to this. Would email no­tifi­ca­tions with a daily di­gest (and set­tings to be in­formed of cer­tain events in­di­vi­d­u­ally and im­me­di­ately) be suffi­cient for a lot of these, or is there some­thing spe­cific about the push-no­tifi­ca­tions from the app that would help with this?

I’ve been think­ing about an app for a while. For no­tifi­ca­tions, I ac­tu­ally think just mak­ing LW a full pro­gres­sive web app and us­ing the no­tifi­ca­tions API of mo­bile browsers seems like the best choice to me, but not con­fi­dent. I don’t think any­one on the team has su­per much ex­pe­rience with na­tive App de­vel­op­ment, so build­ing a full na­tive app would definitely be at least a bit of a learn­ing effort.

This post con­tains my an­swer.

Can you give some spe­cific ex­am­ples of me hav­ing se­cu­rity mind­set, and why they count as hav­ing se­cu­rity mind­set? I’m ac­tu­ally not en­tirely sure what it is or that I have it, and would be hard pressed to come up with such ex­am­ples my­self. (I’m pretty sure I have what Eliezer calls “or­di­nary para­noia” at least, but am con­fused/​skep­ti­cal about “deep se­cu­rity”.)

Com­bin­ing hash func­tions is ac­tu­ally trick­ier than it looks, and some peo­ple are do­ing re­search in this area and de­ploy­ing solu­tions. See https://​​crypto.stack­ex­change.com/​​a/​​328 and https://​​tahoe-lafs.org/​​trac/​​tahoe-lafs/​​wiki/​​OneHun­dredYearCryp­tog­ra­phy. It does seem that if cryp­tog­ra­phy peo­ple had more of a se­cu­rity mind­set (that are not be­ing defeated) then there would be more re­search and de­ploy­ment of this already.