An1lam

• An1lam 15 Oct 2019 23:28 UTC

  3 points

  in reply to: lsusr's comment on: ML is an in­effi­cient market

  Got it—I agree dis­cussing fur­ther prob­a­bly doesn’t make sense with­out a con­crete thing to talk about.

• An1lam 15 Oct 2019 21:03 UTC

  3 points

  on: ML is an in­effi­cient market

  Not say­ing you’re wrong or that your ex­pe­rience is in­valid but this does not match my ex­pe­rience try­ing to do ap­plied ML work or ML re­search. (More ex­pe­rienced peo­ple feel free to chime in and tell me how wrong I am...) Granted, I’m not that ex­pe­rienced, but my ex­pe­rience so far has been that most ideas I’ve had are ei­ther ideas that peo­ple had and tried or ac­tu­ally pub­lished on.

  Fur­ther, my sense is that mul­ti­ple ex­pe­rienced re­searchers of­ten have all had similar ideas and the one who suc­ceeds is the one who over­comes some key ob­sta­cle that oth­ers failed to over­come pre­vi­ously, so it’s not just that I’m new/​bad.

  I do think this mostly ap­plies to ar­eas that ML re­searchers think are in­ter­est­ing and some­what tractable. So, for ex­am­ple, I wouldnt make this claim about safety re­search and don’t know how much it ap­plies to ges­ture recog­ni­tion in par­tic­u­lar.

  That said, since you’re claiming your tools are bet­ter, I will note that the ML com­mu­nity does seem open to switch­ing tools in gen­eral, as ev­i­denced by the Theano to Ten­sorflow /​ PyTorch (some­what) shifts of the past few years.

  Would you be will­ing to de­scribe at least at a high level what these tools let you do?

  Edit: I’m more skep­ti­cal of the ob­ject-level claims than the ti­tle claim. As­sum­ing you’re us­ing the clas­si­cal defi­ni­tion of effi­cient mar­ket, I agree that “ML tool­ing” adop­tion doesn’t fol­low effi­cient mar­ket dy­nam­ics in at least one re­spect.

• An1lam 14 Oct 2019 14:50 UTC

  2 points

  on: An1lam’s Short Form Feed

  Thing I des­per­ately want: tablet na­tive spaced rep­e­ti­tion soft­ware that lets me draw flash­cards. Cloze dele­tions are just boxes or hand-drawn oc­clu­sions.

• An1lam 11 Oct 2019 2:43 UTC

  1 point

  on: Gears vs Behavior

  Good post.

  I agree with your point re­gard­ing ML’s his­tor­i­cal fo­cus on black­box pre­dic­tion. That said, there has been some in­trigu­ing re­cent work (ex­am­ple 1, ex­am­ple 2), which I’ve only just started look­ing at, on try­ing to learn causal mod­els.

  I bring this up be­cause I think the ques­tion of how causal model learn­ing hap­pens and how learn­ing sys­tems can do it may po­ten­tially be rele­vant to the work you’ve been writ­ing about. It’s pri­mar­ily of in­ter­est to me for differ­ent rea­sons, re­lated to ap­ply­ing ML sys­tems to sci­en­tific dis­cov­ery. In par­tic­u­lar, in do­mains where com­ing up with causal hy­pothe­ses is harder at scale.

• An1lam 9 Oct 2019 0:06 UTC

  1 point

  in reply to: Pattern's comment on: Are there tech­ni­cal/​ob­ject-level fields that make sense to re­cruit to LessWrong?

  Fixed, same HTTPS prob­lem Rae­mon com­mented on above.

• An1lam 7 Oct 2019 16:10 UTC

  1 point

  in reply to: gwern's comment on: What are your strate­gies for avoid­ing micro-mis­takes?

  Just wanted to note this was definitely helpful and not too gen­eral. Weirdly enough, I’ve read parts of the SRE book but for some rea­son was com­part­men­tal­iz­ing it in my “en­g­ineer­ing” bucket rather than see­ing the con­nec­tion you pointed out.

• An1lam 5 Oct 2019 23:25 UTC

  29 points

  in reply to: Viliam's comment on: De­bate on In­stru­men­tal Con­ver­gence be­tween LeCun, Rus­sell, Ben­gio, Zador, and More

  Meta: This is in re­sponse to both this and com­ments fur­ther up the chain re­gard­ing the level of the de­bate.

  It’s worth not­ing that, at least from my per­spec­tive, Ben­gio, who’s definitely not in the LW bub­ble, made good points through­out and did a good job of mod­er­at­ing.

  On the other hand, Rus­sell, ob­vi­ously more par­tial to the LW con­sen­sus view, threw out some “zingers” early on (such as the fol­low­ing one) that didn’t de­rail the de­bate but eas­ily could’ve.

  Thanks for clear­ing that up—so 2+2 is not equal to 4, be­cause if the 2 were a 3, the an­swer wouldn’t be 4? I sim­ply pointed out that in the MDP as I defined it, switch­ing off the hu­man is the op­ti­mal solu­tion, de­spite the fact that we didn’t put in any emo­tions of power, dom­i­na­tion, hate, testos­terone, etc etc. And your solu­tion seems, well, frankly ter­rify­ing, al­though I sup­pose the NRA would ap­prove.

• An1lam 5 Oct 2019 17:21 UTC

  1 point

  in reply to: Pattern's comment on: [Link] Tools for thought (Ma­tuschak & Niel­son)

  The lat­ter. To be clear, ex­er­cises are great but I think they’re of­ten not enough, in par­tic­u­lar for top­ics where it’s harder to build in­tu­ition just by think­ing. The vi­su­al­iza­tions in that post would be an ex­am­ple of a pro­to­type of the sorts of vi­su­al­iza­tions I’d want for a data-heavy topic.

  Re­gard­ing text­book prob­lems,the sub­set of things for which text­book prob­lems sub­sti­tute for rather than com­ple­ment in­ter­ac­tive vi­su­al­iza­tions seems rel­a­tively small, es­pe­cially out­side of more the­o­ret­i­cal do­mains. Even for some­thing like math, imag­ine if an ad­di­tion to ex­er­cises, your text­book let you play with 3Blue1Brown-style vi­su­al­iza­tions of the thing you’re learn­ing about.

  To give an­other ex­am­ple, say I’m learn­ing about eco­nomics at the in­tro level. Typ­i­cal text­books will have ques­tions about sup­ply & de­mand curves, diminish­ing marginal util­ity, etc. My claim is that most peo­ple will build a deeper un­der­stand­ing of these con­cepts by hav­ing ac­cess to some sort of in­ter­ac­tive mod­els to probe in ad­di­tion to the stan­dard ex­er­cises at the end of the chap­ter.

• An1lam 5 Oct 2019 4:38 UTC

  2 points

  in reply to: JohnBuridan's comment on: [Link] Tools for thought (Ma­tuschak & Niel­son)

  One is­sue I have is his idea that the medium of con­tent should be mnemon­i­cally based. This both­ers me be­cause I pre­sume that if your con­tent is re­ally good, pro­fes­sion­als and ex­perts will read it as well. And since the way that they read is differ­ent from the man­ner of a novice, they should be able to ig­nore and not be in­ter­rupted or slowed by tools de­signed for novices. I feel like there are two re­but­tals to this:

  1. In both this and pre­vi­ous es­says, Michael Niel­sen has speci­fi­cally ad­dressed the point that tools of thought should scale up to re­search-level think­ing, so he (and pre­sum­ably Ma­tuschak) are aware of the is­sue of only ap­peal­ing to novices.

  2. I read the post you linked and I agree with your points about well-writ­ten books’ table of con­tents act­ing as a mem­ory aid and out­line. But, Ma­tuschak’s points about books still ring true to me. Just be­cause books are great tools for learn­ing doesn’t mean we can’t do bet­ter. To give one ex­am­ple of a limi­ta­tion of books out­side of mem­ory not ad­dressed by your post, books don’t provide any way for me to an­swer ques­tions about the ideas be­ing dis­cussed be­yond what I can vi­su­al­ize in my head (in par­tic­u­lar in cases where the ideas are heav­ily quan­ti­ta­tive). To give one ex­am­ple of what this could look like, Bret Vic­tor has posed the thought ex­per­i­ment of what it would be like if you could ex­plore mod­els of cli­mate change as you read about them.

• An1lam 5 Oct 2019 4:23 UTC

  1 point

  in reply to: MakoYass's comment on: What are your strate­gies for avoid­ing micro-mis­takes?

  This is a great point. Not sure why I phrased it the way I did origi­nally in ret­ro­spect. I up­dated the ques­tion to re­flect your point.

• An1lam 5 Oct 2019 0:12 UTC

  1 point

  in reply to: Dagon's comment on: What are your strate­gies for avoid­ing micro-mis­takes?

  Yeah, with cod­ing, unit test­ing plus as­ser­tions plus check­ing my in­tu­itions against the code as John Went­worth de­scribed does in fact seem to work fairly well for me. I think the difficulty with alge­bra is that there’s not always an ob­vi­ous sec­ondary check you can do.

[Question] What are your strate­gies for avoid­ing micro-mis­takes?

• An1lam 30 Sep 2019 2:17 UTC

  4 points

  in reply to: Raemon's comment on: Are there tech­ni­cal/​ob­ject-level fields that make sense to re­cruit to LessWrong?

  Yeah, I have a ton of con­fir­ma­tion bias push­ing me to agree with this (be­cause for me the two are definitely re­lated), but I’ll add that I also think spend­ing a lot of time pro­gram­ming helped me make re­duc­tion­ism “a part of me” in a way it wasn’t be­fore. There are just very few other ac­tivi­ties where you’re forced to ex­press what you want or a con­cept to some­thing that fun­da­men­tally can only un­der­stand a limited log­i­cal vo­cab­u­lary. Math is similar but I think pro­gram­ming makes the re­duc­tion­ist el­e­ment more salient be­cause of the com­piler and be­cause pro­gram­ming tends to in­volve more mun­dane work.

• An1lam 30 Sep 2019 2:11 UTC

  2 points

  in reply to: Spiracular's comment on: Are there tech­ni­cal/​ob­ject-level fields that make sense to re­cruit to LessWrong?

  Wow, thanks for your de­tailed re­ply! I’m go­ing to just sort of re­ply to a ran­dom sam­pling of stuff you said (hope that’s OK).

  I sus­pect one thing that might ap­peal to these sorts of peo­ple, which we have a chance of be­ing able to provide, is an in­ter­est­ing ap­plied-re­searcher-tar­geted semi-plain-lan­guage (or highly-vi­sual, or flow-chart/​check­list, or oth­er­wise ac­cessibly pre­sented) ex­pla­na­tion of cer­tain as­pects of statis­tics that are par­tic­u­larly likely to be rele­vant to these fields.

  Makes sense, I’ve been learn­ing more statis­tics re­cently and would have ap­pre­ci­ated some­thing like this too.

  Small sam­ple sizes, but I think in the biol­ogy refer­ence class, I’ve seen more peo­ple bounce off of Eliezer’s writ­ing style than the pro­gram­ming refer­ence class does (fairly typ­i­cal “reads-as-ar­ro­gant” stuff; I didn’t per­son­ally bounce off it, so I’m trans­mit­ting this sec­ond­hand). I don’t think there’s any­thing to be done about this; just shar­ing the im­pres­sion. Per­son­ally, I’ve felt mo­ments of an­noy­ance with ran­dom LWers who re­ally don’t have an in­tu­itive feel for the nu­ances for evolu­tion, but Eliezer is ac­tu­ally one of the peo­ple who seems to have a re­ally solid grasp on this par­tic­u­lar topic.

  Spec­u­la­tion but do you think this might also be be­cause peo­ple in more ap­plied sci­ences tend to be more skep­ti­cal of long chains of rea­son­ing in gen­eral? My sense is that do­ing biol­ogy (or chemisty) lab work gives you a mostly healthy but strong skep­ti­cism of the­o­riz­ing with­out feed­back loops be­cause the­o­riz­ing about biol­ogy is so hard.

  Net­work­ing and ca­reer-de­vel­op­ment-wise… quite frankly, I think we have some, but not a ton to offer biol­o­gists di­rectly.

  That’s fair. I do think it’s worth dis­t­in­guish­ing be­tween the ra­tio­nal­ist com­mu­nity in a spe­cific case and LW it­self, even though they’re ob­vi­ously strongly over­lap­ping. I say this be­cause I can imag­ine a world where LW at­tracts a mostly so­cially sep­a­rate group of biol­ogy-in­ter­ested folks who post and en­gage but don’t nec­es­sar­ily live in Berkeley.

• An1lam 28 Sep 2019 21:36 UTC

  1 point

  on: At­tain­able Utility The­ory: Why Things Matter

  Kind of weird ques­tion: what are you us­ing to write these? A tablet of some sort pre­sum­ably?

• An1lam 27 Sep 2019 3:19 UTC

  3 points

  on: An1lam’s Short Form Feed

  To­day I at­tended the first of two talks in a two-part mini-work­shop on Vari­a­tional In­fer­ence. It’s in­ter­est­ing to think of from the per­spec­tive of my re­cent mus­ings about more sci­ence-y vs. en­g­ineer­ing mind­sets be­cause it high­lighted the im­por­tance of en­g­ineer­ing/​al­gorith­mic progress in widen­ing Bayesian meth­ods’ applicability

  The pre­sen­ter, who’s a fairly well known figure in prob­a­bil­is­tic ML and has de­vel­oped some well known statis­ti­cal in­fer­ence al­gorithms, talked about how part of the rea­son so much time was spent de­bat­ing philo­soph­i­cal is­sues in the past was be­cause Bayesian in­fer­ence wasn’t com­pu­ta­tion­ally tractable un­til the de­vel­op­ment of Gibbs Sam­pling in the ’90s by Gelfand & Smith.

  To be clear, the type of progress I’m talk­ing about is still “sci­en­tific” in the sense of it mostly in­volves ap­plied math and find­ing good ways to ap­prox­i­mate pos­te­rior dis­tri­bu­tions. But, it’s “en­g­ineer­ing” in the sense that it’s the messy sort of work I talked about in my other post, where messy means a lot of the meth­ods don’t have a good the­o­ret­i­cal back­ing and in­volve mak­ing ques­tion­able (at least ex ante) statis­ti­cal as­sump­tions. Now, the counter is of course that we don’t have a the­o­ret­i­cal back­ing yet, but there still may be one in the fu­ture.

  I’ll prob­a­bly have more to say about this when the work­shop’s over but I partly just wanted to record my thoughts while they were fresh.

• An1lam 23 Sep 2019 19:43 UTC

  1 point

  in reply to: crabman's comment on: An1lam’s Short Form Feed

  From cur­sory Googling, it looks like ten­sor net­works are mostly used for un­der­stand­ing quan­tum sys­tems. I’m not op­posed to learn­ing about them, but is there a good re­source you can point me to that in­tro­duces them in­de­pen­dent of the physics con­cepts? Were you learn­ing them for use in physics?

  For ex­am­ple, have you hap­pened to read this Google AI pa­per in­tro­duc­ing their Ten­sorNet­works library and giv­ing an overview?

• An1lam 22 Sep 2019 20:07 UTC

  4 points

  on: An1lam’s Short Form Feed

  ML-re­lated math trick: I find it eas­ier to imag­ine a 4D ten­sor, say of di­men­sions $B\times F\times M\times N$, as a big ma­trix with di­men­sions $B\times F$ within which are nested ma­tri­ces of di­men­sions $M\times N$. The nice thing about this is, at least for me, it makes it eas­ier to imag­ine ap­ply­ing op­er­a­tions over the $B\times F$ ma­tri­ces in par­allel, which is some­thing I’ve had to thing about a num­ber of times do­ing ML-re­lated pro­gram­ming, e.g. try­ing to figure out how write the code to ap­ply a 1D con­volu­tion-like op­er­a­tion to an en­tire batch in par­allel.

• An1lam 21 Sep 2019 1:33 UTC

  5 points

  in reply to: Raemon's comment on: Refram­ing Impact

  Yup, I have (and the un­trol­lable math­e­mat­i­cian one). I dashed off that com­ment but re­ally meant some­thing like, “I hope this trend takes off.”

• An1lam 20 Sep 2019 22:38 UTC

  7 points

  on: Refram­ing Impact

  I en­joyed the post and in par­tic­u­lar re­ally liked the illus­trated for­mat. Definitely plan­ning to read the rest!

  I’m now wish­ing more tech­ni­cal blog posts were illus­trated like this...