johnswentworth

• johnswentworth 13 Dec 2019 22:36 UTC

  LW: 2 AF: 1

  AF

  in reply to: G Gordon Worley III's comment on: Ex­am­ples of Causal Abstraction

  Yes definitely. I’ve omit­ted ex­am­ples from soft­ware and math be­cause there’s no “fuzzi­ness” to it; that kind of ab­strac­tion is already bet­ter-un­der­stood than the more prob­a­bil­is­ti­cally-fla­vored use-cases I’m aiming for. But the the­ory should still ap­ply to those cases, as the limit­ing case where prob­a­bil­ities are 0 or 1, so they’re use­ful as a san­ity check.

• johnswentworth 13 Dec 2019 22:32 UTC

  2 points

  in reply to: Hazard's comment on: Causal Ab­strac­tion Toy Model: Med­i­cal Sensor

  “it de­pends on what the dis­tri­bu­tions are, but there is an­other sim­ple stat you can com­puter from the $M_i$, which com­bined with their av­er­age, gives you all the info you need”

  Yes, as­sum­ing it’s a max­i­mum en­tropy dis­tri­bu­tion (e.g. nor­mal, dirich­let, beta, ex­po­nen­tial, ge­o­met­ric, hy­per­ge­o­met­ric, … ba­si­cally all the dis­tri­bu­tions we typ­i­cally use as fun­da­men­tal build­ing blocks). If it’s not a max­i­mum en­tropy dis­tri­bu­tion, then the rele­vant in­for­ma­tion can’t be sum­ma­rized by a sim­ple statis­tic; we need to keep around the whole dis­tri­bu­tion P[X=x | M] for ev­ery pos­si­ble value of x. In the max­ent case, the sum­mary statis­tics are suffi­cient to com­pute that dis­tri­bu­tion, which is why we don’t need to keep around any­thing else.

• johnswentworth 13 Dec 2019 19:20 UTC

  28 points

  on: Un­der what cir­cum­stances is “don’t look at ex­ist­ing re­search” good ad­vice?

  Pos­si­bly tan­gen­tial, but I have found that the “try it your­self be­fore study­ing” method is a very effec­tive way to learn about a prob­lem/​field. It also lends a gut-level in­sight which can be use­ful for origi­nal re­search later on, even if the origi­nal at­tempt doesn’t yield any­thing use­ful.

  One ex­am­ple: my fresh­man year of col­lege, I ba­si­cally spent the whole month of win­ter break bang­ing my head against 3-sat, try­ing to find an effi­cient al­gorithm to solve it and also just gen­er­ally play­ing with the prob­lem. I knew it was NP-com­plete, but hadn’t stud­ied re­lated top­ics in any sig­nifi­cant depth. Ob­vi­ously I did not find any effi­cient al­gorithm, but that month was prob­a­bly the most valuable-per-unit-time I’ve spent in terms of un­der­stand­ing com­plex­ity the­ory. After­wards, when I prop­erly stud­ied the origi­nal NP-com­plete­ness proof for 3-sat, re­duc­tion proofs, the polyno­mial hi­er­ar­chy, etc, it was filled with mo­ments of “oh yeah, I played with some­thing like this, that’s a clever way to ap­ply it”.

  Bet­ter ex­am­ple: I’ve spent a huge amount of time build­ing mod­els of fi­nan­cial mar­kets, over the years. At one point I no­ticed some struc­tures had shown up in one model which looked an awful lot like util­ity func­tions, so I fi­nally got around to prop­erly study­ing Ar­row & De­breu-style equil­ibrium mod­els. Sure enough, I had de­rived most of it already. I even had some pieces which weren’t in the text­books (pieces es­pe­cially use­ful for fi­nan­cial mar­kets). That also nat­u­rally lead to read­ing up on more ad­vanced eco­nomic the­ory (e.g. re­cur­sive macro), which I doubt I would have un­der­stood nearly as well if I hadn’t been run­ning into the same ideas in the wild already.

  What links here?

  • Un­der what cir­cum­stances is “don’t look at ex­ist­ing re­search” good ad­vice? by Kaj_Sotala (13 Dec 2019 13:59 UTC; 55 points)

• johnswentworth 13 Dec 2019 17:39 UTC

  2 points

  in reply to: jmh's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  No prob­lem, and they are great ques­tions. :)

• johnswentworth 13 Dec 2019 4:19 UTC

  3 points

  in reply to: steve2152's comment on: Com­pu­ta­tional Model: Causal Di­a­grams with Symmetry

  Nice ex­am­ple.

  In this ex­am­ple, both sce­nar­ios yield ex­actly the same ac­tual be­hav­ior (as­sum­ing we’ve set the pa­ram­e­ters ap­pro­pri­ately), but the coun­ter­fac­tual be­hav­ior differs—and that’s ex­actly what defines a causal model. In this case, the coun­ter­fac­tu­als are “what if we in­serted a differ­ent re­sis­tor?” and “what if we ad­justed the knob on the sup­ply?”. If it’s a voltage sup­ply, then the voltage → cur­rent model cor­rectly an­swers the coun­ter­fac­tu­als. If it’s a cur­rent sup­ply, then the cur­rent → voltage model cor­rectly an­swers the coun­ter­fac­tu­als.

  Note that all the coun­ter­fac­tual queries in this ex­am­ple are phys­i­cally grounded—they are prop­er­ties of the ter­ri­tory, not the map. We can ac­tu­ally go swap the re­sis­tor in a cir­cuit and see what hap­pens. It is a mis­take here to think of “the ter­ri­tory” as just the re­sis­tor by it­self; the sup­ply is a crit­i­cal de­ter­mi­nant of the coun­ter­fac­tual be­hav­ior, so it needs to be in­cluded in or­der to talk about causal­ity.

  Of course, there’s still the ques­tion of how we de­cide which coun­ter­fac­tu­als to sup­port. That is mainly a prop­erty of the map, so far as I can tell, but there’s a big catch: some sets of coun­ter­fac­tual queries will re­quire keep­ing around far less in­for­ma­tion than oth­ers. A given ter­ri­tory sup­ports “nat­u­ral” classes of coun­ter­fac­tual queries, which re­quire rel­a­tively lit­tle in­for­ma­tion to yield ac­cu­rate pre­dic­tions to the whole query class. In this con­text, the lumped cir­cuit ab­strac­tion is one such ex­am­ple: we keep around just high-level sum­maries of the elec­tri­cal prop­er­ties of each com­po­nent, and we can an­swer a whole class of queries about voltage or cur­rent mea­sure­ments. Con­versely, if we had a few queries about the read­ings from a voltage probe, a few queries about the mass of var­i­ous cir­cuit com­po­nents, and a few queries about the num­ber of pro­tons in a wire mod 3… these all re­quire com­pletely differ­ent in­for­ma­tion to an­swer. It’s not a nat­u­ral class of queries.

  So nat­u­ral classes of queries im­ply nat­u­ral ab­stract mod­els, pos­si­bly in­clud­ing nat­u­ral causal mod­els. There will still be some choice in which queries we care about, and what in­for­ma­tion is ac­tu­ally available will play a role in that choice (i.e. even if we cared about num­ber of pro­tons mod 3, we have no way to get that in­for­ma­tion).

  I have not yet for­mu­lated all this enough to be highly con­fi­dent, but I think in this case the voltage → cur­rent model is a nat­u­ral ab­strac­tion when we have a voltage sup­ply, and vice versa for a cur­rent sup­ply. The “cor­rect” model, in each case, can cor­rectly pre­dict be­hav­ior of the re­sis­tor and knob coun­ter­fac­tu­als (among oth­ers), with­out any ad­di­tional in­for­ma­tion. The “in­cor­rect” model can­not. (I could be miss­ing some other class of coun­ter­fac­tu­als which are eas­ily an­swered by the “in­cor­rect” mod­els with­out ad­di­tional in­for­ma­tion, which is the main rea­son I’m not en­tirely sure of the con­clu­sion.)

  Thanks for bring­ing up this ques­tion and ex­am­ple, it’s been use­ful to talk through and I’ll likely re-use it later.

• johnswentworth 12 Dec 2019 23:06 UTC

  6 points

  in reply to: Rob Bensinger's comment on: Is Ra­tion­al­ist Self-Im­prove­ment Real?

  Some­thing about the “sci­ence is frag­ile” ar­gu­ment feels off to me. Per­haps it’s that I’m not re­ally think­ing about RCTs; I’m look­ing at Archimedes, New­ton, and Feyn­man, and go­ing “surely there’s some­thing small that could have been tweaked about cul­ture be­fore­hand to make some of this low-hang­ing sci­en­tific fruit get grabbed ear­lier by a bunch of de­cent thinkers, rather than ev­ery­thing need­ing to wait for lone ge­niuses”.

  I’d pro­pose that there’s a mas­sive qual­i­ta­tive differ­ence be­tween black-box re­sults (like RCTs) and gears-level model-build­ing (like Archimedes, New­ton, and Feyn­man). The lat­ter are where ba­si­cally all of the big gains are, and it does seem like so­ciety is un­der-in­vested in build­ing gears-level mod­els. One pos­si­ble eco­nomic rea­son for the un­der-in­vest­ment is that gears-level mod­els have very low de­pre­ci­a­tion rates, so they pay off over a very long timescale.

Ex­am­ples of Causal Abstraction

• johnswentworth 12 Dec 2019 20:41 UTC

  13 points

  in reply to: jmh's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  Those are all rea­son­able ques­tions to ask and points to raise, and I’m not go­ing to go to bat defend­ing any of the sug­ges­tions I made off the top of my head when writ­ing the origi­nal ques­tion. The point of the origi­nal ques­tion was to see if any­body out there had pub­li­ca­tions ask­ing/​an­swer­ing the sort of ques­tions you pose, and it looks like the an­swer is “no”.

  For some of these ques­tions, as you ar­gue, it’s pos­si­ble that the lack of liter­a­ture is be­cause there re­ally isn’t any­thing in­ter­est­ing to be found. But at least some of these ques­tions would be in­ter­est­ing to have an an­swer to re­gard­less of what the an­swer is—e.g. your ex­am­ple “Is the body’s nor­mal state of op­er­a­tion one of scarcity or non-scarcity?”.

  More gen­er­ally, a bet­ter anal­ogy than the dog pic­ture would be the pe­ri­odic table. At first glance (and cer­tainly be­fore the de­vel­op­ment of quan­tum the­ory) an ar­gu­ment could be made that it doesn’t tell us any­thing we can’t figure out with­out it. But it did hint at what ques­tions to ask—e.g. undis­cov­ered el­e­ments and their prop­er­ties. If in­sulin acts as a price sig­nal (even with­out a ra­tioning role) or if the body’s fat stores are gov­erned by an in­ter­nally-rep­re­sented dis­count rate, then that im­me­di­ately sug­gests that a va­ri­ety of differ­ent cell types would look at those sig­nals to de­ter­mine their be­hav­ior—pos­si­bly cell types and be­hav­iors not yet ex­am­ined. It also pre­dicts that cells which con­vert one re­source into an­other would ex­am­ine the cor­re­spond­ing price sig­nals, and ad­just pro­duc­tion based on rel­a­tive prices. Like the pe­ri­odic table, these ideas sug­gest re­la­tion­ships to look for.

• johnswentworth 12 Dec 2019 1:30 UTC

  2 points

  in reply to: waveman's comment on: Causal Ab­strac­tion Toy Model: Med­i­cal Sensor

  Thanks, I had con­sid­ered adding some­thing at the top but didn’t ac­tu­ally do that. Will add it now.

• johnswentworth 12 Dec 2019 0:17 UTC

  20 points

  on: Com­pet­i­tive Mar­kets as Distributed Backprop

  There are two sep­a­rate lenses through which I view the idea of com­pet­i­tive mar­kets as back­prop­a­ga­tion.

  First, it’s an ex­am­ple of the real meat of eco­nomics. Many peo­ple—in­clud­ing economists—think of eco­nomics as study­ing hu­man mar­kets and ex­change. But the the­ory of eco­nomics is, to a large ex­tent, gen­eral the­ory of dis­tributed op­ti­miza­tion. When we un­der­stand on a gut level that “price = deriva­tive”, and mar­kets are just im­ple­ment­ing back­prop, it makes a lot more sense that things like mar­kets would show up in other fields—e.g. AI or biol­ogy.

  Se­cond, com­pet­i­tive mar­kets as back­prop­a­ga­tion is a great ex­am­ple of the power of view­ing the world in terms of DAGs (or cir­cuits). I first got into the habit of think­ing about com­pu­ta­tion in terms of DAGs/​cir­cuits when play­ing with gen­er­al­iza­tions of back­prop­a­ga­tion, and the equiv­alence of mar­kets and back­prop was one of the first in­ter­est­ing re­sults I stum­bled on. Since then, I’ve re­lied more and more on DAGs (with sym­me­try) as my go-to model of gen­eral com­pu­ta­tion.

  Dan­gling Threads

  The most in­ter­est­ing dan­gling thread on this post is, of course, the ex­ten­sion to out-of-equil­ibrium mar­kets. I had origi­nally promised a post on that topic, but I no longer in­tend to ever write that post: I have gen­er­ally de­cided to avoid pub­lish­ing any­thing at all re­lated to novel op­ti­miza­tion al­gorithms. I doubt that simu­lat­ing mar­kets would be a game-changer for op­ti­miza­tion in AI train­ing, but it’s still a good idea to de­clare op­ti­miza­tion off-limits in gen­eral, so that ab­sence of an ob­vi­ous pub­li­ca­tion never be­comes a hint that there’s some­thing in­ter­est­ing to be found.

  A more sub­tle dan­gling thread is the gen­er­al­iza­tion of the idea to more com­plex sup­ply chains, in which one in­ter­me­di­ate can be used in mul­ti­ple ways. Although I’ve left the origi­nal foot­note in the ar­ti­cle, I no longer think I know how to han­dle non-tree-shaped cases. I am quite con­fi­dent that there is a way to in­te­grate this into the model, since it must hold for “price=deriva­tive” to hold, and I’m sure the economists would have no­ticed by now if “price=deriva­tive” failed in non­triv­ial sup­ply chain mod­els. That said, it’s non­triv­ial to in­te­grate into the model: if a pro­gram reads the same value in two places, then both places read the same value, whereas if two pro­cesses con­sume the same sup­ply-chain in­ter­me­di­ate, then they will typ­i­cally con­sume differ­ent amounts which add up to the amount pro­duced. Trans­lat­ing a pro­gram into a sup­ply chain, or vice versa, thus pre­sum­ably re­quires some non­triv­ial trans­for­ma­tions in gen­eral.

• johnswentworth 11 Dec 2019 22:36 UTC

  3 points

  in reply to: Dagon's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  That’s not nec­es­sary for all re­sults. It would be rele­vant to some—e.g. mon­e­tary eco­nomics (ob­vi­ously), bud­gets con­straints, and any­thing where the role of money as an in­cen­tive is cru­cial. But it’s not needed for e.g. much of price the­ory, which is the main sort of ap­pli­ca­tion I imag­ine. In­deed, if we look at Glen Weyl’s defi­ni­tion of price the­ory, it im­me­di­ately sounds like it would be ap­pli­ca­ble to many prob­lems in biol­ogy.

  (Also, I sus­pect one could work around the ab­sence of a bud­get con­straint by di­rectly ob­serv­ing the con­sump­tion func­tion.)

• johnswentworth 11 Dec 2019 22:26 UTC

  4 points

  in reply to: johnswentworth's comment on: Effect het­ero­gene­ity and ex­ter­nal val­idity in medicine

  UPDATE Dec 2019: Based on Cinelli & Pearls re­sponse to the OP (& as­so­ci­ated pa­per), it does in­deed look like all the rele­vant in­for­ma­tion can be in­te­grated into a DAG model.

  Over the course of this thread, I came to the im­pres­sion that an un­nec­es­sary fo­cus on iden­ti­fi­a­bil­ity was the main root prob­lem with the OP. Now it looks like that was prob­a­bly wrong. How­ever, based on the Cinelli & Pearl pa­per, it does look like causal DAGs + Bayesian prob­a­bil­ity (or even non-Bayesian prob­a­bil­ity, for the ex­am­ple at hand) are all we need for this use-case.

• johnswentworth 11 Dec 2019 21:54 UTC

  19 points

  on: Gen­er­al­iz­ing Ex­per­i­men­tal Re­sults by Lev­er­ag­ing Knowl­edge of Mechanisms

  That’s a re­ally nice piece. It shows how to for­mu­late the rele­vant back­ground knowl­edge in the graph­i­cal lan­guage, how to de­rive the in­tu­itive re­sults which Huit­feldt et al pre­dicted, and how the struc­tural for­mu­la­tion cor­rectly pre­dicts when the in­tu­itive re­sults fail. Well done.

Causal Ab­strac­tion Toy Model: Med­i­cal Sensor

• johnswentworth 11 Dec 2019 18:53 UTC

  9 points

  in reply to: romeostevensit's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  In case peo­ple are con­fused by this: I be­lieve the anal­ogy is that peo­ple think of eco­nomics as a the­ory of hu­man mar­kets, even though the math is largely about gen­eral prop­er­ties of dis­tributed op­ti­miza­tion.

• johnswentworth 11 Dec 2019 18:49 UTC

  2 points

  in reply to: jmh's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  See my re­sponse to leggi’s com­ment, with the dog pic­ture, as well as to Daniel V.

• johnswentworth 11 Dec 2019 18:29 UTC

  5 points

  in reply to: leggi's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  You should check out the ex­pla­na­tion of in­sulin/​glu­cose reg­u­la­tion in my re­view of De­sign Prin­ci­ples of Biolog­i­cal Cir­cuits.

  eco­nomic mod­els = made up by humans

  phys­iol­ogy = mil­len­nia of chance and adap­tions un­der var­i­ous pres­sures, cre­at­ing im­mensely com­plex sys­tems that just blow my mind.

  Again, read that re­view. A cen­tral point of the book is that evolved sys­tems re­peat­edly con­verge on similar pat­terns, be­cause those are the pat­terns which work well. Eco­nomics, on the other hand, is largely a math­e­mat­i­cal dis­ci­pline study­ing gen­eral prop­er­ties of things-which-work-well for op­ti­miza­tion. Through that lens, it makes a lot of sense to ap­ply eco­nomic mod­els to biolog­i­cal sys­tems.

  I would sug­gest re­search­ing the con­nec­tions be­tween hunger/​in­take/​di­ges­tion/​ab­sorp­tion/​blood glu­cose/​beta cells/​in­sulin/​cell up­take/​cell de­mand/​ex­er­tion/​re­nal ex­cre­tion of glu­cose/​polyuria/​poly­dip­sia—
  it’s com­plex.

  Biolog­i­cal mod­els have much to teach us. If we ask the ques­tions and no­tice the con­nec­tions rather than try­ing to make them fit with our ex­pla­na­tion.

  Con­sider this pic­ture:

  

  There’s a dog in the pic­ture. Once you see the dog, there’s still a lot go­ing on in the pic­ture, but the whole thing makes a lot more sense.

  That’s what the­ory is about. It’s not “try­ing to make [re­al­ity] fit our ex­pla­na­tion”, it’s about notic­ing hid­den struc­ture. That’s the sort of value I ex­pect eco­nomic the­ory would provide in phys­iol­ogy.

  The struc­ture of in­sulin/​glu­cose reg­u­la­tion is the struc­ture of a mar­ket. Yes, the sys­tem is com­pli­cated, and there’s a lot of mov­ing pieces—that’s true in mar­kets too. But over long timescales, the pieces are in­ter­act­ing pri­mar­ily through a price sig­nal—in­sulin—and that’s the key char­ac­ter­is­tic which makes it a mar­ket, re­gard­less of what­ever else is go­ing on lo­cally. That’s a use­ful piece of hid­den struc­ture.

• johnswentworth 11 Dec 2019 18:14 UTC

  9 points

  in reply to: Daniel V's comment on: Ap­pli­ca­tions of Eco­nomic Models to Phys­iol­ogy?

  What is the differ­ence be­tween a generic “sig­nal” and a “price sig­nal”? What is a “price” in phys­iol­ogy?

  A price sig­nal would need a few prop­er­ties:

  • It would be paired with some phys­iolog­i­cal re­source—it rep­re­sents the price of something

  • It would be non­lo­cal, like e.g. a hor­mone—the point of prices is to co­or­di­nate in a dis­tributed system

  • Lo­cal sys­tems would in­crease/​de­crease their con­sump­tion of the re­source in re­sponse to low/​high price sig­nal, and vice-versa for production

  • Some mechanism would make the price sig­nal high when the re­source is scarce, and vice versa, so that the amount of re­source sup­plied matches up with the amount de­manded.

  In­sulin is a good ex­am­ple: it acts as (in­verse) price sig­nal for glu­cose. It’s a hor­mone, and many cell types through­out the body in­crease/​de­crease their glu­cose con­sump­tion in re­sponse to in­sulin level. The beta cells in the pan­creas act as a mar­ket maker, set­ting in­sulin lev­els so that glu­cose sup­ply matches de­mand over the long run.

  Another ques­tion is which ba­sic as­sump­tions em­braced in eco­nomics can rea­son­ably ap­ply to the units of anal­y­sis in phys­iol­ogy (cells, etc.). Economists already have a hard enough time val­i­dat­ing as­sump­tions for hu­mans.

  This is a point which I think is severely un­der-ap­pre­ci­ated both in and out of eco­nomics: it is of­ten far eas­ier to ap­ply eco­nomic mod­els to sys­tems in biol­ogy than to hu­mans. Hu­mans have com­pli­cated, opaque de­ci­sion-mak­ing pro­ce­dures. We can’t ob­serve those pro­ce­dures di­rectly, and have to make in­di­rect pre­dic­tions about their effects. Cells have de­ci­sion-mak­ing pro­ce­dures which we can di­rectly ob­serve and model, and peo­ple have already built many of those mod­els.

  Con­versely, many of the de­bat­able as­sump­tions eco­nomics make about hu­mans would have di­rectly-ob­serv­able effects in biolog­i­cal sys­tems. If sub­sys­tems’ be­hav­ior can’t be de­scribed by util­ity func­tions, then that will di­rectly re­sult in re­sources con­sumed to pro­duce things and then de­stroy them with­out us­ing them. To the ex­tent that such en­ergy waste is min­i­mized, the sys­tem be­hav­ior can be ap­prox­i­mated by util­ities. Another ex­am­ple: if sub­sys­tems’ re­vealed prefer­ences aren’t con­cave, then that will di­rectly re­sult in in­sta­bil­ities in phys­iolog­i­cal be­hav­ior.

• johnswentworth 11 Dec 2019 4:17 UTC

  4 points

  in reply to: Wei_Dai's comment on: What de­ter­mines the bal­ance be­tween in­tel­li­gence sig­nal­ing and virtue sig­nal­ing?

  It seems like a lot of ex­am­ples of virtue sig­nal­ling re­quire sac­ri­fic­ing in­tel­li­gence, but sac­ri­fic­ing virtue seems like a less com­mon re­quire­ment to sig­nal in­tel­li­gence. So one pos­si­ble model would be that, rather than a pareto fron­tier on which the two trade off sym­met­ri­cally, in­tel­li­gent de­ci­sions are an in­put which are de­struc­tively con­sumed to pro­duce virtue sig­nals—like trees are con­sumed to pro­duce pa­per.

• johnswentworth 10 Dec 2019 19:45 UTC

  9 points

  in reply to: Wei_Dai's comment on: What de­ter­mines the bal­ance be­tween in­tel­li­gence sig­nal­ing and virtue sig­nal­ing?

  Could you ex­pand a bit on why you ex­pect a trade-off be­tween in­tel­li­gence/​virtue sig­nal­ling, as op­posed to two in­de­pen­dent axes? I can sort of see a case where in­tel­li­gence is the “cost” part of “costly virtue sig­nal­ling”, and virtue is the “cost” part of “costly in­tel­li­gence sig­nal­ling”, like the ex­am­ples in tox­o­plasma of rage. On the other hand, look­ing at those ex­am­ples of the dan­gers of run­away IQ sig­nal­ling, they gen­er­ally don’t seem to trade-off against virtue.