# Computational Model: Causal Diagrams with Symmetry

Con­sider the fol­low­ing pro­gram:

``````f(n):
if n == 0:
re­turn 1
re­turn n * f(n-1)
``````

Let’s think about the pro­cess by which this func­tion is eval­u­ated. We want to sketch out a causal DAG show­ing all of the in­ter­me­di­ate calcu­la­tions and the con­nec­tions be­tween them (feel free to pause read­ing and try this your­self).

Here’s what the causal DAG looks like:

Each dot­ted box cor­re­sponds to one call to the func­tion f. The re­cur­sive call in f be­comes a sym­me­try in the causal di­a­gram: the DAG con­sists of an in­finite se­quence of copies of the same sub­cir­cuit.

More gen­er­ally, we can rep­re­sent any Tur­ing-com­putable func­tion this way. Just take some pseu­docode for the func­tion, and ex­pand out the full causal DAG of the calcu­la­tion. In gen­eral, the di­a­gram will ei­ther be finite or have sym­met­ric com­po­nents—the sym­me­try is what al­lows us to use a finite rep­re­sen­ta­tion even though the graph it­self is in­finite.

## Why would we want to do this?

For our pur­poses, the cen­tral idea of em­bed­ded agency is to take these black-box sys­tems which we call “agents”, and break open the black boxes to see what’s go­ing on in­side.

Causal DAGs with sym­me­try are how we do this for Tur­ing-com­putable func­tions in gen­eral. They show the ac­tual cause-and-effect pro­cess which com­putes the re­sult; con­cep­tu­ally they rep­re­sent the com­pu­ta­tion rather than a black-box func­tion.

In par­tic­u­lar, a causal DAG + sym­me­try rep­re­sen­ta­tion gives us all the nat­u­ral ma­chin­ery of causal­ity—most no­tably coun­ter­fac­tu­als. We can ask ques­tions like “what would hap­pen if I reached in and flipped a bit at this point in the com­pu­ta­tion?” or “what value would f(5) re­turn if f(3) were 11?”. We can pose these ques­tions in a well-defined, un­am­bigu­ous way with­out wor­ry­ing about log­i­cal coun­ter­fac­tu­als, and with­out adding any ad­di­tional ma­chin­ery. This be­comes par­tic­u­larly im­por­tant for em­bed­ded op­ti­miza­tion: if an “agent” (e.g. an or­ganism) wants to plan ahead to achieve an ob­jec­tive (e.g. find food), it needs to ask coun­ter­fac­tual ques­tions like “how much food would I find if I kept go­ing straight?”.

The other main rea­son we would want to rep­re­sent func­tions as causal DAGs with sym­me­try is be­cause our uni­verse ap­pears to be one gi­ant causal DAG with sym­me­try.

Be­cause our uni­verse is causal, any com­pu­ta­tion performed in our uni­verse must even­tu­ally bot­tom out in a causal DAG. We can write our pro­grams in any lan­guage we please, but even­tu­ally they will be com­piled down to ma­chine code and run by phys­i­cal tran­sis­tors made of atoms which are them­selves gov­erned by a causal DAG. In most cases, we can rep­re­sent the causal com­pu­ta­tional pro­cess at a more ab­stract level—e.g. in our ex­am­ple pro­gram, even though we didn’t talk about reg­isters or tran­sis­tors or elec­tric fields, the causal di­a­gram we sketched out would still ac­cu­rately rep­re­sent the com­pu­ta­tion performed even at the lower lev­els.

This raises the is­sue of ab­strac­tion—the core prob­lem of em­bed­ded agency. My own main use-case for the causal di­a­gram + sym­me­try model of com­pu­ta­tion is for­mu­lat­ing mod­els of ab­strac­tion: how can one causal di­a­gram (pos­si­bly with sym­me­try) rep­re­sent an­other in a way which makes coun­ter­fac­tual queries on the map cor­re­spond to some kind of coun­ter­fac­tual on the ter­ri­tory? Can that work when the “map” is a sub­DAG of the ter­ri­tory DAG? It feels like causal di­a­grams + sym­me­try are the min­i­mal com­pu­ta­tional model needed to get agency-rele­vant an­swers to this sort of ques­tion.

## Learning

The tra­di­tional ul­ti­mate learn­ing al­gorithm is Solomonoff In­duc­tion: take some black-box sys­tem which spews out data, and look for short pro­grams which re­pro­duce that data. But the phrase “black-box” sug­gests that per­haps we could do bet­ter by look­ing in­side that box.

To make this a lit­tle bit more con­crete: imag­ine I have some python pro­gram run­ning on a server which re­sponds to http re­quests. Solomonoff In­duc­tion would look at the data re­turned by re­quests to the pro­gram, and learn to pre­dict the pro­gram’s be­hav­ior. But that sort of black-box in­ter­ac­tion is not the only op­tion. The pro­gram is run­ning on a phys­i­cal server some­where—so, in prin­ci­ple, we could go grab a screw­driver and a tiny os­cillo­scope and di­rectly ob­serve the com­pu­ta­tion performed by the phys­i­cal ma­chine. Even with­out mea­sur­ing ev­ery voltage on ev­ery wire, we may at least get enough data to nar­row down the space of can­di­date pro­grams in a way which Solomonoff In­duc­tion could not do. Ideally, we’d gain enough in­for­ma­tion to avoid need­ing to search over all pos­si­ble pro­grams.

Com­pared to Solomonoff In­duc­tion, this pro­cess looks a lot more like how sci­en­tists ac­tu­ally study the real world in prac­tice: there’s lots of tak­ing stuff apart and pok­ing at it to see what makes it tick.

In gen­eral, though, how to learn causal DAGs with sym­me­try is still an open ques­tion. We’d like some­thing like Solomonoff In­duc­tion, but which can ac­count for par­tial in­for­ma­tion about the in­ter­nal struc­ture of the causal DAG, rather than just over­all in­put-out­put be­hav­ior. (In prin­ci­ple, we could shoe­horn this whole thing into tra­di­tional Solomonoff In­duc­tion by treat­ing in­for­ma­tion about the in­ter­nal DAG struc­ture as nor­mal old data, but that doesn’t give us a good way to ex­tract the learned DAG struc­ture.)

We already have al­gorithms for learn­ing causal struc­ture in gen­eral. Pearl’s Causal­ity sketches out some such al­gorithms in chap­ter 2, al­though they’re only prac­ti­cal for ei­ther very small sys­tems or very large amounts of data. Bayesian struc­ture learn­ing can han­dle larger sys­tems with less data, though some­times at the cost of a very large amount of com­pute—i.e. es­ti­mat­ing high-di­men­sional in­te­grals.

How­ever, in gen­eral, these ap­proaches don’t di­rectly ac­count for sym­me­try of the learned DAGs. Ideally, we would use a prior which weights causal DAGs ac­cord­ing to the size of their rep­re­sen­ta­tion—i.e. in­finite DAGs would still have nonzero prior prob­a­bil­ity if they have some sym­me­try al­low­ing for finite rep­re­sen­ta­tion, and in gen­eral DAGs with mul­ti­ple copies of the same sub-DAG would have higher prob­a­bil­ity. This isn’t quite the same as weight­ing by min­i­mum de­scrip­tion length in the Solomonoff sense, since we care speci­fi­cally about sym­me­tries which cor­re­spond to func­tion calls—i.e. iso­mor­phic sub­DAGs. We don’t care about graphs which can be gen­er­ated by a short pro­gram but don’t have these sorts of sym­me­tries. So that leaves the ques­tion: if our prior prob­a­bil­ity for a causal DAG is given by a no­tion of min­i­mum de­scrip­tion length which only al­lows com­pres­sion by spec­i­fy­ing re-used sub­cir­cuits, what prop­er­ties will the re­sult­ing learn­ing al­gorithm pos­sess? Is it com­putable? What kinds of data are needed to make it tractable?

• A few com­ments:

• Be­cause of quan­tum physics, the uni­verse is ac­tu­ally not de­scribed by a causal DAG very well (if we re­quire micro­scopic pre­ci­sion). But I agree that in prac­tice there are use­ful ap­prox­i­mate de­scrip­tions that do look like a causal DAG.

• One di­rec­tion to­wards study­ing the learn­ing of sym­met­ric causal DAGs (of a sort) is my ideas about cel­lu­lar de­ci­sion pro­cesses and the more gen­eral “graph de­ci­sion pro­cesses”, see this com­ment.

• As a side note, I’m not con­vinced that “break­ing open black boxes” to solve “em­bed­ded agency” is a use­ful way of think­ing. IMO the prob­lems as­so­ci­ated with “em­bed­ded agency” should be solved us­ing a com­bi­na­tion of in­com­plete mod­els and cor­rectly deal­ing with traps.

• The quan­tum fields them­selves fol­low causal­ity just fine. Things only be­come weird if we in­tro­duce wave-func­tion col­lapse, which isn’t strictly nec­es­sary.

• Hmm, no, not re­ally. The quan­tum fields fol­low “causal­ity” in some quan­tum sense (roughly speak­ing, op­er­a­tors in spacelike sep­a­ra­tions com­mute, and any lo­cal op­er­a­tor can be ex­pressed in terms of op­er­a­tors lo­cal­ized near the in­ter­sec­tion of its past light-cone with any spacelike hy­per­sur­face), which is differ­ent from the sense used in causal DAGsa (in fact you can define “quan­tum causal DAGs” which is a differ­ent math­e­mat­i­cal ob­ject). Vio­la­tion of Bell’s in­equal­ity pre­cisely means that you can’t de­scribe the sys­tem by a causal DAG. If you want to do the MWI, then the wave­func­tion doesn’t even de­com­pose into data that can be lo­cal­ized.

• Vio­la­tion of Bell’s in­equal­ity pre­cisely means that you can’t de­scribe the sys­tem by a causal DAG

No, vi­o­la­tion of Bell’s in­equal­ity means you can’t de­scribe the sys­tem by causal in­ter­ac­tions among par­ti­cles and mea­sure­ments. If we stop think­ing about par­ti­cles and “mea­sure­ments” al­to­gether, and just talk about fields, that’s not an is­sue.

As you say, un­der MWI, the wave­func­tion doesn’t even de­com­pose into data that can be lo­cal­ized. So, in or­der to rep­re­sent the sys­tem us­ing clas­si­cal causal di­a­grams, the “sys­tem state” has to con­tain the whole wave­func­tion. As long as we can write down an equa­tion for the evolu­tion of the wave­func­tion over time, we have a clas­si­cal causal model for the sys­tem.

Quan­tum causal mod­els are cer­tainly a much cleaner rep­re­sen­ta­tion, in this case, but clas­si­cal causal mod­els can still work—we just have to define the “DAG ver­tices” ap­pro­pri­ately.

• It doesn’t re­ally have much to do with par­ti­cles vs. fields. We talk about mea­sure­ments be­cause mea­sure­ments are the thing we ac­tu­ally ob­serve. It seems strange to say you can model the world as a causal net­work if the causal net­work doesn’t in­clude your ac­tual ob­ser­va­tions. If you want to choose a par­tic­u­lar frame of refer­ence and write down the wave­func­tion time evolu­tion in that frame (while ig­nor­ing space) then you can say it’s a causal net­work (which is just a lin­ear chain, and de­ter­minis­tic at that) but IMO that’s not very in­for­ma­tive. It also loses the prop­erty of hav­ing things made of parts, which AFAIU was one of your ob­jec­tives here.

• The wave­func­tion does have plenty of in­ter­nal struc­ture, that struc­ture just doesn’t line up neatly with space. It won’t just be a lin­ear chain, and it will be made of “parts”, but those parts won’t nec­es­sar­ily line up neatly with macro­scopic ob­ser­va­tions/​ob­jects.

And that’s fine—figur­ing out how to do on­tol­ogy map­ping be­tween the low-level “parts” and the high-level “parts” is a cen­tral piece of the prob­lem. Not be­ing able to di­rectly ob­serve vari­ables in the low-level causal di­a­gram is part of that. If we want e.g. a the­ory of ab­strac­tion, then these is­sues are perfect use-cases.

• Thanks for a nice post about causal di­a­grams!

Be­cause our uni­verse is causal, any com­pu­ta­tion performed in our uni­verse must even­tu­ally bot­tom out in a causal DAG.

To­tally agree. This is a big part of the rea­son why I’m ex­cited about these kinds of di­a­grams.

This raises the is­sue of ab­strac­tion—the core prob­lem of em­bed­ded agency. … how can one causal di­a­gram (pos­si­bly with sym­me­try) rep­re­sent an­other in a way which makes coun­ter­fac­tual queries on the map cor­re­spond to some kind of coun­ter­fac­tual on the ter­ri­tory?

Great ques­tion, I re­ally think some­one should look more care­fully into this. A few po­ten­tially re­lated pa­pers:

https://​​arxiv.org/​​abs/​​1105.0158

https://​​arxiv.org/​​abs/​​1812.03789

In gen­eral, though, how to learn causal DAGs with sym­me­try is still an open ques­tion. We’d like some­thing like Solomonoff In­duc­tion, but which can ac­count for par­tial in­for­ma­tion about the in­ter­nal struc­ture of the causal DAG, rather than just over­all in­put-out­put be­hav­ior.

Again, agreed. It would be great if we could find a way to make progress on this ques­tion.

• Causal DAGs with sym­me­try are how we do this for Tur­ing-com­putable func­tions in gen­eral. They show the ac­tual cause-and-effect pro­cess which com­putes the re­sult; con­cep­tu­ally they rep­re­sent the com­pu­ta­tion rather than a black-box func­tion.

This was the main in­ter­est­ing bit for me.

• This isn’t quite the same as weight­ing by min­i­mum de­scrip­tion length in the Solomonoff sense, since we care speci­fi­cally about sym­me­tries which cor­re­spond to func­tion calls—i.e. iso­mor­phic sub­DAGs. We don’t care about graphs which can be gen­er­ated by a short pro­gram but don’t have these sorts of sym­me­tries.

Can you elab­o­rate on this? What would be an ex­am­ple of a graph that can be gen­er­ated by a short pro­gram, but that does not have these sorts of sym­me­tries?

My in­tu­ition is that the class of pro­cesses your de­scribing is Tur­ing com­plete, and there­fore can simu­late any Tur­ing ma­chine, and is thus just an­other in­stance of Solomonoff in­duc­tion with a differ­ent MDL con­stant.

Edit: Rule 110 would be an ex­am­ple.

• Were you sum­moned by this post ac­ci­den­tally us­ing your true name?

• I’ll give an an­swer via anal­ogy: the digit se­quence 123123123123123… is sym­met­ric: the se­quence di­rectly con­tains a copy of it­self. The se­quence 12345678910111213141516… is not sym­met­ric: it will not re­peat and does not con­tain a copy of it­self, al­though there is a pat­tern and it could be gen­er­ated pro­gram­mat­i­cally.

Similarly with DAG sym­me­tries. If we’re pro­duc­ing these DAGs by ex­pand­ing out pro­grams, then the only in­finite pat­terns we’ll see are sym­me­tries—sub­DAGs which re­peat, cor­re­spond­ing to func­tion blocks. We don’t need to worry about DAGs with strange pat­terns which could be gen­er­ated pro­gram­mat­i­cally but can’t just be built by re­peat­ing sub­DAGs.

• The men­tion of cir­cuits in your later ar­ti­cle re­minded me of a cou­ple ar­gu­ments I had on wikipe­dia a few years ago (2012, 2015). I was ar­gu­ing ba­si­cally that cause-effect (or at least the kind of cause-effect re­la­tion­ship that we care about and use in ev­ery­day rea­son­ing) is part of the map, not ter­ri­tory.

Here’s an ex­am­ple I came up with:

Con­sider a 1kΩ re­sis­tor, in two cir­cuits. The first cir­cuit is the re­sis­tor at­tached to a 1V power sup­ply. Here an en­g­ineer would say: “The sup­ply cre­ates a 1V drop across the re­sis­tor; and that voltage drop causes a 1mA cur­rent to flow through the re­sis­tor.” The sec­ond cir­cuit is the re­sis­tor at­tached to a 1mA cur­rent source. Here an en­g­ineer would say: “The cur­rent source pushes a 1mA cur­rent through the re­sis­tor; and that cur­rent causes a 1V drop across the re­sis­tor.” Well, it’s the same re­sis­tor … does a voltage across a re­sis­tor cause a cur­rent, or does a cur­rent through a re­sis­tor cause a voltage, or both, or nei­ther? Again, my con­clu­sion was that peo­ple think about causal­ity in a way that is not rooted in physics, and in­deed if you forced some­one to ex­clu­sively use physics-based causal mod­els, you would be hand­i­cap­ping them. I haven’t thought about it much or delved into the liter­a­ture or any­thing, but this still seems cor­rect to me. How do you see things?

• I came up with the same ex­am­ple in a pa­per on causal anal­y­sis of con­trol sys­tems. A proper causal anal­y­sis would have to open the black box of the voltage or cur­rent source and re­veal the cir­cu­lar pat­terns of cau­sa­tion within. Set up as a voltage source, it is con­tin­u­ously sens­ing its own out­put voltage and main­tain­ing it close to the refer­ence value set on the con­trol panel. Set up as a cur­rent source, it is do­ing cor­re­spond­ing things with the sensed and refer­ence cur­rents.

• Nice ex­am­ple.

In this ex­am­ple, both sce­nar­ios yield ex­actly the same ac­tual be­hav­ior (as­sum­ing we’ve set the pa­ram­e­ters ap­pro­pri­ately), but the coun­ter­fac­tual be­hav­ior differs—and that’s ex­actly what defines a causal model. In this case, the coun­ter­fac­tu­als are “what if we in­serted a differ­ent re­sis­tor?” and “what if we ad­justed the knob on the sup­ply?”. If it’s a voltage sup­ply, then the voltage → cur­rent model cor­rectly an­swers the coun­ter­fac­tu­als. If it’s a cur­rent sup­ply, then the cur­rent → voltage model cor­rectly an­swers the coun­ter­fac­tu­als.

Note that all the coun­ter­fac­tual queries in this ex­am­ple are phys­i­cally grounded—they are prop­er­ties of the ter­ri­tory, not the map. We can ac­tu­ally go swap the re­sis­tor in a cir­cuit and see what hap­pens. It is a mis­take here to think of “the ter­ri­tory” as just the re­sis­tor by it­self; the sup­ply is a crit­i­cal de­ter­mi­nant of the coun­ter­fac­tual be­hav­ior, so it needs to be in­cluded in or­der to talk about causal­ity.

Of course, there’s still the ques­tion of how we de­cide which coun­ter­fac­tu­als to sup­port. That is mainly a prop­erty of the map, so far as I can tell, but there’s a big catch: some sets of coun­ter­fac­tual queries will re­quire keep­ing around far less in­for­ma­tion than oth­ers. A given ter­ri­tory sup­ports “nat­u­ral” classes of coun­ter­fac­tual queries, which re­quire rel­a­tively lit­tle in­for­ma­tion to yield ac­cu­rate pre­dic­tions to the whole query class. In this con­text, the lumped cir­cuit ab­strac­tion is one such ex­am­ple: we keep around just high-level sum­maries of the elec­tri­cal prop­er­ties of each com­po­nent, and we can an­swer a whole class of queries about voltage or cur­rent mea­sure­ments. Con­versely, if we had a few queries about the read­ings from a voltage probe, a few queries about the mass of var­i­ous cir­cuit com­po­nents, and a few queries about the num­ber of pro­tons in a wire mod 3… these all re­quire com­pletely differ­ent in­for­ma­tion to an­swer. It’s not a nat­u­ral class of queries.

So nat­u­ral classes of queries im­ply nat­u­ral ab­stract mod­els, pos­si­bly in­clud­ing nat­u­ral causal mod­els. There will still be some choice in which queries we care about, and what in­for­ma­tion is ac­tu­ally available will play a role in that choice (i.e. even if we cared about num­ber of pro­tons mod 3, we have no way to get that in­for­ma­tion).

I have not yet for­mu­lated all this enough to be highly con­fi­dent, but I think in this case the voltage → cur­rent model is a nat­u­ral ab­strac­tion when we have a voltage sup­ply, and vice versa for a cur­rent sup­ply. The “cor­rect” model, in each case, can cor­rectly pre­dict be­hav­ior of the re­sis­tor and knob coun­ter­fac­tu­als (among oth­ers), with­out any ad­di­tional in­for­ma­tion. The “in­cor­rect” model can­not. (I could be miss­ing some other class of coun­ter­fac­tu­als which are eas­ily an­swered by the “in­cor­rect” mod­els with­out ad­di­tional in­for­ma­tion, which is the main rea­son I’m not en­tirely sure of the con­clu­sion.)

Thanks for bring­ing up this ques­tion and ex­am­ple, it’s been use­ful to talk through and I’ll likely re-use it later.

• For what it’s worth, my cur­rent think­ing on brain al­gorithms is that the brain has a cou­ple low-level prim­i­tives, like tem­po­ral se­quences and spa­tial re­la­tions, and these prim­i­tives are able to rep­re­sent (1) cause-effect, (2) hi­er­ar­chies, (3) com­po­si­tion, (4) analo­gies, (5) re­cur­sion, and on and on, by com­bin­ing these prim­i­tives in differ­ent ways and with differ­ent con­tex­tual “meta­data”. This is my opinion, it’s con­tro­ver­sial in the field of cog­ni­tive sci­ence and I could be wrong. But any­way, that makes me in­stinc­tively skep­ti­cal of world-mod­el­ing the­o­ries where ev­ery­thing re­volves around cause-effect, and equally skep­ti­cal of world-mod­el­ing the­o­ries where ev­ery­thing re­volves around hi­er­ar­chies, etc. etc. I would be much more ex­cited about world-mod­el­ing the­o­ries where all those 5 differ­ent types of re­la­tion­ships (and oth­ers I omit­ted, and shades of gray in be­tween them) are all equally rep­re­sentable.

(This is just an in­stinc­tive re­sponse /​ hot-take. I don’t have a par­tic­u­larly strong opinion that the re­search di­rec­tion you’re de­scribing here is un­promis­ing.)

• I gen­er­ally agree with this think­ing, al­though I’ll high­light that the brain and a hy­po­thet­i­cal AI might not use the same prim­i­tives—they’re on very differ­ent hard­ware, af­ter all. Cer­tainly the gen­eral strat­egy of “start with a few prim­i­tives, and see if they can rep­re­sent all these other things” is the sort of strat­egy I’m af­ter. I cur­rently con­sider causal DAGs with sym­me­try the most promis­ing prim­i­tive. It di­rectly han­dles causal­ity and re­cur­sion, and alongside a suit­able the­ory of ab­strac­tion, I ex­pect it will al­low us to rep­re­sent things like spa­tial/​tem­po­ral re­la­tions, hi­er­ar­chies, analo­gies, com­po­si­tion, and many oth­ers, all in a unified frame­work.

• On fur­ther re­flec­tion, the prim­i­tive of “tem­po­ral se­quences” (more speci­fi­cally high-or­der Markov chains) isn’t that differ­ent from cause-effect. High-or­der Markov chains are like “if A hap­pens and then B and then C, prob­a­bly D will hap­pen next”. So if A and B and C are a per­son mov­ing to kick a ball, and D is the ball fly­ing up in the air...well I guess that’s at least part­way to rep­re­sent­ing cause-effect...

(High-or­der Markov chains are more gen­eral than cause-effect be­cause they can also rep­re­sent non-causal things like the lyrics of a song. But in the op­po­site di­rec­tion, I’m hav­ing trou­ble think­ing of a cause-effect re­la­tion that can not be rep­re­sented as a high-or­der Markov chain, at least at some ap­pro­pri­ate level of ab­strac­tion, and per­haps with some con­text-de­pen­dence of the tran­si­tions.)

I have pretty high con­fi­dence that high-or­der Markov chains are one of the low-level prim­i­tives of the brain, based on both plau­si­ble neu­ral mechanisms and com­mon sense (e.g. it’s hard to say the let­ters of the alpha­bet in re­verse or­der). I’m less con­fi­dent about what ex­actly are the el­e­ments of those Markov chains, and what are the other low-level prim­i­tives, and what’s ev­ery­thing else that’s go­ing on. :-)

Just think­ing out loud :-)

• I’ll high­light that the brain and a hy­po­thet­i­cal AI might not use the same prim­i­tives—they’re on very differ­ent hard­ware, af­ter all

Sure. There are a lot of lev­els at which al­gorithms can differ.

• quick­sort.c com­piled by clang ver­sus quick­sort.c com­piled by gcc

• quick­sort op­ti­mized to run on a CPU vs quick­sort op­ti­mized to run on an FPGA

• quick­sort run­ning on a CPU vs merge­sort run­ning on an FPGA

• quick­sort vs a differ­ent al­gorithm that doesn’t in­volve sort­ing the list at all

There are peo­ple work­ing on neu­ro­mor­phic hard­ware, but I don’t put much stock in any­thing com­ing of it in terms of AGI (the main thrust of that field is low-power sen­sors). So I gen­er­ally think it’s very im­prob­a­ble that we would copy brain al­gorithms at the level of firing pat­terns and synapses (like the first bul­let-point or less). I put much more weight on the pos­si­bil­ity of “copy­ing” brain al­gorithms at like vaguely the sec­ond or third bul­let-point level. But, of course, it’s also en­tirely pos­si­ble for an AGI to be rad­i­cally differ­ent from brain al­gorithms in ev­ery way. :-)

• My guess here is that there are some in­stru­men­tally con­ver­gent ab­strac­tions/​al­gorithms which both a brain and a hy­po­thet­i­cal AGI needs to use. But a brain will have im­ple­mented some of those as hacks on top of meth­ods which evolved ear­lier, whereas an AI could im­ple­ment those meth­ods di­rectly. So for in­stance, one could imag­ine the brain im­ple­ment­ing sim­ple causal rea­son­ing as a hack on top of pre-ex­ist­ing tem­po­ral se­quence ca­pa­bil­ities. When de­sign­ing an AI, it would prob­a­bly make more sense to use causal DAGs as the fun­da­men­tal, and then im­ple­ment tem­po­ral se­quences as ab­stract stick-dags which don’t sup­port many (if any) coun­ter­fac­tu­als.

Pos­si­bly bet­ter ex­am­ple: tree search and logic. Hu­mans seem to han­dle these mostly as hacks on top of pat­tern-match­ers and trig­ger-ac­tion pairs, but for an AI it makes more sense to im­ple­ment tree search as a fun­da­men­tal.

• Sounds ex­cit­ing, and I wish you luck and look for­ward to read­ing what­ever you come up with! :-)

• I re­ally like this model of com­pu­ta­tion and how nat­u­rally it deals with coun­ter­fac­tu­als, sur­prised it isn’t talked about more of­ten.

This raises the is­sue of ab­strac­tion—the core prob­lem of em­bed­ded agency.

I’d like to un­der­stand this claim bet­ter—are you say­ing that the core prob­lem of em­bed­ded agency is re­lat­ing high-level agent mod­els (rep­re­sented as causal di­a­grams) to low-level physics mod­els (also rep­re­sented as causal di­a­grams)?

• I’m say­ing that the core prob­lem of em­bed­ded agency is re­lat­ing a high-level, ab­stract map to the low-level ter­ri­tory it rep­re­sents. How can we char­ac­ter­ize the map-ter­ri­tory re­la­tion­ship based on our knowl­edge of the map-gen­er­at­ing pro­cess, and what prop­er­ties does the map-gen­er­at­ing pro­cess need to have in or­der to pro­duce “ac­cu­rate” & use­ful maps? How do queries on the map cor­re­spond to queries on the ter­ri­tory? Ex­actly what in­for­ma­tion is kept and what in­for­ma­tion is thrown out when the map is smaller than the ter­ri­tory? Good an­swers to these ques­tion would likely solve the usual re­flec­tion/​di­ag­o­nal­iza­tion prob­lems, and also ex­plain when and why world-mod­els are needed for effec­tive goal-seek­ing be­hav­ior.

When I think about how to for­mal­ize these sort of ques­tions in a way use­ful for em­bed­ded agency, the min­i­mum re­quire­ments are some­thing like:

• need to rep­re­sent physics of the un­der­ly­ing world

• need to rep­re­sent the cause-and-effect pro­cess which gen­er­ates a map from a territory

• need to run coun­ter­fac­tual queries on the map (e.g. in or­der to do plan­ning)

• need to rep­re­sent suffi­ciently gen­eral com­pu­ta­tions to make agenty things possible

… and causal di­a­grams with sym­me­try seem like the nat­u­ral class to cap­ture all that.

• Should there be an ar­row go­ing from n*f(n-1) to f (around n==0?) ? The out­put of the sys­tem also de­pends on n*f(n-1), not just on whether or not n is zero.

• The “n==0?” node is in­tended to be a ternary op­er­a­tor; its out­put is n*f(n-1) in the case where n is not 0 (and when n is 0, its out­put is hard­coded to 1).

• the DAG con­sists of an in­finite se­quence of copies of the same sub­cir­cuit.

If it’s in­finite then the pro­gram doesn’t ter­mi­nate. (In the­ory this may be the case if n is a non-nega­tive in­te­ger.)

Ideally, we’d gain enough in­for­ma­tion to avoid need­ing to search over all pos­si­ble pro­grams.

Over all shorter pos­si­ble pro­grams.

(In prin­ci­ple, we could shoe­horn this whole thing into tra­di­tional Solomonoff In­duc­tion by treat­ing in­for­ma­tion about the in­ter­nal DAG struc­ture as nor­mal old data, but that doesn’t give us a good way to ex­tract the learned DAG struc­ture.)

Sounds like SI is be­ing treated as a black box.

We don’t care about graphs which can be gen­er­ated by a short pro­gram but don’t have these sorts of sym­me­tries.

Why do we like pro­grams which use re­cur­sion in­stead of iter­a­tion?

• The non-ter­mi­na­tion point is a bit sub­tle. If you look at the ex­am­ple in the OP, the causal di­a­gram it­self is in­finite, but as long as we pass in a pos­i­tive in­te­ger the out­put value won’t ac­tu­ally de­pend on the whole cir­cuit. One of the con­di­tional nodes will rec­og­nize the base case, and ig­nore what­ever’s go­ing on in the rest of the cir­cuit be­low it. So the pro­gram can ter­mi­nate even if the DAG is in­finite. (Equiv­a­lent con­cep­tu­al­iza­tion: imag­ine run­ning a lazy eval­u­a­tor on the in­finite DAG.)

That said, DAGs with sym­me­try cer­tainly can rep­re­sent com­pu­ta­tions which do not halt. This is a fea­ture, not a bug: there are in fact com­pu­ta­tions which do not halt, and we can ob­serve their phys­i­cal be­hav­ior mid-com­pu­ta­tion. If we want to think about e.g. what my pro­ces­sor is do­ing when run­ning an in­finite loop, then we need a com­pu­ta­tional model which can rep­re­sent that sort of thing, rather than just talk­ing about in­put/​out­put be­hav­ior.