Boundaries enable positive material-informational feedback loops
(Cross-posted from my blog)
[epistemic status: obvious once considered, I think]
If you want to get big things done, you almost certainly need positive feedback loops. Unless you can already do all the necessary things, you need to do/make things that allow you to do/make more things in the future. This dynamic can be found in RPG and economy-management games, and in some actual economic systems, such as industrializing economies.
Material, information, and economy
Some goods that can be used in a positive feedback loop, such as software and inventions, are informational. Once produced, they can be used indefinitely in the future. In economic terms, they are nonrivalous.
Other goods are material, such as manufactured goods and energy. They can’t be copied cheaply. In economic terms, they are rivalrous.
In practice, any long-lasting positive feedback loop contains both informational and material goods, as production of information requires a physical substrate. While ensuring that informational goods can be used in the future is an organization and communication problem (a subject beyond the scope of this post), the problem of ensuring that material goods can be used in the future is additionally a security problem.
An important question to ask is: why haven’t material-informational positive feedback loops already taken over the world? Why don’t we have so much stuff by now that providing for people’s material needs (such as food and housing) is trivial?
To some extent, material-informational positive feedback loops have taken over the world, but they seem much slower than one would naively expect. See cost disease. As an example of cost disease, the average cost of a new house in the USA has quadrupled over a 60-year period (adjusted for inflation!), whereas models of capitalism based on economy-management games such as Factorio (or, more academically, according to the labor/capital based economic models of classical economists such as David Ricardo) would suggest that houses would be plentiful by now. (And no, this isn’t just because of land prices; it costs about $300K to build a house in the US in 2018)
Security and boundaries
I’ve already kind of answered this question by saying that ensuring that material goods can be used in the future is a security problem. If you use one of your material goods to produce another material good, and someone takes this new good, then you can’t put this good back into your production process. Thus, what would have been a positive feedback loop is instead a negative feedback loop, as it leaks goods faster than it produces them.
Solving security issues generally requires boundaries. You need to draw a boundary in material space somewhere, differentiating the inside from the outside, such that material goods (such as energy) on the inside don’t leak out, and can potentially have positive feedback loops. There are many ways to prevent leaks across a boundary while still allowing informational and material to pass through sometimes, such as semiporous physical barriers and active policing. Regardless of the method to enforce the boundary, the boundary has to exist in some geometrical sense for it to make sense to say that e.g. energy increases within this system.
Not all security issues are from other agents; some are from non-agentic processes. Consider a homeostatic animal. If the animal expends energy to warm its body, and this warmth escapes, the animal will fail to realize gains from the energy expenditure. Thus, the animal has a boundary (namely, skin) to solve this “security problem”. The cold air particles that take away heat from the animal are analogous to agents that directly take resources, though obviously less agentic. While perhaps my usage of the word “security” to include responses to nonagentic threats is nonstandard, I hope it is clear that these are on the same spectrum as agentic threats, and can be dealt with in some of the same ways.
It is also worth thinking about semi-agentic entities, such as microorganisms. One of the biggest threats to a food store is microorganisms (i.e. rotting), and slowing the negative feedback loops depleting food stores requires solving this security problem using a boundary (such as a sealed container or a subset of the air that is colder than the outside air, such as in a refrigerator).
Property rights are a simple example of boundaries. Certain goods are considered to be “owned” by different parties, such that there is common agreement about who owns what, and people are for one reason or another not motivated to take other people’s stuff. Such division of goods into sets owned by different parties is a set of boundaries enabling positive feedback loops, which are especially salient in capitalism.
What about trust between different entities? A complex ecosystem will contain entities satisfying a variety of niches, which include parasitism and predation (which are on the same spectrum). A trust network can be thought of as a way for different entities to draw various boundaries, often fuzzy ones, that mostly exclude parasites/predators, such that there are few leaks from inside this boundary to outside this boundary (which would include parasitism/predation by entities outside the boundary). There are “those who you trust” and “those who you don’t trust” (both fuzzy sets), and you assign more utility to giving resources to those you trust, as this allows for positive feedback loops within a system that contains you (namely, the trust network).
Externalities and sustainability
Since no subsystem of the world is causally closed, all positive feedback loops have externalities. By definition, the outside world is only directly affected by these externalities, and is only affected by what happens within the boundary to the extent that this eventually leads to externalities. A wise designer of a positive feedback loop will anticipate its externalities, and set it up such that the externalities are overall desirable to the designer. After all, there is no point to creating a positive feedback loop unless its externalities are mostly positive [EDIT: unless the boundary contains things that have intrinsic value].
A positive feedback loop’s externalities modify its environment, affecting its own ability to continue; for example, a positive feedback loop of microorganisms eating food will exhaust itself by consuming the food. So, different positive feedback loops are environmentally sustainable to different extents. Both production and conquest generate positive feedback loops, as Ben Hoffman discusses in this post, but production is much more environmentally sustainable than conquest.
One way to increase environmental sustainability is to move more processes to the inside of the boundary. For example, a country that is consuming large amounts of iron (driving up iron prices) may consider setting up its own iron mines. Thus, the inside of the boundary becomes more like an economy of its own. This is sometimes known as import replacement.
Of course, the environmental sustainability of a positive feedback loop can also be a negative, as it is better for some processes (such as rotting) to limit or exhaust themselves, thus transitioning to negative feedback or a combination of positive and negative feedback. Processes that include intentionally-designed positive and negative feedback can be much more environmentally sustainable than processes that only have positive feedback loops designed in, since they can limit their growth when such growth would be unsustainable.
While in theory the philosophy of effective altruism would imply a strong (and likely overwhelming) emphasis on creating and maintaining environmentally sustainable positive feedback loops with positive externalities, typically-recommended EA practices (such as giving away 10% of one’s income) are negative feedback loops (the more you make, the more you give away). While in theory the place the resources are given to could have a faster positive feedback loop than just investing in yourself, your friends, and your projects, in practice I rarely believe claims of this form that come from the EA movement; for example, if a country has a high rate of poverty, that indicates that the negative feedback loops (such as corruption) are likely stronger than the positive ones, and that giving resources is ineffective. Thus, I cannot in good conscience allow anything like current EA ideology to substantially control resource allocation in most systems I create, even though EA philosophy taken to its logical conclusion would get the right answer on the importance of boundaries and positive feedback loops.
Policy suggestions
How do these ideas translate to action? One suggestion is that, if you are trying to do something big, you use one or more positive feedback loops, and ask yourself the following questions about each one:
What’s the generator of my positive feedback loop (i.e. what’s the process that turns stuff into more stuff)?
What is the boundary within which the positive feedback increases resources?
How am I reducing leakage across this boundary?
What are the externalities of this positive feedback loop?
How environmentally sustainable is this positive feedback loop?
Are there built-in negative feedback loops that increase environmental sustainability?
(thanks to Bryce Hidysmith for a conversation that led to this post)
That might be true, and is one of the possible stories that I’m keeping track of. But another story is one of mostly positive feedback loops that have barely got of the ground. In that case the resources infusion does net good, allocating resources to places where the marginal payoff per unit is higher (because those places have not hit the shoulder of the S-curve yet.)
I think this makes sense as a critique of certain parts of EA, but I’m not sure how to apply the idea more generally. For example, a lot of valuable things people do consist of just offering goods and services on the global market, for which I guess “the boundary within which the positive feedback increases resources” would just be Earth or the global economy. If they do this as part of a company or country, I guess those would be additional boundaries, but how does it help for an individual to think about these boundaries more explicitly?
To the extent that it does make sense for someone to study the internal and external dynamics of such boundaries, I note that the field of economics has studied a lot of related issues under a somewhat different framing (see for example property rights, trade, division of labor, comparative advantage, finance, theory of the firm, and developmental economics), most of which are not mentioned by this post.
Maybe the author is thinking of some specific categories of “doing big things” for which the current framing makes more sense than a standard economics one, in which case I think it would be helpful to say more about what these categories are, and give some examples of such projects and how the author would answer the proposed questions for them.
Markets are examples of this phenomenon (both individuals interacting with a market and getting returns to capital, and the market itself growing), but they are pretty limited when dealing with goods that are hard to evaluate (see: legibility, market for lemons). So a more general theory is needed.
It’s possible that these ideas are obvious to some of the best economists, but this post contains content that I didn’t find in my Econ 101 class. For example, Econ 101 would not have led me to believe that cost disease was a thing, or that it was caused by parasitic/predatory processes. I would be curious to hear if there are good economics papers on this (the closest thing I’ve heard about is guard labor).
In any case I intend these ideas to have broader applicability than markets. Here are some examples:
growth of populations in an ecosystem
growth of a single organism
resource accumulation in particular households
career capital
growth of a new social/political movement
spread of better safety ideas and practices in the AGI field
growth of a new research lab
etc etc. I’m going to answer the questions for the last one (since this one is one of my major plans):
Some generators are (researchers, material goods) → ideas. Buying stuff is money → material goods. Writing about ideas is (ideas, researchers) → (clearer ideas, prestige). Spinoff businesses from the research lab would be (ideas, researchers) → (researchers, material goods, money, wisdom). Giving grants to other individuals (and possibly recruiting those individuals) is (money, prestige) → (ideas, researchers). And so on. You can compose these generators together to turn stuff into more stuff.
The boundary includes the research organization, and to some extent its spinoff businesses.
Retention of researchers (by making the research lab a nice place to work), property rights, reputation.
Public ideas, externalities of spinoff businesses, effects of non-research needs of the researchers and the institution (e.g. community), taking researchers from wherever they would have been otherwise, effects on researchers who leave.
Main sustainability issues are: picking low hanging fruit (can’t generate more ideas), recruiting all the best researchers and running out.
It’s possible to reduce recruiting if low hanging fruit is already picked and/or there aren’t as many good researchers left. Import replacement is possible by training people into better researchers (e.g. by starting or working with a university).
Some of this could be analyzed in an economics framework (e.g. in theory of how firms work); I haven’t seen enough of economic theories of this sort of thing to comment on how the different frameworks would compare on metrics such as clarity and emphasis on the important questions to ask.
But markets for lemons (and asymmetric information in general) are well studied in economics. To give a sense of how economists approach these types of problems, I quote from the Wikipedia article on theory of the firm:
Baumol’s (who introduced the term) own explanation of cost disease seems pretty good (and does not involve parasitic/predatory processes). (BTW he wrote a whole book on the cost disease which seems to focus mostly on health care.) To quote Wikipedia:
See this article and paper which apply Baumol’s idea to the housing sector. I note that home quality and size have increased quite a lot in 60 years which must also be part of the explanation for the price rise.
If these don’t seem like sufficient explanation to you, rent seeking and public choice theory in general serve as additional explanations and are studied under economics (and are perhaps closer to your “parasitic/predatory processes”).
A lot of the answers don’t seem particularly exciting (e.g., make the research lab a nice place to work, reduce recruiting if low hanging fruit is already picked). What do you see as the major new insights of your framework (i.e., beyond what common sense, economic theory, and standard practice would tell you), when applied to running a research lab?
In general: as I said at the top of the post, this is mostly pretty obvious, so I don’t think most of the content itself will be anything new to the better economists, though perhaps the framing, or the derivation from materialism and control theory rather than rational agency, would be somewhat novel (I don’t actually know). I think a lot of people already have intuitions about this, but spelling them out explicitly allows for more forms of processing to happen on the model, such as formal analysis, and being able to refer to it in conversation. Additionally, a model that can make sense of a variety of phenomena (e.g. biological, economic, and social) has utility for allowing intuitions to be transferred between domains. In any case, I found it useful to write this for my own understanding.
I have a pretty low opinion of the average work in economics, although I think the best work is pretty good. Economics seems pretty ideological, see this comment thread. Therefore, even if I get models from the econ literature, I additionally need to do first-principles derivations of econ concepts (such as the derivation in this post) in order to know which things to pay attention to and to trust.
Moreover, for an idea this obvious, it seems like lots of people (e.g. EAs) don’t take it into account. So perhaps it isn’t actually obvious to most people, or perhaps it is and people are being ideological, in which case well-argued and concise counter-ideology is useful.
Some more specific responses:
What I meant was that a more general theory than simply a theory of markets is needed. And indeed, firms aren’t markets, so economic theories of firms+markets are more general than theories of markets. Additionally, economics doesn’t normally study all the examples I listed (though the field is big enough that maybe you could find analyses of most of these if you looked hard enough), while they are obvious applications of the model in this post.
It really doesn’t explain housing; labor costs for construction work grow at about the rate of inflation, not the rate of increase in house prices (source: ENR construction cost index; unfortunately you have to pay to get the data, I can show a copy to you if you want. Main upshot: a bundle of materials and labor used for construction increased from $375 to $3539 from 1950 to 2000, while home prices increased from $2939 to $199600, where neither of these are inflation adjusted. It makes sense that labor/materials would increase at about inflation rate if inflation is measured using CPI, and indeed CPI increased by about 8x in this time period). I haven’t looked at the paper in detail but it looks like it’s basing things on the ratio of productivity in different sectors, so it seems their model would predict it’s explained by a rise in labor price, which is empirically false.
When the research lab plan is spelled out, indeed it looks pretty obvious, but not everyone is going to (a) use the search strategy of looking for environmentally sustainable positive feedback loops with positive externalities, and (b) actually go through all the thinking steps suggested by the checklist (See: The Checklist Manifesto). Furthermore, going through the thinking steps on this plan makes it clearer that this has a decent chance of being a good idea, and suggests further questions to ask to determine viability (e.g. thinking about what the rate of spinoff businesses is, and how profitable they will be). “Standard practice” will give a bunch of advice that is good as well as a bunch of advice that is bad; models are necessary to know which is which, and to get better outcomes than others who are following standard practice.
I worry that the obviousness is misleading, because economies actually are made of more or less rational agents, and as a result have a lot of anti-inductive elements, which are not captured by a control theory based derivation. For example, it fails to suggest that if it’s possible to make abnormal returns by operating a research lab, people would be doing that already and thereby drive the risk-adjusted returns down to a level that’s commensurate with other investments, or that if you manage to figure out how to better operate a research lab, other research labs can copy that and thereby reduce your returns down to a normal level, or new research labs will start up and compete with you.
In that case it’s probably explained by preferences for greater quality and size in housing (explained by the income effect and housing being in part a positional good), plus increased regulation (studied under public choice theory).
This is a really good point, thanks. This points at some areas of my strategy that aren’t explained in this post (these areas contain things like asymmetric weapons and using decision theories that pass the mirror test).
The first explanation would imply that people are willing to pay 6x as much for a 2000-style house than a 1950-style house (ignoring factors like 1950-style houses being scarcer now than they were in 1950), which seems false. A public choice theory framework for regulation assumes that these regulations are generally in people’s interest, whereas they often aren’t (people aren’t very informed about what housing regulations are good, and regulatory capture is a thing); indeed, if people wouldn’t be willing to pay 6x as much for a 2000-style house than a 1950-style house after knowing about these additional regulations, then that indicates that these regulations aren’t in their interest. Perhaps there are one or more good explanations within the field of economics for this phenomenon, but it does not seem like the search strategy you are using to produce economics explanations for this phenomenon is getting good explanations at a very high rate, which indicates that the field of economics is drawing little attention to good explanations for this, or is drawing attention away from the good explanations.
I only intend it to be part of the explanation, and surviving 1950-style houses are probably bigger and higher quality than the average 1950 house (which is what the cost comparison was based on).
I don’t think that’s true. Rent seeking and regulatory capture are major themes in public choice theory. (Wait, by “in people’s interest” do you mean “in someone’s interest” or “in the interest of the general public”? I think public choice theory definitely doesn’t assume the latter, and even the former is debatable in that it predicts a lot of waste which isn’t in anyone’s interest.)
What is the explanation that your framework generates? I don’t think it was spelled out in the post, or maybe I missed it?
Oh, I got public choice theory confused with social choice theory, you’re right.
So, I’m still confused about this (I plan to investigate this more), but the framework of this post would posit some combination of: processes that produce houses are hard to imitate and have gotten worse over time as knowledge is lost; regulatory capture; coordination being harder as better attacks on existing coordination systems have been developed (e.g. more ways of pretending to work, more bullshit jobs that are considered part of a normal business); some kind of coordination among house-builders. Since there are lots of possible explanations, this isn’t very enlightening on its own, and more investigation is needed.
Upon writing this, it seems like my framework isn’t clearly better at explaining the phenomenon than the field of economics (they both posit that there could be many causes), until further investigation has been done; however, that isn’t the assertion I was originally making, and also it’s kind of a moot point since I previously thought you were arguing that the content of this post was obvious, and responding to that.
(A meta comment: I feel like some of your comments could be summarized as “this is already obvious to people who study economics” and that your tone conveys that you mean this as some kind of objection. I think this post is quite relevant to discourse norms around saying obvious things.)
I don’t think obviousness was my point. Sorry if I gave that impression. What I wanted to convey was that economists having been studying these issues for decades and have made a lot of progress, so it probably doesn’t make sense to start over with a new framework, unless you could explain why the standard economics ones are problematic and how the new framework fixes those problems. In particular, I think the ideas of comparative advantage and asymmetric information are really powerful but do not seem to have analogues in your framework (or at least they haven’t been spelled out yet).
This points at a thing that’s been bothering me for a while in the discourse around economics.
Often there are specific concepts or models worth learning from a discipline. For instance, comparative advantage and asymmetric information are important, but not all that complicated and hard to understand, and once you understand them you can apply them in new circumstances without holding onto the formalism that professionals fit them into.
In much rarer cases, an entire disciplinary framework is worth preserving. Physics in the Newtonian paradigm (by which I mean to include most of 20th Century physics as well) is an example—you can’t really understand “momentum” or “energy” except in relation to a much larger mathematized framework, which has a huge amount of descriptive, explanatory, and predictive power.
My sense of economics is that many concepts are important, but the frameworks are systematically obscuring a lot of what’s important to see. A lot of what I tried to do with Talents and There is a war was explain a bunch of stuff I’d learned from reading widely about economics, in nonideological terms that would help the reader apply patterns economists know about to concrete situations without providing anything pretending to be a comprehensive framework, and without claiming the mantle of what I consider spurious intellectual authority.
Most comments I see of the type “you should engage more with economics” don’t really seem to be thinking about costs clearly, which does not exactly reflect well on the discipline. In particular, it doesn’t seem like “economists have thought about this stuff a lot and made some progress” tells us much about whether the framework of economics is worth the time learning. The Roman Catholic Church has thought about moral and political philosophy a bunch, but while I might recommend someone read Augustine or some other Catholic writer to learn some particular insights or models that I think are relevant to their interests, I wouldn’t think of suggesting they engage with the Catholic framework rather than starting over. The same with academic research psychology. Why should I think better of academic economics?
The burden of evidence is a lot lower for recommendations like “this specific book or paper by an economist contains models that I expect would make this substantially clearer or more complete” or “this subfield of economics has standard arguments against your position that I’d need a clear response to in order to accept your model.”
There’s another reason to engage with a field, which is to contribute to the field—i.e. they might want to learn what someone has to say, but be unable to engage with it if it doesn’t speak their language. But almost no one seems to be saying this.
Can you expand on this? How are the frameworks systematically obscuring a lot of what’s important to see?
Economics doesn’t seem very ideological to me, or maybe I’m just too brainwashed to see it. Can you explain more?
Ah, yeah I wasn’t thinking in terms of “costs of learning economics”, because learning it was a lot of fun for me. Kind of curious why people who otherwise seem very similar to myself (such as Jessica) don’t find it fun.
Economics (as a real world phenomena) is really complex and often counter-intuitive. It’s very easy to make mistakes when thinking about it and academia has already made and fixed a lot of those mistakes. Studying academic economics gave me intuitions and formal tools that make it much easier to see such mistakes in myself and others. See this recent thread as an example.
It’s true though that I haven’t been thinking that others might find it very costly to learn, in which case learning some detached concepts and simplified models might be better than nothing, unless it makes one feel overconfident instead of just confused. So I’d prefer to see some disclaimers on such posts like “Please try to learn academic economics, but if that’s not fun for you or you don’t have enough time, these simplified concepts/models/frameworks might help. Keep in mind that real world economics is really complex and often counterintuitive, and these concepts/models/frameworks haven’t been as widely vetted or extensively tested as concepts/models/frameworks in academic economics.”
Are there any economics textbooks you’d recommend?
Sorry, of the economics textbooks I’ve read, I’ve either forgotten which ones I used, or they haven’t been updated since the 90s. And a lot of what I learned were from things like lectures, articles, and papers, instead of textbooks. I can only suggest some subfields in economics that seem especially interesting to me, and recommend that you start directly with graduate level textbooks (or review papers if textbooks don’t exist yet), because the undergraduate ones tend to oversimplify a lot of things.
game theory (both cooperative and non-cooperative, ETA: but see this comment for why cooperative game theory isn’t very widely known)
industrial organization
public choice theory
economics of property rights
theory of the firm
family economics
information economics
market/mechanism design
finance
microeconomics (for general concepts like comparative advantage, transaction costs, and deadweight loss)
macroeconomics
micro foundations of macroeconomics
monetary theory
This is helpful, thanks!
Ok, this is much easier to respond to, thanks.
I agree with Ben here and won’t repeat what he said.
In general, I think thinking things through yourself from first principles is a valuable exercise even if someone has already thought the thing through, so even if the economics field functioned better than it did, I would still not agree that “it probably doesn’t make sense to start over with a new framework, unless you could explain why the standard economics ones are problematic and how the new framework fixes those problems.”
Re comparative advantage: this is pretty easy to talk about in my framework; you can have different generators (with different conversion ratios and rates of conversion) in different locations, and produce faster feedback loops by using the more-efficient generators, which requires moving stuff around.
Re asymmetric information: also pretty easy, this is information produced in one location that is not communicated to other locations.
Given that the “first principles” must by necessity be much simplified compared to the real world, how do you know whether the derivations come anywhere close to explaining it? Academia has done a lot of the necessary vetting/testing for its frameworks so I don’t think it’s a good idea to start over with different principles (unless, again, you can explain why they’re likely to solve some problems in the standard ones).
Can your framework derive the well known consequences of asymmetric information in economics, such as it often leading to failure to agree upon mutually beneficial deals?
Academia’s models are also simplified. Given things like the replication crisis, I am really not convinced that academia is good at vetting things outside STEM. Generation of known-good ideas can be separated into generating ideas and checking them; generating ideas can be useful even they can’t be fully checked yet. In practice, to vet my models, I look at things like: which conclusions are logically sound (e.g. that growing economies require positive material feedback loops), whether it matches up with information I have, compatibility with other models/intuitions (where incompatibility could mean either model is wrong), and so on. I don’t think this is that different from what the most generative academics do. If I were reading academic papers, I would be doing the same checks to determine what to trust and how to integrate the ideas into my own models.
Kind of. I haven’t discussed rational agents yet. But, it’s possible to say that some systems will be more ecologically fit if they hide certain information, and some will be more ecologically fit if they only make trades given the info that making this trade would gain some necessary resource, from which it is derived that ecologically fit systems will fail to make trades that increase the fitness of both of them, where ecological fitness could be defined in terms of speed and sustainability of positive feedback loop. There are different advantages/disadvantages to thinking about things this way instead of in terms of rational agency.
I’m confused by this paragraph; the standard economics definition for externality is “the cost or benefit that affects a party who did not choose to incur that cost or benefit” (Wikipedia); so I’m interpreting externality here to mean effects that leak out through the boundary to the outside world. But why then “there is no point to creating a positive feedback loop unless its externalities are mostly positive”? You don’t need the effects of the feedback loop that leak out of the boundary to be overall positive; it’s enough for the effects that remain within the boundary (which are the ones that you capture) to be overall positive. After all the standard economics definition for “externality” is something that isn’t directly relevant for the agents causing the externality, but you seem to be using the term to refer to something that the agents constructing the feedback loop do need to take into account.
This is not the definition I’m using.
This is the definition I’m using.
The things that happen within the boundary, other than the externalities, are epiphenomenal. They could still be important because e.g. they contain moral patients, and I guess my statement was wrong because of this fact. So I edited the relevant sentence.
Got it. The way you used it in a sentence without specifically defining it first, and after having referenced economics earlier, made it unclear whether it was supposed to be understood in the economics sense or not.
If EA focused more on feedback loops, then there’d be less focus on donating money to charity. How would you like these resources to be deployed instead?
At a very high level, a post-EA strategy might look something like this:
Invest in yourself, your friends, and your projects.
Become less confused about how the world works and what strategies are effective for doing things in it. Reading from a variety of fields (math, history, decision theory, economics, etc) is a good way to start, as is doing first-principles analyses and writing about them, and visiting places you think might be important to investigate.
Find friends you can talk to about this, and who you can coordinate with.
Make one or more plans for causing nice things to happen, which will probably involve environmentally sustainable positive feedback loops with positive externalities.
Execute on these plans until they becomes obsolete.
Not everyone has to do all these things, some can support people doing this, or be one member of a group that does this. Also, some people don’t care enough to actually do these things, and probably these people shouldn’t do strategy, and should instead do things they actually care about.
I basically agree with this approach. I have sometimes said that if I could change one thing about EA, it would be that I want more EAs to feel like there job is to understand the world and how the world works (rationalists are overall better on this dimension, though they have other problems.)
[Note: I’m currently training a practice of noticing when what I say, or what others say, aligns with our personal [social] incentives. The statement above aligns with my incentives in so far as I like figuring things out, and apparently can do it. So if the statement above were true, that would imply that I am doing the “right thing” more than others who are doing other work.]
I’m curious to hear more detail about what you imagine for step 4. What sort of “nice things” do you have in mind? What kind of plans?
Thanks for writing this! This is very closely related to Andrew Critch’s «Boundaries» Sequence, 2022. Part 3a formalizes boundaries in terms of Markov blankets, and leakage in terms of conditional mutual information.
I’ve also expanded on such leakage (“infiltration and exfiltration”) in my post my conceptualizations of infiltration and exfiltration from the «Boundaries» Sequence