Wildfire of strategicness

It may not be feasible to make a mind that makes achievable many difficult goals in diverse domains, without the mind also itself having large and increasing effects on the world. That is, it may not be feasible to make a system that strongly possibilizes without strongly actualizing.

But suppose that this is feasible, and there is a mind M that strongly possibilizes without strongly actualizing. What happens if some mental elements of M start to act strategically, selecting, across any available domain, actions predicted to push the long-term future toward some specific outcome?

The growth of M is like a forest or prairie that accumulates dry grass and trees over time. At some point a spark ignites a wildfire that consumes all the accumulated matter.

The spark of strategicness, if such a thing is possible, recruits the surrounding mental elements. Those surrounding mental elements, by hypothesis, make goals achievable. That means the wildfire can recruit these surrounding elements toward the wildfire’s ultimate ends. By recruiting more dry matter to the wildfire, the wildfire burns hotter and spreads further.

Also by hypothesis, the surrounding mental elements don’t themselves push strongly for goals. Seemingly, that implies that they do not resist the wildfire, since resisting would constitute a strong push. We can at least say that, if the totality of the mental elements surrounding the wildfire is going to notice and suppress the wildfire, it would have to think at least strategically enough to notice and close off all the sneaky ways by which the wildfire might wax. This implies that the surrounding mental elements do a lot of thinking and have a lot of understanding relevant to strategic takeovers, which itself seemingly makes more available the knowledge needed for strategic takeovers.

Capabilities overhang provides fuel for a wildfire of strategicness. That implies that it’s not so easy to avoid wrapper-minds.

This is very far from being a watertight argument, and it would be nice if the conclusion were false; how is it false? Maybe, as is sometimes suggested, minds selected to be creative are selected to keep themselves open to new ideas and therefore to new agency, implying that they’re selected to prevent strategic takeovers, which would abhor new agency?