It seems possible that one could invent a measure of “control power”
I think the likelihood of this being helpful is small, but I know of two sort-of-adjacent efforts. Both of which took place under the auspices of DARPA’s META Program, a program for improving systems engineering.
The first is a complexity metric, which they define as unexpected behavior of any kind and attempt to quantify in terms of information entropy. The part about the development of the metric begins on page 4.
The second is an adaptability metric. This one is considerably fussier; they eventually had to produce several metrics because of tradeoffs, and then tried to produce a valuation method so you could compare the metrics properly. It relies on several specific techniques which I have no knowledge of, and is much more heavily anchored in current real applications, but the crux of the effort seems to align with the “choices don’t change later choices” section above.
This post feels to me like the same type of conversation that would have been helpful in the work of these two papers, so I mention them on the off-chance the relationship works both ways.
With strategic clarity we would know what to do. Specifically, we would know…
- who the relevant actors are
- what actions are available to use
- how the future might develop from those actions
- what good sequences of actions (plans) are
- how to best prioritize plans
- that we have not missed any important considerations
Out of curiosity, has your research so far uncovered any example domains which have strategic clarity? Or do you have an intuition for domains that do?
What do you expect the signal of successful private strategy research to be?
There don’t seem to be that many outliers around to me, which strongly suggests either the research isn’t being done or it is failing to yield results.
This might be more fundamental than I initially thought. If people have a preference for shared information, this provides an upward pressure on communication in general—there is now a reward for telling people things just because, and also a reward for listening to people tell you things. This is entirely separate from any instrumental value the shared information may have.
I suspect that a lot of traditional things do double-duty, being instrumentally valuable and encouraging mutual information. Material example: the decorations on otherwise mundane items, like weapons or cauldrons.
It occurs to me that any given practice will probably be selected for resilience more than it will be for efficiency—efficiency is a threshold that must be met, but beyond that maximizing the likelihood that the minimum will be met seems more likely to propagate.
If people have an intrinsic desire to share information, then it seems likely that a practice for which sharing information is more common is more likely to persist. Hence the prevalence of so many multi-step processes; every additional step is another opportunity to share information (teach, tell where resources are, etc).
Weather. In a nutshell, bad weather makes battles harder. This is because walking a long way while wet sucks, it damages supplies and equipment, it increases the likelihood of disease, and there are intermittent dangers like flooding that are hard to predict in unfamiliar territory. In general, people know how to manage these things where they live, so the worse the weather, the bigger an advantage for the defender (or at least whoever marched less).
I once did a thought experiment where I tried to figure out how divination practices might directly help decisions.
The Druids were legendarily learned. What information we have says they were responsible for maintaining the oral history of their people, and for management of sacrifices, and reading of omens and the weather. They were reputed to have advanced knowledge of plants and animals.
I wondered about divination before battle. Naturally, birds aren’t really random—I expect a lot of people have noticed things like how they suddenly go quiet when a wet gust of wind blows through immediately prior to a storm. I expect if I were a Druid, I would have spent a lot of time watching birds, and know more things like this.
As a keeper of the oral history, I’ll know the reported outcome of previous battles and some important details about them (the weather, say).
Things like how many warriors my tribe has I can see with my eyes, and whether the other guys have more or less can be had by scouting like usual.
There’s also the matter of appeasing the gods, and offering them sacrifice. Now there’s a story from Greek myth about how early on the gods were tricked into accepting the fatty, gristly parts of the animal as the best parts, on the grounds that the smoke from burning those was better able to reach Olympus and nourish them. This agrees with casual observation: when I ruin a steak on the grill it smokes a lot more than when I ruin chicken on the grill. Smoke is a pretty good indicator of things like wind direction and strength, and further when it rises it can do things like show you where the wind changes above your level (like in smokestacks where it suddenly gets sheared off at a certain height).
So bird behavior provides information about barometric pressure, and the smoke from a sacrifice provides information about the movement of air pretty high up, and the oral history provides a sort of prior for similar circumstances.
So, if I were a Druid and knew what Druids know, I could make better than average predictions about the outcome of a battle if I made a burnt offering and read omens from birds.
I’ve been thinking about this problem from the other direction lately, particularly regarding divination practices; namely, now that we are habituated to the idea that there is a rational explanation for everything, how can we expect rituals—even useful ones—to survive over time?
My naive answer is that we cannot, and everything will slowly fall away as it comes into focus and the lack of causal mechanism is revealed.
On the other hand, people love stage magic. Almost everyone knows it is a trick beforehand, yet we are entertained. Mostly the surprisal and the possibility of belief is sufficient, but it seems to me that people are often more entertained when they spot the trick for themselves. The only real letdown is when a trick is immediately explained and proves deceptively simple, in my view.
This makes me suspect it might be possible to either reclaim or design new rituals, which would require a balancing act between having the utility explanation available and keeping the explanation separate from the experience of the ritual.
Also, the image for the bottom link is busted.
From Introducing: Asabiyah, which itself summarizes one concept from Ibn Khaldun’s Muqaddimah:
Thus Khaldun notes:
“The consequences of common descent, though natural, still are something imaginary. The real thing to bring about the feeling of close contact is social intercourse, friendly association, long familiarity, and the companionship that results from growing up together having the same wet nurse, and sharing the other circumstances of life and death. If close contact is established in such a manner, the result will be affection and cooperation.”
In Ibn Khaldun’s thought, conquest itself seems to be the driving force behind the consolidation of two asabiyah into one. Once a weaker tribal group is defeated, its leaders removed and men of valor killed, pacified, or subsumed under a new organization so utterly that the ‘tit for tat’ vengeance schemes so common to nomadic society (which Ibn Khaldun sees as the root cause of war) are no longer possible, then their asabiyah can be swallowed up in the larger group’s. What is key here is that the other groups – after their initial defeat – are not coerced into having the same feeling of asabiyah as the main group. Asabiyah that must be coerced is not asabiyah at all (this is a theme Ibn Khaldun touches on often and we will return to it in more detail when we talk about why asabiyah declines in civilized states). Instead, those who have been allowed to join the conquering host slowly start to feel its asabiyah be subsumed as the two groups “enter into close contact,” sharing the same trials, foods, circumstances, and becoming acquainted with the others’ customs, but just as importantly, sharing the same set of incentives. Once the losers are are forced together with the winners, defeat for the main clan is defeat for all; glory for the main clan is glory for all; booty gained by the main clan’s conquests becomes booty to be shared with all. Once people from a subordinate group begin to feel like the rise and fall of their own fortunes is inextricably linked to the fate of the group that overpowered them then they become willing to sacrifice and die for the sake of this group, for it has become their group.
Shared experiences create a lot of mutual information, and enough of it builds the fearsome bonds which populate our legends and histories.
More on semi-altruism: Darwin’s Unfinished Symphony argues that what sets humans apart is our ability to reliably teach (review here).
If we consistently teach well, it seems to me it must consistently yield rewards for the teachers. As with the argument for storytellers in the example above, this may be in the form of benefits from social connections. But, because we know time discounting is a thing and it takes time to teach people something, it seems likely to me that there is an intrinsinc reward for teaching. A reasonable way to describe teaching would be mutualizing information.
Seemingly altruistic actions such as creating art and music may qualify. From Sexual Selection Through Mate Choice Does Not Explain the Evolution of Art and Music:
Miller, however, criticizes the idea that “art conveys cultural values and socializes the young,” writing that,
“The view that art conveys cultural values and socializes the young seems plausible at first glance. It could be called the propaganda theory of art. The trouble with propaganda is that it is usually produced only by large institutions that can pay propagandists. In small prehistoric bands, who would have any incentive to spend the time and energy producing group propaganda? It would be an altruistic act in the technical biological sense: a behavior with high costs to the individual and diffuse benefits to the group. Such altruism is not usually favored by evolution.”
The answer to Miller’s question—who produces the propaganda?—is quite clear in the ethnographic data: the old men do.
Altruism is not usually favored by evolution, but if the same mechanism by which we prefer to spread our genetic information recognizes other kinds of information, then it would not feel altruistic from the inside. Rather, making songs and having other people sing them would be its own reward in precisely the same way having children does.
That’s an interesting point.
Unfortunately, there isn’t enough data to make good performance comparisons to my knowledge. Although I would definitely watch a medium-production-value documentary that does the work with what is available.
I note that the most advanced aircraft the United States (officially) has is the F-22. It was designed to take advantage of a few future technologies, like advanced materials and electronics.
It was also designed ~30 years ago. That’s three decades of Moore’s Law, materials science advancements and the proliferation of metamaterials, and so on. So when accounting for the possibility of it being a real craft in the air, I ask myself questions like “what could plausibly fit into that 30 years worth of advancements?”
I also note that one of the areas where we have seen considerable improvement is in compressing the design-build pipeline, which is to say we make prototypes faster than we used to. I therefore expect that the gap between what is possible on paper and what can actually fly is shorter than it was in the 1980s.
as they would not risk to crash it by flying between two airplanes in tight formation.
as they would not risk to crash it by flying between two airplanes in tight formation.
This is incorrect. They shouldn’t risk crashing by flying between a tight formation, but you’ve got to consider that people who work in top secret programs are mostly just regular people who don’t talk about their work. There is plenty of room in top secret military projects for all the same jackassery that happens in public projects, like incompetence, pranks, deliberately dangerous tests, etc. Arguably more so, since they are sheltered from scrutiny.
And this ignores more prosaic explanations like an autopilot glitch. Alpha Go made weird decisions because it was misreading the apparent score, a pilot AI would certainly encounter similar problems at some point.
My impression agrees. I am inclined to say that Chapman seems to be targeting the kind of rationality criticized in Seeing Like a State, save that In the Cells of the Eggplant is about how unsatisfying the perspective is rather than the damage implementation does.
That could be. I had assumed that when referring to the literature he was including some number of real-world examples against which those models are measured, like the number of lawsuits over breach of contract versus the estimated number of total contracts, or something. Reviewing the piece I realize he didn’t specify that, but I note that I would be surprised if the literature didn’t include anything of the sort and also that it would be unusual for him to neglect current real examples.
While I have no reason to suspect Hanson’s summary of the agency literature is inaccurate, I feel like he really focused on the question of “should we expect AI agents on average to be dangerous” and concluded the answer was no, based on human and business agents.
This doesn’t seem to address Christiano’s true concern, which I would phrase more like “what is the likelihood at least one powerful AI turns dangerous because of principal agent problems.”
One way to square this might be to take some of Hanson’s own suggestions to imagine a comparison case. For example, if we look at the way real businesses have failed as agents in different cases, and then assume the business is made of Ems instead, does that make our problem worse or better?
My expectation is that it would mostly just make the whole category higher-variance; the successes will be more successful, but the failures will do more damage. If everything else about the system stays the same, this seems like a straight increase in catastrophic risk.
we do not yet have working nuclear fusion reactors
Heh—this works exceptionally well because we do have reactors that reliably fuse things, sufficiently reliable that at least a few dozen private citizens have built them in their garage. This suggests getting what we want out of them should be pretty easy, yet the efficiency threshold is tough to crack.
Strategy is the search for victory.
Suppose we take search completely literally.
1. We have a current environment.
2. We need to find as many future branches in which we are victorious as possible.
3. We try to preserve as many possible victorious branches as possible, and try to screen as many defeat branches as possible.
4. Once victory becomes likely, we can begin to discriminate between better or worse victories.
The strategy itself is essentially the rule we use for making interventions: the causal theory of success.
We need a reference class of victories, on which to base our theory of success.
We need a reference class of defeats, for the purposes of murphyjitsu. This seems to be unusually important, because it looks kind of like avoiding errors is the more important feature, the closer we get to total mastery of the environment. I think this is largely captured by things like doctrine and training, but we haven’t done a good job of capturing it in terms of decisions.
Related to: Macroscopic Predictions. This is kind of like using Gibbs Rule at the level of “interventions we can make” to predict victory.
I experimented with manipulating the filter bubble, and while I find noticing I am in a filter bubble a useful trick for avoiding unconscious bias, I don’t find it useful when deliberately thinking about a specific thing.
Consider bubble-hopping: I find a better way is to deliberately spend time inside each of the relevant filter bubbles instead. The central benefit is that this naturally pulls your perspective above the fray. If we entertain the notion that the goal is to get the problem solved, we’ll need to understand what the conversation is among the different groups anyway.
As a practical matter, my impression from doing this periodically is that most of the time the conversations are completely different, to the extent that it is hard to recognize they are talking about the same thing.