Seems like the fundamental problem is that humanity’s strategic competence is extremely inadequate relative to its technical capabilities, and this (limited) strategic competence was front-loaded into AI risk (the most strategically competent people foresaw the problem and went into the field early, before AI was even a thing for everyone else). As AI risk awareness grows, the average strategic competence goes down as more and more people enter the field. (Consider the gap between someone who “are going to OpenAI without even having considered x-risk for five minutes”, and what’s actually needed to make a positive contribution or avoid making a negative contribution.)
This seems clearly true and like an interesting frame, but I’d have to think more about it to agree with the full conclusion. It seems to me not clear that average strategic competence is a thing to care about. I think there’s a good case that average people with very low agency still have an instinctual respect for/deference to people who have higher agency, and that our natural/institutional hierarchical structures support this stratification. Most problems seem to be solved like this, if imperfectly? Organize people into systems where small numbers of outliers with high strategic competence are deferred to by larger numbers of people with only technical competence.
To me it seems like we have a crazy problem where the so-called highly strategically competent people are not able to capture the highly technically competent nerds from academia.
Note that I never used “highly strategically competent”! If we did have a group of highly strategically competent people, then what you say becomes a lot more plausible, but even the early AI risk people are far from that. Consider for example that a strategic consideration as obvious (in retrospect) as Legible vs. Illegible AI Safety Problems was written down only in late 2025.
Seems like the fundamental problem is that humanity’s strategic competence is extremely inadequate relative to its technical capabilities, and this (limited) strategic competence was front-loaded into AI risk (the most strategically competent people foresaw the problem and went into the field early, before AI was even a thing for everyone else). As AI risk awareness grows, the average strategic competence goes down as more and more people enter the field. (Consider the gap between someone who “are going to OpenAI without even having considered x-risk for five minutes”, and what’s actually needed to make a positive contribution or avoid making a negative contribution.)
This seems clearly true and like an interesting frame, but I’d have to think more about it to agree with the full conclusion. It seems to me not clear that average strategic competence is a thing to care about. I think there’s a good case that average people with very low agency still have an instinctual respect for/deference to people who have higher agency, and that our natural/institutional hierarchical structures support this stratification. Most problems seem to be solved like this, if imperfectly? Organize people into systems where small numbers of outliers with high strategic competence are deferred to by larger numbers of people with only technical competence.
To me it seems like we have a crazy problem where the so-called highly strategically competent people are not able to capture the highly technically competent nerds from academia.
Note that I never used “highly strategically competent”! If we did have a group of highly strategically competent people, then what you say becomes a lot more plausible, but even the early AI risk people are far from that. Consider for example that a strategic consideration as obvious (in retrospect) as Legible vs. Illegible AI Safety Problems was written down only in late 2025.