AGI Hype: Why Industry Benefits from Existential Policy Focus
Epistemic status: exploratory. I am open to new arguments and sources.
In Short
The AGI hype coming from the AI industry is a marketing and public relations strategy, a fundraising tool, but also a policy red herring. Applying the policy-making resources to existential dangers means less focus on the down-to-earth issues of intellectual property, algorithmic transparency, or concentration of power. The public and policy-makers should remain aware of the unreliability of the industry’s claims and not let themselves be distracted by the overblown statements.
Narrow vs General AI
The discussion about AI policy is inherently intertwined with the discussion about AI safety. Our technical abilities, both in terms of building and controlling AI, determine our policies. So when that discussion also encompasses AGI, the policy and regulation for a state-of-the-art AI seems less than relevant. In extremis, one might even dispute whether “AGI policies” make any sense at all, as the economical and societal transformation with reach scales transcending our current policy-making and epistemic capabilities.
Such radical arguments are grounded in the transformative qualities of AGI. At least, if AGI is defined as a possibly autonomous intelligence, matching or surpassing human capabilities across virtually all cognitive tasks. Tellingly, though, the definitions of AGI used by the industry insiders have been getting broader of late. While OpenAI’s charter[1] defines AGI as outperforming humans, their more recent statements refer to “systems that can achieve performance levels comparable to humans across a broad spectrum of tasks”[2]. This moving of goalposts would allow the reliable agent, a remote worker replacement that OpenAI is currently promising, to be labelled as AGI.
And yet, somewhat inconsistently, this broad definition does not prevent the industry from pushing the narrative of AI—and AGI—as an unstoppable force of nature, which even its creators cannot control. So is AGI “comparable” to humans, sometimes, or is it an inevitable existential threat? And how does the AI industry benefit from blending these concepts?
Potential Scenarios
Let’s consider the following scenarios. (Again, “AGI” here stands for AI matching or surpassing human cognitive capabilities. “Narrow AI” refers to an AI confined to well‑defined tasks, a tool augmenting human work—transformative at an economic, but not existential level.)
| Policies target narrow AI | Policies target AGI | |
| AGI happens | Societies are caught by surprise, as the new technology is wildly out of scope of the existing policies. Industry implications: unclear | Regulations are potentially relevant in dealing with the transformation, depending on its pace and scale. Industry implications: unclear |
| AGI doesn’t happen | Policies shape the actual applications of AI, potentially circumscribing AI industry’s power. Industry implications: negative | Loose regulation allows the AI industry to harvest the gains non-AGI technologies with few guardrails. Industry implications: positive |
Industry Incentives
With these options in mind, policies focused on AGI are the best bet for the AI industry. Scenarios involving the emergence of AGI have unpredictable implications both for the societies and for the industry. However, the policy-making focused on transcendental trajectory combined with narrow AI gives the industry a huge advantage. Regulatory focus shifts towards existential issues and away from intellectual property, algorithmic transparency, concentration of power, or economic exclusion. Weak or non-existent regulation in these operational matters, paired with actually controllable technology, lays out a smooth path to profits and influence. Of course, if the AGI revolution happens, the current business models likely cease to make sense anyway, so policies make less of a difference. It is only rational for anyone with a stake in the AI industry to advocate for the AGI-focused policies. And even though the AGI hype-instigated fear often leads to support for stronger regulations, in practice geopolitical concerns lead to deprioritizing safety concerns—exactly because they AGI is seen as a strategic asset in the AI arms race[3][4].
Regardless of these incentives, the “force of nature” narrative still holds if we assume that it is exactly the industry insiders that know most about the technology, and can therefore reliably predict its development. However, academia still publishes the bulk of AI research papers[5], even as the privatisation of research takes off. We do have a large and reliable expert body outside of the Silicon Valley available to advise us. And their narratives tend to be more level-headed[6].
Takeaways
I do not argue that everyone predicting AGI by 2030 does so out of self-serving motivations. And perhaps the senior industry stakeholders are genuinely concerned about humanity’s future and would not mislead the public for their own gain. While we can analyze their incentives, though, it is impossible to present evidence on their intentions. There is no way to know when we are dealing with honest enthusiasm, and when with a cynical sleight of hand. All in all, though, when coming from the AI industry, the AGI hype should be interpreted as a public relations distraction as much as (or more so than) technological insight. Let’s not let panic fuel either passive fatalism, nor divert attention and resources away from the narrow AI policy-making.
PS I am leaving out of this argument:
- whether AGI (by any definition) will or will not happen within a specific timeframe,
- all the other ways in which the AGI hype is serving the industry.
- ^
OpenAI Charter: “artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work”
- ^
Definition from their leaked documents (AI+). Reportedly (TechCrunch), the definition in their agreement with Microsoft is even more pragmatic, and defined by the profit-making rather than cognitive capabilities of AGI.
- ^
- ^
- ^
- ^
AAAI 2025 Presidential Panel On The Future of AI Research—March 2024 - “The majority of respondents (76%) assert that “scaling up current AI approaches” to yield AGI is “unlikely” or “very unlikely” to succeed”
If I believe AGI is imminent, that it’s an extinction risk, and that therefore, we should stop building AI; how do I get people to take the second half of that seriously, given that policymakers and funders seem to respond to that argument as though everything after “therefore” is empty and meaningless, to be used only as a shibboleth to calm the public? To put this another way—if I imagine myself in a world where AGI predictions primarily have the effect of hype, and that thus people who are systematically dishonest have an incentive to make them and manipulate anyone else who believes the prediction into making it counterproductively; then, that doesn’t change my prediction that I think it’s imminent.
if you imagine yourself in a world where agi is imminent, does that change your prediction about whether any of the people predicting it are honest about their reasons? does it change any policy proposals? because if, eg, you’d say “well, they’re honest, and might be right, but those are bad policies for that world, too” then I’m much more enthusiastic about policies that focus on immediately measurable harms. But if your policies are chosen for worlds where you actually think AGI is extremely unlikely, rather than merely that hype serves altman’s interests, then I think you’re making a serious mistake.
I’m in favor of whatever policies actually reduce damage from AI, and if it is the case that we live in a world where AGI is not near, I only want policies for the AI we have now. If we live in a world where AGI is near, then I want policies that intervene most effectively to prevent damage from AI including AGI, which might well be best designed as policies that focus on current harms first—in my view, most of the current harms are from mechanisms similar in kind but not intensity to how AGI would be bad, and so policies that are designed to handle them may be quite helpful in preventing doom. But of course samuel alternate boy is likely to disagree.
If you believe AGI is imminent, then of course you want to develop policies that will address the related problem. I do not think we should summarily dismiss the potential risks of AGI, and I do say explicitly that I am not arguing about the AGI timelines or probabilities here. What I do argue is that we should not base our belief in imminent AGI—and therefore policy choices—solely on the messaging from the AI industry players. And they, whether we like it or not, are now to great extent shaping the public discourse.
Which of these two approaches:
Sam Altman says AGI is coming > let’s focus all policy effort and resources on his scenarios of AGI, or
AGI is potentially coming > let’s review arguments and research from across the field to weigh the probabilities > let’s distribute policy effort and resources across short-, mid-, and long-term risks accordingly
do you think will bear better policy choices? Approach #2 may well conclude that AGI is extremely likely, but these conclusions will have a sounder and broader base.