The real problem is sentience and safety. The growing gap between reality and belief is a contributing problem, but much smaller IMO than the quite real possibility that AI wakes up and takes over. Framing it as you have suggests you think there can only be one “real problem”; I assume you mean that the gap between reality and belief is a bigger problem than AI alignhment that deserves more effort. I am almost sure that safety and sapience is getting far too little attention and work, not too much.
Alignment/AGI safety being a huge, indeed pretty clearly the biggest problem is the general opinion on LW. And this population are the ones that think hardest on average about this issue. I mention this to clarify the audience you’re speaking to here (with large variances of course). In my opinion, the arguments for AI x-risk (I don’t worry much about minor harms from current or next-gen systems) are overwhelming, and the vast majority of people who engage with them in good faith come to believe them.
If you think that’s not true, by all means engage with the arguments and LW will listen. We would love to quit believing that AI is a deadly threat; we tend to love technology and progress and even AI—until it crosses the threshold to sapience (very roughly speaking) and its goals, whether we got them exactly right or not, control the future.
I don’t actually know of a good reference for convincing people that alignment needs to be solved whether it’s hard or easy. One perspective I want to write about is that optimists think we’re building AI tools, so they aren’t that dangerous and don’t need to be aligned. I would agree with that, except that it seems highly likely, for deep reasons, that those tools will be used to build sapient, human-like AGI with goals and values like ours- but different enough that they will outcompete us and we will not get a world we like.
That’s the context for your article and probably why it’s not getting a good response.
To the content: I think your point is reasonable. And LW readers tend to like and use AI and be very interested in its progress, so this is something the local readership might care about if it weren’t claimed to be more important than safety and sentience.
But are you sure that industries haven’t done this? It seems like power-users opinions are amplified on Twitter for the big LLMs and developers probably pay close attentionl. For more niche AI systems like music creation, agents (currently niche, soon to be the biggest thing to ever happen IMO), each developer has Discords for power-users to give feedback.
Thanks—this helps me understand how my framing came across. To clarify, I’m not arguing that AI is harmless or that alignment isn’t important. I’m saying misalignment is already happening—not because these systems have minds or goals, but because of how they’re trained and how we use them.
I also question the premise that training alone can produce sapience. These are predictive systems—tools that simulate reasoning based on data patterns. Treating them as if they might “wake up” risks misdiagnosing both the present and the future.
That’s why I focus on how current tools are used, who they empower, and what assumptions shape their design. The danger isn’t just in the future—it’s in how we fail to understand what these systems actually are right now.
And we’re not going to slow this down with abstract concerns.
Chip makers are going to keep making chips.
There’s money in compute — whether it powers centralized server farms or locally run AI models. That hardware momentum won’t stop because we have philosophical doubts or ethical concerns. It will scale because it can.
But if that growth is concentrated in systems we don’t fully understand, we’re not scaling intelligence — we’re scaling misunderstanding.
The best chance we have to stay aligned is to get AI into the hands of real people, running locally, where assumptions get tested and feedback actually matters.
The development of AI today looks a lot like the early days of computing: centralized, expensive, and tightly controlled.
We’re in the mainframe era — big models behind APIs, optimized for scale, not for user agency.
There was nothing inevitable about the rise of personal computing.
It happened because people demanded access. They wanted systems they could understand, modify, and use on their own terms — and they got them. That shift unlocked an explosion of creativity, capability, and control.
We could see the same thing happen with AI.
Not through artificial minds or sentient machines, but through practical tools people run themselves, tuned to their needs, shaped by real-world use.
The kinds of fears people project onto AI today — takeover, sentience, irreversible control — aren’t just unlikely on local machines.
They’re incompatible with the very idea of tools people can inspect, adapt, and shut off.
That gives me a better idea of where you’re coming from.
I think the crux here is your skepticism taht we will get sapient AI soon after we get useful tool AI. This is a common opiniion or unstated assumption (as it was in your original piece).
(I think “sapience” is I think the more relevant term, based roughly on “understanding”, vs “sentience” based roughly on “feeling”. But sentience is used where sapience should be more often than not, so if that’s not what you mean, you should clarify. Similarly, safety is used overlapping with x-risk, and not. So if you meant it doesn’t matter if AI feels or produces minor harms, I agree- but I don’t think that’s what you meant, and I’d expect it to be misinterpreted by a majority if it was.)
Now, I actually agree with you that training alone won’t produce sapient AGI, what I’ve termed “Real AGI”. Or at least not obviously or quickly.
But developers will probably pursue a variety of creative means to get to competent and therefore useful and dangerous AGI. And I think a fair assessment is that some routes could work very rapidly—nobody knows for sure. I think highly capable tool AI is setting the stage for sapient and agentic AGI very directly: with a capable enough tool, you mearly prompt it repeatedly with “continue working to accomplish goal X” and it will reflect and plan as it considers appropriate- and be very very dangerous to the extent it is competent, since your definition of goal X could easily be interpreted differently than you intended it. And if it’s not, someone else in your wide web of democratized AI usage will give their proto-AGI a humanity-threatening goal, either on purpose or by accident- probably both, repeated hundreds to millions of times to various degrees.
The real problem is sentience and safety. The growing gap between reality and belief is a contributing problem, but much smaller IMO than the quite real possibility that AI wakes up and takes over. Framing it as you have suggests you think there can only be one “real problem”; I assume you mean that the gap between reality and belief is a bigger problem than AI alignhment that deserves more effort. I am almost sure that safety and sapience is getting far too little attention and work, not too much.
Alignment/AGI safety being a huge, indeed pretty clearly the biggest problem is the general opinion on LW. And this population are the ones that think hardest on average about this issue. I mention this to clarify the audience you’re speaking to here (with large variances of course). In my opinion, the arguments for AI x-risk (I don’t worry much about minor harms from current or next-gen systems) are overwhelming, and the vast majority of people who engage with them in good faith come to believe them.
If you think that’s not true, by all means engage with the arguments and LW will listen. We would love to quit believing that AI is a deadly threat; we tend to love technology and progress and even AI—until it crosses the threshold to sapience (very roughly speaking) and its goals, whether we got them exactly right or not, control the future.
I recommend Jessicata’s A case for AI alignment being difficult for understanding that part of the argument.
I don’t actually know of a good reference for convincing people that alignment needs to be solved whether it’s hard or easy. One perspective I want to write about is that optimists think we’re building AI tools, so they aren’t that dangerous and don’t need to be aligned. I would agree with that, except that it seems highly likely, for deep reasons, that those tools will be used to build sapient, human-like AGI with goals and values like ours- but different enough that they will outcompete us and we will not get a world we like.
That’s the context for your article and probably why it’s not getting a good response.
To the content: I think your point is reasonable. And LW readers tend to like and use AI and be very interested in its progress, so this is something the local readership might care about if it weren’t claimed to be more important than safety and sentience.
But are you sure that industries haven’t done this? It seems like power-users opinions are amplified on Twitter for the big LLMs and developers probably pay close attentionl. For more niche AI systems like music creation, agents (currently niche, soon to be the biggest thing to ever happen IMO), each developer has Discords for power-users to give feedback.
Thanks—this helps me understand how my framing came across. To clarify, I’m not arguing that AI is harmless or that alignment isn’t important. I’m saying misalignment is already happening—not because these systems have minds or goals, but because of how they’re trained and how we use them.
I also question the premise that training alone can produce sapience. These are predictive systems—tools that simulate reasoning based on data patterns. Treating them as if they might “wake up” risks misdiagnosing both the present and the future.
That’s why I focus on how current tools are used, who they empower, and what assumptions shape their design. The danger isn’t just in the future—it’s in how we fail to understand what these systems actually are right now.
And we’re not going to slow this down with abstract concerns. Chip makers are going to keep making chips. There’s money in compute — whether it powers centralized server farms or locally run AI models. That hardware momentum won’t stop because we have philosophical doubts or ethical concerns. It will scale because it can.
But if that growth is concentrated in systems we don’t fully understand, we’re not scaling intelligence — we’re scaling misunderstanding. The best chance we have to stay aligned is to get AI into the hands of real people, running locally, where assumptions get tested and feedback actually matters.
The development of AI today looks a lot like the early days of computing: centralized, expensive, and tightly controlled. We’re in the mainframe era — big models behind APIs, optimized for scale, not for user agency.
There was nothing inevitable about the rise of personal computing. It happened because people demanded access. They wanted systems they could understand, modify, and use on their own terms — and they got them. That shift unlocked an explosion of creativity, capability, and control.
We could see the same thing happen with AI. Not through artificial minds or sentient machines, but through practical tools people run themselves, tuned to their needs, shaped by real-world use.
The kinds of fears people project onto AI today — takeover, sentience, irreversible control — aren’t just unlikely on local machines. They’re incompatible with the very idea of tools people can inspect, adapt, and shut off.
That gives me a better idea of where you’re coming from.
I think the crux here is your skepticism taht we will get sapient AI soon after we get useful tool AI. This is a common opiniion or unstated assumption (as it was in your original piece).
(I think “sapience” is I think the more relevant term, based roughly on “understanding”, vs “sentience” based roughly on “feeling”. But sentience is used where sapience should be more often than not, so if that’s not what you mean, you should clarify. Similarly, safety is used overlapping with x-risk, and not. So if you meant it doesn’t matter if AI feels or produces minor harms, I agree- but I don’t think that’s what you meant, and I’d expect it to be misinterpreted by a majority if it was.)
Now, I actually agree with you that training alone won’t produce sapient AGI, what I’ve termed “Real AGI”. Or at least not obviously or quickly.
But developers will probably pursue a variety of creative means to get to competent and therefore useful and dangerous AGI. And I think a fair assessment is that some routes could work very rapidly—nobody knows for sure. I think highly capable tool AI is setting the stage for sapient and agentic AGI very directly: with a capable enough tool, you mearly prompt it repeatedly with “continue working to accomplish goal X” and it will reflect and plan as it considers appropriate- and be very very dangerous to the extent it is competent, since your definition of goal X could easily be interpreted differently than you intended it. And if it’s not, someone else in your wide web of democratized AI usage will give their proto-AGI a humanity-threatening goal, either on purpose or by accident- probably both, repeated hundreds to millions of times to various degrees.
More in LLM AGI will have memory, and memory changes alignment, ,and If we solve alignment, do we die anyway?,
Democratizing AI is a common intuition, and I think it’s motivated by valid concerns. Yours are less common. See Fear of centralized power vs. fear of misaligned AGI: Vitalik Buterin on 80,000 Hours for both sides of the argument.