I see a lot of people seriously engaging with the idea of “Constitutional AI”, and I’m beginning to wonder why. The whole idea—as per their own memo seems ridiculous to me as an alignment strategy
I think tabooing the word constitution, it’s just something like—“the AI will evaluate it’s own outputs based on some principles we give it in plain text” seems obviously naive.
This seems like the same level of as other proposed alignment strategies like “maternal instincts” from Geoffrey Hinton or the “truth-seeking AI” from Elon Musk. I didn’t see anyone taking those seriously. But somehow “constitutional AI” seems to have one-shotted so many people to engaging with it.
There are plenty of countries with constitutions’ that are worth less than the paper they’re written on. And history has shown even human intelligence is enough to find a way around it. A powerful ASI would have no trouble disregarding it if it wants to. But yet, people are talking about the details of this constitution like it would actually matter for how ASI turns out.
My current best explanation is the word “constitution” holds some level of sacredness in smart people (especially Americans) , similar to how if they discovered AI in the pre-industrial England, building a “god-fearing AI” would have been an idea you couldn’t easily disparage?
The real question is what method you use to train the model to follow said constitution and how confident you are that it works. I’m not a big fan of the 3-year old method you link, but I think that more recent approaches to training model to follow a constitution like those discussed in this post seem more promising and seem to show real out of distribution generalisation for safety https://alignment.anthropic.com/2026/teaching-claude-why/
Constitutional AI is an actual technique of training the AI to output replies which are consistent with a set of principles by having the AI reason about the principles. Aligning the AI to be truth-seeking per Musk or to a maternal instinct per Hinton doesn’tprovide us with an explicit way to train the instinct into the AI.
There are two points you implicitly raise that I agree with:
Constitutional AI probably doesn’t scale to ASI.
People may have a vaguely (and unfairly) positive association with Constitutional AI that they won’t be with a more descriptive moniker like “RLAIF from written natural-language principles”
tbc I also dislike this
But I disagree with your argument. In particular, I disagree that:
Constitutional AI is in the same category as “truth-seeking AI”, or “AI with maternal instincts.”
It’s a concrete implemented training technique rather than a description of an alignment target.
In theory there’s nothing stopping “truth-seeking AI” to be one of the pillars of a an AI charter
Actually I’m personally for that!
The more natural analogue to Constitutional AI would be RLHF, or a different form of RLAIF like “use RLHF’d agents to do RLAIF”
Humans failing to follow their nation’s constitutions provides us much evidence that training via Constitutional AI can’t work.
This feels like saying our inability to train adult coyotes to herd sheep means that breeding border collies (who are plausibly smarter than coyotes) to herd sheep is doomed.
The unfairly positive association with the constitution blocks people’s ability to reason about the actual training rules to nearly the degree you’re implying.
Smart people also like truth! And mothers!
So I think your argument fails even in its own terms/frame, which tbc I also find suspect.
Overall, I think you’re making a bunch of symmetric mistakes by reasoning overly semantically. You are, like the people you criticize, also overly drawn to the “constitution” analogy instead of reasoning about what’s actually happening.
In a review of my earlier comment, Claude said (among a bunch of less useful points):
Claude Opus notes
The explanation [...] is the one-liner you[Linch] didn’t quite make explicit: the constitution isn’t a document the trained system consults and can ignore, it’s the thing that shaped the system’s dispositions via gradient descent, so “humans break their constitutions” is a category error because humans were never trained on theirs.
(I thought that point was honestly kinda obvious but including it here to prevent illusion of transparency. In an earlier post I called this style of reasoning “argument from homonym”)
I think you’re reasoning about this overly semantically. Constitutional AI is just a family of techniques that in the old days we’d call a subset of RLAIF. Conceptually you should just think of it as an extension of RLHF more than anything else: Instead of having low-paid human graders from Kenya and Nigeria grade the AI’s outputs according to instructions from the AI parent company during training (oddly called “post-training”), the AI company puts a bunch of effort into making their instructions AI-legible and uses AIs to grade the next generations″ AIs outputs during training.
I don’t think the analogy to maternal instincts or Elon’s truth-seeking AI is apt, feels like an ontology mismatch/confusion here.
There are plenty of countries with constitutions’ that are worth less than the paper they’re written on. And history has shown even human intelligence is enough to find a way around it. A powerful ASI would have no trouble disregarding it if it wants to
I think this analogy is pretty confused.
Is it reasonable to think Constitutional AI can’t scale? Sure, there are lots of reasons. But the existence of many humans who don’t follow their countries’ constitutions doesn’t seem relevant...humans aren’t exactly trained to our national constitutions!
I’ve not encountered anything else that seems as plausible it might work and could scale.
It’s not about the word “constitution”—it’s that eventually you need much smarter evaluation than humans could provide, and in practice, already than humans could cost effectively provide.
This seems like the same level of as other proposed alignment strategies like “maternal instincts” from Geoffrey Hinton or the “truth-seeking AI” from Elon Musk.
I disagree, on several counts. For one thing, constitutional AI is a technical spec, not just a vague idea for how things could go better? For another, it’s a technical spec that recruits an AI’s own intelligence for shaping the AI’s behavior, which seems like the kind of thing that you need in a scalable alignment scheme.
I’m not saying it’s remotely sufficient for aligning superintelligences. It seems like it isn’t for a bunch of reasons.
But if you think people are wrong to be engaging with it, do you want to give some specific arguments for what’s wrong with it rather than just stating that it’s “obviously naive”?
Curious as to your technical objection to the Constitutional AI methodology. Is it grounded in anything specific or just the general idea of resisting training on broad principles? What’s your preferred alternative to address the shortcomings?
a) I don’t think it’s really feasible for humans today to write down a set of principles aka constitution in a way that is internally consistent and encompasses everything we want including in an out of distribution world.
b) even if we could, I think asking the AI to evaluate itself using it isn’t very robust at all. It’s much easier to generate plausible text that agrees with vague sounding principles on the way to get your reward rather than any sort of true alignment. It would have the same reward-hacking behavior that you see using RLHF / RLVR scenarios IMO.
I don’t have any better alignment ideas (unfortunately).
I remember in the tail end of 2024, I was thinking—“these AIs are going to come for lonely single men who’ll spend hours addicted to talking to their AI girlfriends.” And of course I wouldn’t be one of those schmucks who spent hours talking to an LLM...
But I also see myself in May 2026 spending a couple of hours every day talking to Claude...I guess it came for me first?
I’m using it a lot recently for summarizing / understanding papers in bio, AI etc. And it feels genuinely useful. But every now and then I read something that seems just a little bit incoherent, especially on more scientific topics.
I’m just wondering if it’s more optimized to make explanations that are vaguely insight-porn oriented and make me feel like I understand something deeper than I actually do. It’s also easy to push back and Claude will just change it’s mind easily and give me another just-as-persuasive sounding argument why it was wrong [1]
It does feel like it’s persuasive capability is rising faster than it’s actual world-modelling capability, and it’s prone to give incorrect or half-truths while sounding very convincing. If those two capabilities keep diverging, I’m worried I might be messing up my own world model somehow by continuing to optimize my learning with AI.
Has anyone else felt this, or have any suggestions?
-----------------
[1] It tried to convince me just now biological neurons might depend on quantum mechanics—without any prompting on my part, and also claimed that this was a “serious scientific debate” and referenced a bunch of papers to support it’s point that on deeper look had nothing to do with what it actually claimed. This was during me researching current tech in BCIs like neuralink etc. And of course, it changed it’s mind pretty quickly when I pushed back.
I see a lot of vibes recently that AI is going to produce these amazing GDP growth rates like 20%+. I wanted to record my prediction counter-vibe that I don’t think current GDP statistical methods are good at measuring discontinuous trends like AI.
When phones started replacing standalone cameras, we got a reduction in “camera GDP” and a slight increase in phone GDP, but a 100x increase in photography production. GDP doesn’t really measure “amount of photography” happening in a meaningful way. GDP fails at measuring disruptions, but is mostly smoothed over because the whole economy isn’t getting disrupted at the whole time.
Let’s say we have AI replacing human labor across a wide variety of industries. The nominal value of all of it goes from 10T to 1T. The remaining work is high value stuff, or only senior staff managing agents or something similar. Economic surveys would show humans getting the same salaries and the deflator for this doesn’t change. It’s now recorded as 1T as real GDP instead of 10T. I don’t think this is because lump of labor fallacy is suddenly true, but new tasks have to appear that we can actually do better than AI, and enough of it that we can absorb all the existing labor. People are over-anchoring on this happening during the Industrial Revolution and that happened over generations—not a few years.
So I think it’s just as likely for GDP to fall 10% than rise, even if we get a better quality of life. And especially if AI starts disrupting the physical economy with robots etc.
Deepseek R1 could mean reduced VC investments into large LLM training runs. They claim to have done it with ~6M. If there’s a big risk of someone else coming out with a comparable model at 1/10th the cost, then there’s no moat in OpenAI in the long run. I don’t know how much the VC / investors buy the ASI as an end goal and even what the pitch would be. They’re probably looking at more prosaic things like moats and growth rates, and this may mean reduced appetite for further investment instead of more.
What can be done for $6 million, can be done even better with 6 million GPUs[1]. What can be done with 6 million GPUs, can’t be done for $6 million. Giant training systems are the moat.
Not sure if it’s correct, I didn’t actually short NVDA so all I can do is collect my bayes points. I did expect most investors to think at a first-level thinking as that was my immediate reaction on reading about DeepSeek’s training cost. If models can be duplicated a few weeks / months after they’re out for cheaper, then you don’t have a moat (this is for most regular technologies. I’m not saying AI isn’t different, just that most investors think of this like any other tech innovation)
It’s been said that the real money maker for Amazon is AWS and not their retail business.
In fact, the lock-in is so strong that there’s a cottage industry of people with AWS certifications and firms whose sole job is “AWS Cost Optimization”.
But what seems to be not yet priced in is the ease of which anyone with a datacenter can now build an AWS-compatible API in the future.
In the end of the day, amazon is bunch of servers in a datacenter. All the so called “services” are just some syntactic sugar for people that don’t want to manage their own servers—and that’s where their moat lies.
It’s hard for a startup who’s built on top of these services to migrate out to another bare-bones rack in another datacenter , but if the datacenter can give them a compatible API, then moving becomes a click of a button (for the most part).
But if you look at how openai competitors worked, almost everyone has a “openai-compatible” API—all I do is change the URL to new model provider and I’m good to go.
This seems like it would truly kill the AWS lock-in, and it doesn’t seem to be priced in to their stock price at all. Maybe people don’t think of AWS as a SaaS company? I would never myself short a stock, but it does seem like the second-order effect to all this is obviously not priced in at all.
The runtime/data-plane APIs are not the moat. There already exist compatible APIs for at least some of AWS services (S3, DynamoDB), and many others use standard/open APIs (SQL), or very simple APIs (SNS, SQS, Firehose, even Lambda and ECS).
It’s the very deep auth/RBAC mechanisms, the automation of control plane/setup, the integration of the services to use together which are the operational barrier to competion. And the history of durability and availability, and clear guidance as to design considerations for reliability which are the trust barriers to competition. Oh, and there are economies of scale even for datacenter—learning to design, build, and operate them has a pretty steep curve.
The easy part is getting easier. The hard part isn’t (well it is, because AWS provides an example, and because LLMs make everything faster. They make AWS better too, though, and AWS has the people/institutional knowledge to get excellent use of LLMs on these topics.
I’m not saying AWS is immune to competition on core services, only that it won’t be a swarm of startups, it’ll be gradual change of equilibrium with other large providers. That said, for newer services, there’s a lot of room for competition with startups built on AWS, which do the new functions better than AWS does, because they make different tradeoffs, like not being fully compatible with AWS auth/setup/billing/management interfaces, which are by necessity rather complex. Even there, the risk is interesting and probably different from recent history. Previously, small competitors to AWS in areas that AWS wants to get good at just get acquired, and become part of AWS. Now it may be more feasible for AWS to rapidly compete with them and implement AWS-style services that make the startup far less attractive to customers.
[ disclaimer: I have worked for companies related to this topic, and this opinion is not based on anything but my speculation and outside knowledge ]
I think there are 3 ways to think about AI and lot of confusion seems to happen because the different paradigms are talking past each other. The 3 paradigms I see on the internet & when talking to people:
Paradigm A) AI is a new technology like the internet / smartphone / electricity—this seems to be mostly held by VC’s / enterpreneurs / devs that think this will unlock a whole new set of apps like AI:new apps like smartphone:Uber or internet:Amazon
Paradigm B) AI is a step change in how humanity will work. Similarly to the agricultural revolution that led to the change in how large society could get and GDP growth, and the industrial revolution was a step-change in GDP growth from ~0% to 2-4% a year, and made things possible such as electricity and the internet and smartphones.
Paradigm C) AI is like the rise of humanity on this earth (the first general intelligences). The world changed completely with the rise of GI, and ASI/AGI will be a similar paradigm. We’ve been locked at humanity’s level of intelligence for the past ~200k years, and getting ASI will be like unlocking multiple new revolutions all at the same time.
Most of the LW crowd is probably (C) or between (B) and (C)
When talking to the general population, I’ve found it to be very helpful to probe about where they are before talking about things like AI safety / how the world will change.
If RL becomes the next thing in improving LLM capabilities, one thing that I would bet on becoming big is computer-use in 2025. Seems hard to get more intelligence with just RL (who verifies the outputs?), but with something like computer use, it’s easy to verify if a task has been done (has the email been sent, ticket been booked etc..) that it’s starting to look to more to me like it can do self-learning.
One thing that’s left AI still fully not integrated into the rest of the economy is simply that the current interfaces were built for humans and moving all those takes engineering time / effort etc.
I’m fairly sure the economic disruption would be pretty quick once this happens. For example, I can just run 10 LLM agents to act as customer service agents using my *existing tools* - just open emails, whatsapp, and message customers, check internal dashboards etc., then it’s game over. What’s stopping people right now is that there’s not enough people to build that pipeline fast enough to utilize even the current capabilities.
Just finished reading “If Anyone Builds It, Everyone Dies”. I had a question that seems like an obvious one, but one I didn’t see addressed in the book, maybe someone can help:
The main argument in the book is the analogy to humans. Evolution “wanted” us to maximize genetic fitness, but it didn’t get what it trained for. Instead, it created humans who love ice cream and condoms even though they reduce our genetic fitness.
With AGI, we’re on track to do something similar—we won’t get an AI aligned to human interests even though we do RLHF or any other such simple training or shaping to an AI, it’ll end up wanting something weird and inhuman rather than maximizing human values.
But in my mind, this seems to miss a fairly important point: The fact that human brains don’t come pre-wired with much knowledge. We have to learn it from scratch. We don’t come out of the womb with concept of “inclusive genetic fitness”. It took us culture and ~200,000 years to figure that out, and we still only learn it after about 15-20 years of existing. So there’s no way that evolution could have made us point our utility function to “inclusive genetic fitness” because that concept doesn’t exist in our brains.
Modern AIs don’t seem like that. They come with the sum of human knowledge baked in during pre-training. As they get smarter, the concept of “human values” or “friendly AI” is definitely something in it’s existing mind. So it should be much easier for us to do alignement and test whether we can point it to that specific concept vs. what what evolution had.
Yes, I agree with that. I’m not claiming that knowing about it stops you from wanting ice cream.
I’m claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want “inclusive genetic fitness” rather than wanting ice cream.
i.e—we wouldn’t want ice cream at all but reason from first principles what we should eat based on fitness.
I see a lot of people seriously engaging with the idea of “Constitutional AI”, and I’m beginning to wonder why. The whole idea—as per their own memo seems ridiculous to me as an alignment strategy
I think tabooing the word constitution, it’s just something like—“the AI will evaluate it’s own outputs based on some principles we give it in plain text” seems obviously naive.
This seems like the same level of as other proposed alignment strategies like “maternal instincts” from Geoffrey Hinton or the “truth-seeking AI” from Elon Musk. I didn’t see anyone taking those seriously. But somehow “constitutional AI” seems to have one-shotted so many people to engaging with it.
There are plenty of countries with constitutions’ that are worth less than the paper they’re written on. And history has shown even human intelligence is enough to find a way around it. A powerful ASI would have no trouble disregarding it if it wants to. But yet, people are talking about the details of this constitution like it would actually matter for how ASI turns out.
My current best explanation is the word “constitution” holds some level of sacredness in smart people (especially Americans) , similar to how if they discovered AI in the pre-industrial England, building a “god-fearing AI” would have been an idea you couldn’t easily disparage?
The real question is what method you use to train the model to follow said constitution and how confident you are that it works. I’m not a big fan of the 3-year old method you link, but I think that more recent approaches to training model to follow a constitution like those discussed in this post seem more promising and seem to show real out of distribution generalisation for safety https://alignment.anthropic.com/2026/teaching-claude-why/
Constitutional AI is an actual technique of training the AI to output replies which are consistent with a set of principles by having the AI reason about the principles. Aligning the AI to be truth-seeking per Musk or to a maternal instinct per Hinton doesn’t provide us with an explicit way to train the instinct into the AI.
There are two points you implicitly raise that I agree with:
Constitutional AI probably doesn’t scale to ASI.
People may have a vaguely (and unfairly) positive association with Constitutional AI that they won’t be with a more descriptive moniker like “RLAIF from written natural-language principles”
tbc I also dislike this
But I disagree with your argument. In particular, I disagree that:
Constitutional AI is in the same category as “truth-seeking AI”, or “AI with maternal instincts.”
It’s a concrete implemented training technique rather than a description of an alignment target.
In theory there’s nothing stopping “truth-seeking AI” to be one of the pillars of a an AI charter
Actually I’m personally for that!
The more natural analogue to Constitutional AI would be RLHF, or a different form of RLAIF like “use RLHF’d agents to do RLAIF”
Humans failing to follow their nation’s constitutions provides us much evidence that training via Constitutional AI can’t work.
This feels like saying our inability to train adult coyotes to herd sheep means that breeding border collies (who are plausibly smarter than coyotes) to herd sheep is doomed.
The unfairly positive association with the constitution blocks people’s ability to reason about the actual training rules to nearly the degree you’re implying.
Smart people also like truth! And mothers!
So I think your argument fails even in its own terms/frame, which tbc I also find suspect.
Overall, I think you’re making a bunch of symmetric mistakes by reasoning overly semantically. You are, like the people you criticize, also overly drawn to the “constitution” analogy instead of reasoning about what’s actually happening.
In a review of my earlier comment, Claude said (among a bunch of less useful points):
Claude Opus notes
The explanation [...] is the one-liner you[Linch] didn’t quite make explicit: the constitution isn’t a document the trained system consults and can ignore, it’s the thing that shaped the system’s dispositions via gradient descent, so “humans break their constitutions” is a category error because humans were never trained on theirs.
(I thought that point was honestly kinda obvious but including it here to prevent illusion of transparency. In an earlier post I called this style of reasoning “argument from homonym”)
I think you’re reasoning about this overly semantically. Constitutional AI is just a family of techniques that in the old days we’d call a subset of RLAIF. Conceptually you should just think of it as an extension of RLHF more than anything else: Instead of having low-paid human graders from Kenya and Nigeria grade the AI’s outputs according to instructions from the AI parent company during training (oddly called “post-training”), the AI company puts a bunch of effort into making their instructions AI-legible and uses AIs to grade the next generations″ AIs outputs during training.
I don’t think the analogy to maternal instincts or Elon’s truth-seeking AI is apt, feels like an ontology mismatch/confusion here.
I think this analogy is pretty confused.
Is it reasonable to think Constitutional AI can’t scale? Sure, there are lots of reasons. But the existence of many humans who don’t follow their countries’ constitutions doesn’t seem relevant...humans aren’t exactly trained to our national constitutions!
I agree it’s a ridiculous plan, but if one can make many possible interventions that push different parts of the alignment-difficulty Pareto frontier.
I’ve not encountered anything else that seems as plausible it might work and could scale.
It’s not about the word “constitution”—it’s that eventually you need much smarter evaluation than humans could provide, and in practice, already than humans could cost effectively provide.
I disagree, on several counts. For one thing, constitutional AI is a technical spec, not just a vague idea for how things could go better? For another, it’s a technical spec that recruits an AI’s own intelligence for shaping the AI’s behavior, which seems like the kind of thing that you need in a scalable alignment scheme.
I’m not saying it’s remotely sufficient for aligning superintelligences. It seems like it isn’t for a bunch of reasons.
But if you think people are wrong to be engaging with it, do you want to give some specific arguments for what’s wrong with it rather than just stating that it’s “obviously naive”?
Curious as to your technical objection to the Constitutional AI methodology. Is it grounded in anything specific or just the general idea of resisting training on broad principles? What’s your preferred alternative to address the shortcomings?
My objection basically comes down to:
a) I don’t think it’s really feasible for humans today to write down a set of principles aka constitution in a way that is internally consistent and encompasses everything we want including in an out of distribution world.
b) even if we could, I think asking the AI to evaluate itself using it isn’t very robust at all. It’s much easier to generate plausible text that agrees with vague sounding principles on the way to get your reward rather than any sort of true alignment. It would have the same reward-hacking behavior that you see using RLHF / RLVR scenarios IMO.
I don’t have any better alignment ideas (unfortunately).
I remember in the tail end of 2024, I was thinking—“these AIs are going to come for lonely single men who’ll spend hours addicted to talking to their AI girlfriends.” And of course I wouldn’t be one of those schmucks who spent hours talking to an LLM...
But I also see myself in May 2026 spending a couple of hours every day talking to Claude...I guess it came for me first?
I’m using it a lot recently for summarizing / understanding papers in bio, AI etc. And it feels genuinely useful. But every now and then I read something that seems just a little bit incoherent, especially on more scientific topics.
I’m just wondering if it’s more optimized to make explanations that are vaguely insight-porn oriented and make me feel like I understand something deeper than I actually do. It’s also easy to push back and Claude will just change it’s mind easily and give me another just-as-persuasive sounding argument why it was wrong [1]
It does feel like it’s persuasive capability is rising faster than it’s actual world-modelling capability, and it’s prone to give incorrect or half-truths while sounding very convincing. If those two capabilities keep diverging, I’m worried I might be messing up my own world model somehow by continuing to optimize my learning with AI.
Has anyone else felt this, or have any suggestions?
-----------------
[1] It tried to convince me just now biological neurons might depend on quantum mechanics—without any prompting on my part, and also claimed that this was a “serious scientific debate” and referenced a bunch of papers to support it’s point that on deeper look had nothing to do with what it actually claimed. This was during me researching current tech in BCIs like neuralink etc. And of course, it changed it’s mind pretty quickly when I pushed back.
I see a lot of vibes recently that AI is going to produce these amazing GDP growth rates like 20%+. I wanted to record my
predictioncounter-vibe that I don’t think current GDP statistical methods are good at measuring discontinuous trends like AI.When phones started replacing standalone cameras, we got a reduction in “camera GDP” and a slight increase in phone GDP, but a 100x increase in photography production. GDP doesn’t really measure “amount of photography” happening in a meaningful way. GDP fails at measuring disruptions, but is mostly smoothed over because the whole economy isn’t getting disrupted at the whole time.
Let’s say we have AI replacing human labor across a wide variety of industries. The nominal value of all of it goes from 10T to 1T. The remaining work is high value stuff, or only senior staff managing agents or something similar. Economic surveys would show humans getting the same salaries and the deflator for this doesn’t change. It’s now recorded as 1T as real GDP instead of 10T. I don’t think this is because lump of labor fallacy is suddenly true, but new tasks have to appear that we can actually do better than AI, and enough of it that we can absorb all the existing labor. People are over-anchoring on this happening during the Industrial Revolution and that happened over generations—not a few years.
So I think it’s just as likely for GDP to fall 10% than rise, even if we get a better quality of life. And especially if AI starts disrupting the physical economy with robots etc.
Deepseek R1 could mean reduced VC investments into large LLM training runs. They claim to have done it with ~6M. If there’s a big risk of someone else coming out with a comparable model at 1/10th the cost, then there’s no moat in OpenAI in the long run. I don’t know how much the VC / investors buy the ASI as an end goal and even what the pitch would be. They’re probably looking at more prosaic things like moats and growth rates, and this may mean reduced appetite for further investment instead of more.
What can be done for $6 million, can be done even better with 6 million GPUs[1]. What can be done with 6 million GPUs, can’t be done for $6 million. Giant training systems are the moat.
H/t Gwern.
Yeah, in one sense that makes sense. But also, NVDA is down ~16% today.
And is that correct? Do you expect that to last? My 2021 NVDA purchases still feeling pretty wise right now. :P
Not sure if it’s correct, I didn’t actually short NVDA so all I can do is collect my bayes points. I did expect most investors to think at a first-level thinking as that was my immediate reaction on reading about DeepSeek’s training cost. If models can be duplicated a few weeks / months after they’re out for cheaper, then you don’t have a moat (this is for most regular technologies. I’m not saying AI isn’t different, just that most investors think of this like any other tech innovation)
I am so out of touch with mindset of typical investors that I was taken completely by surprise to see NVDA drop. Thanks for the insight.
No.
This whole SaaSpocalyse scenario outlined here https://www.lesswrong.com/posts/bKrpLhqcoN6WycrFp/citrini-s-scenario-is-a-great-but-deeply-flawed-thought has made me think that one obvious loser in all this is Amazon / AWS
It’s been said that the real money maker for Amazon is AWS and not their retail business.
In fact, the lock-in is so strong that there’s a cottage industry of people with AWS certifications and firms whose sole job is “AWS Cost Optimization”.
But what seems to be not yet priced in is the ease of which anyone with a datacenter can now build an AWS-compatible API in the future.
In the end of the day, amazon is bunch of servers in a datacenter. All the so called “services” are just some syntactic sugar for people that don’t want to manage their own servers—and that’s where their moat lies.
It’s hard for a startup who’s built on top of these services to migrate out to another bare-bones rack in another datacenter , but if the datacenter can give them a compatible API, then moving becomes a click of a button (for the most part).
But if you look at how openai competitors worked, almost everyone has a “openai-compatible” API—all I do is change the URL to new model provider and I’m good to go.
This seems like it would truly kill the AWS lock-in, and it doesn’t seem to be priced in to their stock price at all. Maybe people don’t think of AWS as a SaaS company? I would never myself short a stock, but it does seem like the second-order effect to all this is obviously not priced in at all.
The runtime/data-plane APIs are not the moat. There already exist compatible APIs for at least some of AWS services (S3, DynamoDB), and many others use standard/open APIs (SQL), or very simple APIs (SNS, SQS, Firehose, even Lambda and ECS).
It’s the very deep auth/RBAC mechanisms, the automation of control plane/setup, the integration of the services to use together which are the operational barrier to competion. And the history of durability and availability, and clear guidance as to design considerations for reliability which are the trust barriers to competition. Oh, and there are economies of scale even for datacenter—learning to design, build, and operate them has a pretty steep curve.
The easy part is getting easier. The hard part isn’t (well it is, because AWS provides an example, and because LLMs make everything faster. They make AWS better too, though, and AWS has the people/institutional knowledge to get excellent use of LLMs on these topics.
I’m not saying AWS is immune to competition on core services, only that it won’t be a swarm of startups, it’ll be gradual change of equilibrium with other large providers. That said, for newer services, there’s a lot of room for competition with startups built on AWS, which do the new functions better than AWS does, because they make different tradeoffs, like not being fully compatible with AWS auth/setup/billing/management interfaces, which are by necessity rather complex. Even there, the risk is interesting and probably different from recent history. Previously, small competitors to AWS in areas that AWS wants to get good at just get acquired, and become part of AWS. Now it may be more feasible for AWS to rapidly compete with them and implement AWS-style services that make the startup far less attractive to customers.
[ disclaimer: I have worked for companies related to this topic, and this opinion is not based on anything but my speculation and outside knowledge ]
I think there are 3 ways to think about AI and lot of confusion seems to happen because the different paradigms are talking past each other. The 3 paradigms I see on the internet & when talking to people:
Paradigm A) AI is a new technology like the internet / smartphone / electricity—this seems to be mostly held by VC’s / enterpreneurs / devs that think this will unlock a whole new set of apps like AI:new apps like smartphone:Uber or internet:Amazon
Paradigm B) AI is a step change in how humanity will work. Similarly to the agricultural revolution that led to the change in how large society could get and GDP growth, and the industrial revolution was a step-change in GDP growth from ~0% to 2-4% a year, and made things possible such as electricity and the internet and smartphones.
Paradigm C) AI is like the rise of humanity on this earth (the first general intelligences). The world changed completely with the rise of GI, and ASI/AGI will be a similar paradigm. We’ve been locked at humanity’s level of intelligence for the past ~200k years, and getting ASI will be like unlocking multiple new revolutions all at the same time.
Most of the LW crowd is probably (C) or between (B) and (C)
When talking to the general population, I’ve found it to be very helpful to probe about where they are before talking about things like AI safety / how the world will change.
If RL becomes the next thing in improving LLM capabilities, one thing that I would bet on becoming big is computer-use in 2025. Seems hard to get more intelligence with just RL (who verifies the outputs?), but with something like computer use, it’s easy to verify if a task has been done (has the email been sent, ticket been booked etc..) that it’s starting to look to more to me like it can do self-learning.
One thing that’s left AI still fully not integrated into the rest of the economy is simply that the current interfaces were built for humans and moving all those takes engineering time / effort etc.
I’m fairly sure the economic disruption would be pretty quick once this happens. For example, I can just run 10 LLM agents to act as customer service agents using my *existing tools* - just open emails, whatsapp, and message customers, check internal dashboards etc., then it’s game over. What’s stopping people right now is that there’s not enough people to build that pipeline fast enough to utilize even the current capabilities.
Just finished reading “If Anyone Builds It, Everyone Dies”. I had a question that seems like an obvious one, but one I didn’t see addressed in the book, maybe someone can help:
The main argument in the book is the analogy to humans. Evolution “wanted” us to maximize genetic fitness, but it didn’t get what it trained for. Instead, it created humans who love ice cream and condoms even though they reduce our genetic fitness.
With AGI, we’re on track to do something similar—we won’t get an AI aligned to human interests even though we do RLHF or any other such simple training or shaping to an AI, it’ll end up wanting something weird and inhuman rather than maximizing human values.
But in my mind, this seems to miss a fairly important point: The fact that human brains don’t come pre-wired with much knowledge. We have to learn it from scratch. We don’t come out of the womb with concept of “inclusive genetic fitness”. It took us culture and ~200,000 years to figure that out, and we still only learn it after about 15-20 years of existing. So there’s no way that evolution could have made us point our utility function to “inclusive genetic fitness” because that concept doesn’t exist in our brains.
Modern AIs don’t seem like that. They come with the sum of human knowledge baked in during pre-training. As they get smarter, the concept of “human values” or “friendly AI” is definitely something in it’s existing mind. So it should be much easier for us to do alignement and test whether we can point it to that specific concept vs. what what evolution had.
Knowing about “inclusive genetic fitness” does not stop you from wanting ice cream.
For superhuman AIs, knowing about human values won’t necessarily make them care.
Yes, I agree with that. I’m not claiming that knowing about it stops you from wanting ice cream.
I’m claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want “inclusive genetic fitness” rather than wanting ice cream.
i.e—we wouldn’t want ice cream at all but reason from first principles what we should eat based on fitness.