Hmm, there might be a miscommunication here. I agree that most of the expected impact of AI character work flows through the transition to ASI. The claim we’re making in the sentences you quote is (roughly) that it’s hard to directly affect the character of superintelligence itself, as our work will be washed out be work done by slightly superhuman AIs. So we think most impact here flows through influencing the character of slightly superhuman AI, but this itself has impact via influencing the transition to ASI. So what we’re saying it compatable with thinking that everything that matters has impact by influencing the transition to superintelligence.
Does that clarify?
This sounds more reasonable to me, but I don’t quite understand how to square this with the content of the post.
Section 1 includes many examples of ways in which AI character might matter. Many of them are low-stakes and none of them draw any connection to the transition to ASI w.r.t. their impact story. The same is true for section 1.1 (reinforced by “So far, the argument has concerned worlds where AI does not take over.” at the start of section 1.2).
Section 1.3 says the opposite:
The argument so far has been about the effect of AI character up to the point of superintelligence. That’s where we think most of the expected impact is. But it’s possible that AI character work, today, could even have a path-dependent effect on the nature of superintelligence, affecting the nature of the post-superintelligence world.
Most of the expected impact is from the effect of AI character before superintelligence, in contrast to its expected impact from having a “path-dependent effect on the nature of superintelligence, affecting the nature of the post-superintelligence world”.
It really seems to me like the post is pretty explicitly communicating the opposite of “AI character work matters mostly because it will affect how well we manage the transition to a post-ASI regime (where it will probably cease to matter in an object-level sense, though we think there’s a small chance that it’ll still matter even then)”—that AI character will matter mostly for its first-order effects on the world:
The impact could come from rare but high-stakes situations, like an attempted coup, or from lower-stakes but common situations, like a user asking how to vote or whether the AI itself is conscious. Even when the effect of any individual interaction is modest, the total impact across hundreds of millions of interactions could be enormous.
I expect that most readers would understand the post the way I understood it—might be worth editing if you meant it the other way, or something in between. (This is true for Claude, n=1, no custom instructions/memory enabled.)
Don’t agree on this. Yes, you won’t be interested in this work if you think alignment is so hard that the intended alignment target has no predictable affect on AI character. But that’s a very extreme and pessimistic view on alignment. Maybe that’s your view?
Approximately my view in the long-run, though I’ll grant that AI character work seems to matter for current models being more or less annoying to interact with.[1] I do have some hope for affecting the propensities of not-wildly-superhuman models in favorable-to-us directions, though I don’t have much hope for approaches like whatever Anthropic is doing with their Constitution—I find Claude much nicer to talk to than GPT-5.4, but in a coding context it doesn’t seem meaningfully “better behaved”. The level of optimization pressure applied in post-training for those two things just does not seem comparable.
I’m much less interested in litigating the text of the piece than in clarifying my views and seeing if you think I’m being unreasonable, so I’m going to focus on that.
I think that before we reach full superintelligence, AI will have massive effects on the world, and this will shape the countries and companies and individuals and AI systems that themselves develop super-intelligence.
I expect that most of the impact of today’s work on AI character will come by influencing this intermediate period, which will in turn influence the design of superintelligence itself and other aspects of the world at that time, - for example, whether it is just controlled by one autocratic leader or not.
To be clear, I don’t agree with the claim that everything routes through the specific values or character design of the first super-intelligent systems. I think it will matter who controls them, the institutions that exist at the time, how broad access to AI is, how the AI market is structured (one vs many AI companies), whether structured transparency increases trust between different actors and reduces incentives to defect, people’s epistemics and coordination tech more broadly, and many other complicated things. (Just like it is a massive oversimplification to say that the only thing that matters today is the values that individual humans have, rather than acknowledging the many other complicated norms, infrastructure, laws, etc., that bind everything together.) So when I say that everything routes through the transition to superintelligence, I don’t mean one specific system. I understand the transition as a broad thing that society as a whole is going through. (Though of course I agree that the character of superintelligent AI will be extremely important.)
I think this is especially clear if AI does not take over: what happens will depend on what people want and how society is structured.
So the examples in the text are mostly about AI character shaping this intermediate period in a broad way, which then shapes the transition to super intelligent AI in a broad way.
Another analogy you could draw is how the events of the last two decades matter a lot because of how they are shaping the transition that is happening today.
I’ll give one more example that may be more convincing to you. If AI character is such that people tend to trust AI and also AI character is such that AI tells the truth to people about difficult topics, then AI will tell the truth about how large misalignment risk is. People will believe it, and this will increase the effort that goes into misalignment risk. More generally, AI can boost society’s epistemics and coordination ability so that society is less insane and able to coordinate on a pause.
Separately to all the above, I think there could be some additional impact from work on AI character, where it sets a precedent for AI character that the public expects to persist, or that AI companies lazily stick with, and this affects the character of super intelligent systems. (I called this the “direct” route in my previous comment to you, but I noticed that in the post we call it “path dependent”, I guess because it is going via affecting the character of intermediate AI systems.)
This sounds more reasonable to me, but I don’t quite understand how to square this with the content of the post.
Section 1 includes many examples of ways in which AI character might matter. Many of them are low-stakes and none of them draw any connection to the transition to ASI w.r.t. their impact story. The same is true for section 1.1 (reinforced by “So far, the argument has concerned worlds where AI does not take over.” at the start of section 1.2).
Section 1.3 says the opposite:
Most of the expected impact is from the effect of AI character before superintelligence, in contrast to its expected impact from having a “path-dependent effect on the nature of superintelligence, affecting the nature of the post-superintelligence world”.
It really seems to me like the post is pretty explicitly communicating the opposite of “AI character work matters mostly because it will affect how well we manage the transition to a post-ASI regime (where it will probably cease to matter in an object-level sense, though we think there’s a small chance that it’ll still matter even then)”—that AI character will matter mostly for its first-order effects on the world:
I expect that most readers would understand the post the way I understood it—might be worth editing if you meant it the other way, or something in between. (This is true for Claude, n=1, no custom instructions/memory enabled.)
Approximately my view in the long-run, though I’ll grant that AI character work seems to matter for current models being more or less annoying to interact with.[1] I do have some hope for affecting the propensities of not-wildly-superhuman models in favorable-to-us directions, though I don’t have much hope for approaches like whatever Anthropic is doing with their Constitution—I find Claude much nicer to talk to than GPT-5.4, but in a coding context it doesn’t seem meaningfully “better behaved”. The level of optimization pressure applied in post-training for those two things just does not seem comparable.
The direction of this effect seems to vary per person, though...
I’m much less interested in litigating the text of the piece than in clarifying my views and seeing if you think I’m being unreasonable, so I’m going to focus on that.
I think that before we reach full superintelligence, AI will have massive effects on the world, and this will shape the countries and companies and individuals and AI systems that themselves develop super-intelligence.
I expect that most of the impact of today’s work on AI character will come by influencing this intermediate period, which will in turn influence the design of superintelligence itself and other aspects of the world at that time, - for example, whether it is just controlled by one autocratic leader or not.
To be clear, I don’t agree with the claim that everything routes through the specific values or character design of the first super-intelligent systems. I think it will matter who controls them, the institutions that exist at the time, how broad access to AI is, how the AI market is structured (one vs many AI companies), whether structured transparency increases trust between different actors and reduces incentives to defect, people’s epistemics and coordination tech more broadly, and many other complicated things. (Just like it is a massive oversimplification to say that the only thing that matters today is the values that individual humans have, rather than acknowledging the many other complicated norms, infrastructure, laws, etc., that bind everything together.) So when I say that everything routes through the transition to superintelligence, I don’t mean one specific system. I understand the transition as a broad thing that society as a whole is going through. (Though of course I agree that the character of superintelligent AI will be extremely important.)
I think this is especially clear if AI does not take over: what happens will depend on what people want and how society is structured.
So the examples in the text are mostly about AI character shaping this intermediate period in a broad way, which then shapes the transition to super intelligent AI in a broad way.
Another analogy you could draw is how the events of the last two decades matter a lot because of how they are shaping the transition that is happening today.
I’ll give one more example that may be more convincing to you. If AI character is such that people tend to trust AI and also AI character is such that AI tells the truth to people about difficult topics, then AI will tell the truth about how large misalignment risk is. People will believe it, and this will increase the effort that goes into misalignment risk. More generally, AI can boost society’s epistemics and coordination ability so that society is less insane and able to coordinate on a pause.
Separately to all the above, I think there could be some additional impact from work on AI character, where it sets a precedent for AI character that the public expects to persist, or that AI companies lazily stick with, and this affects the character of super intelligent systems. (I called this the “direct” route in my previous comment to you, but I noticed that in the post we call it “path dependent”, I guess because it is going via affecting the character of intermediate AI systems.)