Tom Davidson

Karma: 1,670

Tom Davidson 27 Mar 2026 8:53 UTC
2 points
−1
in reply to: habryka’s comment on: AI character is a big deal
I disagree—I think we just use it for the broader thing. What sentences do we use it to refer to literary character?
I’m particularly surprised that you say the post equivocates, given that you seem to understand me when I reframe the post in terms of the alignment target.

Tom Davidson 27 Mar 2026 8:51 UTC
4 points
−2
in reply to: habryka’s comment on: AI character is a big deal
Then it has a truly terrible title and I will gladly take bets at 5:1 odds that the primary context other people, including Forethought people, will link to this will be in discussion about things like the Anthropic Constitution, or other approaches centrally determined around choosing the literary character that AIs are aiming to imitate.
Yes, your reaction and the reaction of other commenters here certainly show the title is very confusing/misleading for at least some readers. I don’t know how common that reaction is. It’s not something I saw in others who commented on the draft, but we didn’t seek out “classic LW doomer” types and in hindsight we should have.
Maybe a better name for this type of work than “AI character” would be “the alignment target”, that’s a useful update
Why do you think the Anthropic Constitution and OAI model spec are just about choosing AI’s literary character? They are describing the whole alignment target, not just the “literary character” parts of it. So yes, I agree other ppl will associate this post with things like Anthropic Constitution, but disagree that means they’ve misunderstood. And I totally think LWers who think about the alignment target should be weighing in about what these docs say, not treating them as unimportant.
On the positive side, it seems like you correctly understood my most recent attempt to explain the content of this post.

Tom Davidson 26 Mar 2026 8:51 UTC
2 points
0
in reply to: RobertM’s comment on: AI character is a big deal
why incomplete?

Tom Davidson 26 Mar 2026 8:51 UTC
2 points
0
in reply to: habryka’s comment on: AI character is a big deal
Less important: TBC, it’s really not about “literary character”. I think I was quite clear in my reply to J Bostom that we are indeed using AI character in a broad way to refer to “the alignment target”. If AI generalises in quite different ways to humans, then I would just say that AI character is different to any human character. But if the way that it generalises is predictable and not crazy complex, I think it will still be important to think about it, and I think that we could likely still use human language to describe it.

Tom Davidson 26 Mar 2026 8:51 UTC
7 points
0
in reply to: habryka’s comment on: AI character is a big deal
Hmm, i’m surprised we’re having so much trouble understanding each other here. Thanks for bearing with me!
Let me just try a new way of framing things and see if it helps.
This post is basically saying “more ppl should think about what the alignment target should be”.
You point out that “A lot of people are thinking about the big picture of how to steer/control/align AIs, including how to use AI systems to align future AI systems.”
I agree with that! And some of the time they are thinking about what the alignment target should be. (Though they’re mostly thinking about how to actually obtain a given alignment target giving risks of misalignment, not about what the target should be.) But their key lens for “what should the alignment target be” is “what alignment target will keep humanity in control of powerful AI, all the way to superintelligence”.
Relative to this, the main new contribution of this post is to suggest a new lens for thinking about what the alignment target should be. In short, a lens that takes into account the risks of coups and the upside possibility of making the future amazing conditional on humans staying in control.^[1] We’re claiming that the choice of alignment target could have big effects on both of these things, and that people haven’t thought much about that lens.
Does that make sense?
1. ^
  As I write this, I wonder if you are sceptical that coups would be bad because humans still stay in control, and also sceptical that humans might stay in control but there would still be large differences in how good the future is? If so, that might explain why this is unintuitive to you.

Tom Davidson 25 Mar 2026 8:39 UTC
3 points
0
in reply to: RobertM’s comment on: AI character is a big deal
I’m much less interested in litigating the text of the piece than in clarifying my views and seeing if you think I’m being unreasonable, so I’m going to focus on that.
I think that before we reach full superintelligence, AI will have massive effects on the world, and this will shape the countries and companies and individuals and AI systems that themselves develop super-intelligence.
I expect that most of the impact of today’s work on AI character will come by influencing this intermediate period, which will in turn influence the design of superintelligence itself and other aspects of the world at that time, - for example, whether it is just controlled by one autocratic leader or not.
To be clear, I don’t agree with the claim that everything routes through the specific values or character design of the first super-intelligent systems. I think it will matter who controls them, the institutions that exist at the time, how broad access to AI is, how the AI market is structured (one vs many AI companies), whether structured transparency increases trust between different actors and reduces incentives to defect, people’s epistemics and coordination tech more broadly, and many other complicated things. (Just like it is a massive oversimplification to say that the only thing that matters today is the values that individual humans have, rather than acknowledging the many other complicated norms, infrastructure, laws, etc., that bind everything together.) So when I say that everything routes through the transition to superintelligence, I don’t mean one specific system. I understand the transition as a broad thing that society as a whole is going through. (Though of course I agree that the character of superintelligent AI will be extremely important.)
I think this is especially clear if AI does not take over: what happens will depend on what people want and how society is structured.
So the examples in the text are mostly about AI character shaping this intermediate period in a broad way, which then shapes the transition to super intelligent AI in a broad way.
Another analogy you could draw is how the events of the last two decades matter a lot because of how they are shaping the transition that is happening today.
I’ll give one more example that may be more convincing to you. If AI character is such that people tend to trust AI and also AI character is such that AI tells the truth to people about difficult topics, then AI will tell the truth about how large misalignment risk is. People will believe it, and this will increase the effort that goes into misalignment risk. More generally, AI can boost society’s epistemics and coordination ability so that society is less insane and able to coordinate on a pause.
Separately to all the above, I think there could be some additional impact from work on AI character, where it sets a precedent for AI character that the public expects to persist, or that AI companies lazily stick with, and this affects the character of super intelligent systems. (I called this the “direct” route in my previous comment to you, but I noticed that in the post we call it “path dependent”, I guess because it is going via affecting the character of intermediate AI systems.)
What links here?
- Tom_Davidson's comment on AI character is a big deal by Forethought (EA Forum; 25 Mar 2026 8:46 UTC; 4 points)

Tom Davidson 25 Mar 2026 8:21 UTC
1 point
−8
in reply to: RobertM’s comment on: AI character is a big deal
Okay, I think I can agree with you that it would have been better for our post to explicitly disclaim that if alignment is very hard, then this work is less valuable. The value of this work depends on there being some predictable link between the character A and the one we get.
(Though honestly, I think there is a decent case to be made, even from a very pessimistic perspective, for thinking more about trying to affect the world as AI does take over.)
I pretty strongly disagree with you if you think that every post on a topic like this has to be really doomy and apologetic about focusing on anything other than misalignment work or pausing AI.
On the other hand, this level of confidence would be pretty controversial even among the AI lab employees working on these techniques.
Not sure what level of confidence you’re referring to. I’m personally at about 10% for misaligned AI takeover. I don’t think the post comes across as highly confident that AI alignment will be solved. We explicitly discuss that the work can be valuable if AI does take over, acknowledging that possibility.

Tom Davidson 25 Mar 2026 8:16 UTC
2 points
0
in reply to: habryka’s comment on: AI character is a big deal
I think you know what I mean? Very few ppl are thinking systematically about the big picture of what character AI should have. E.g. should AI follow instructions to create secretly loyal AI? Should it proactively report to ppl if someone’s trying to do this? Should it sabotage them? If it can avert a catastrophe by temporarily seizing some fairly large amount of power, should it? Is Anthropic’s current approach right, vs OAI’s, vs hardcore corrigibility

Tom Davidson 24 Mar 2026 17:48 UTC
4 points
0
in reply to: RobertM’s comment on: AI character is a big deal
Re your point 2, as you say, there’s a lot to discuss here. It might not be tractable to resolve, but let me try and say a few quick things.
“Might be easier to hit as an alignment target”
You’re providing links to arguments that alignment is very hard. We’re not trying to sweep aside or ignore this literature. (Though I’ll that I don’t think that those arguments at all decisive. I think it’s very unclear how hard alignment will ultimately be.) I think it’s plausible that certain alignment targets, if we aim for them, would be less likely to result in AI takeover. As you say, corrigibility is a classic example given here. This just seems like a really important consideration to keep in mind when choosing what AI character AI developers should be aiming for. I think it would be a mistake to take the arguments for alignment being difficult that you’ve given and then give up on choosing an alignment target that is less likely to result in AI takeover in practice. Even if you’re pretty pessimistic, it seems like this kind of thing could reduce the chance of AI takeover by multiple percentage points, from 95% down to 90%. But I wonder if we’re talking past each other here.
“Might yield safe AI even if only partially hit” - myopia seems both technically difficult and actively counter the economic incentives of the AI labs, value is fragile, etc.
Object level: agree in terms of the economic incentives, but sceptical about the other two claims. Meta: my point is just that these types of argument should be taken into account when choosing AI character.
For these two points so far, I want to be clear that they are not the main subject of the post. As you say, there’s a lot of literature on them already because people have mostly thought about AI catastrophe from the perspective of how to avoid AI take over. That’s why we did not emphasise them.
“Might produce AI that cooperates even if misaligned”
I think your objections here are way too quick, but it’s a lot to get into. Have you read this post explaining the idea of deals? The most convincing case study is when the AI cannot easily take over. It maybe has a 1% (or 0.01%) chance of taking over, and therefore it’s relatively cheap to incentivize it to cooperate with humans. I also think you can potentially make AI risk-averse with respect to the resources it personally controls without being risk-averse with respect to how much progress it makes in its work.
Zooming out from all of the above, I get the vibe from your comments that maybe you just don’t think people should work on anything other than misalignment, given how high your probability of AI takeover is. Is that right? To me it seems very reasonable for people to work on AI character under the assumption that alignment will be not too difficult, because I think that’s a real possibility that we should put significant credence in.

Tom Davidson 24 Mar 2026 16:30 UTC
4 points
0
in reply to: J Bostock’s comment on: AI character is a big deal
Thanks for the engagement!
Yep, we’re using “character” in a broad sense to mean the stable behaviourial dispositions of AI.
I really don’t think “utility function” would be a better word here. That word connotes that the AI will optimise hard for some well specified criteria. Similarly “goals” implies that AI will be consequentialist in a looser sense, that we don’t want to assume. I’m more sympathetic to “values”, but even that connotes more of a consequentialist frame than I want to assume here, and “character” seems better.
We could have used “AI behaviourial dispositions” instead of “AI character”, but it’s less catchy.
1. The ways in which AIs act will be important
  … [this] claim seems somewhat trivial to me
Yep, I’m sympathetic to this being pretty trivial! But when writing this post we got sceptical pushback along the lines of the objections we discuss. And others (e.g. Robert M’s comment) object based on AI alignment being hard so that designing AI character for worlds where alignment succeeds isn’t worth doing—a position which I think is misguided. Most importantly, very very few people are thinking about how to design AI character. Most of the historical discussion has focussed on choosing an alignment target that makes AI takeover less likely (e.g. corrigibility). There’s some old discussion of CEV, but I think CEV is unsatisfactory for a many reasons. Currently it’s just a few ppl at each leading AI company writing their model spec documents. That’s nowhere near the amount of effort this problem deserves!

Tom Davidson 24 Mar 2026 16:30 UTC
4 points
0
in reply to: RobertM’s comment on: AI character is a big deal
The argument so far has been about the effect of AI character up to the point of superintelligence. That’s where we think most of the expected impact is.

As I mentioned a few months ago in response to Effective altruism in the age of AGI, believing in ASI is kind of a “totalizing” belief. If you take ASI seriously, then “most of the expected impact” of , where is anything that might affect how the transition to ASI goes, is in that effect.
Many of the “Pathways to impact” are the kinds of things that might affect whether our transition to a post-ASI future goes well or not. But the way you phrased the first sentence of section 1.2 suggests that you’re thinking of those pathways to impact as mattering mostly in worlds where we don’t transition to a post-ASI future?^[1] If so, this seems backwards to me.
Hmm, there might be a miscommunication here. I agree that most of the expected impact of AI character work flows through the transition to ASI. The claim we’re making in the sentences you quote is (roughly) that it’s hard to directly affect the character of superintelligence itself, as our work will be washed out be work done by slightly superhuman AIs. So we think most impact here flows through influencing the character of slightly superhuman AI, but this itself has impact via influencing the transition to ASI. So what we’re saying it compatable with thinking that everything that matters has impact by influencing the transition to superintelligence.
Does that clarify?
1. It seems to mostly be taking for granted a question that is the subject of much dispute—whether we will be able to meaningfully solve for whatever parts of the alignment problem need to be solved to robustly instill specific values into these AIs.
Don’t agree on this. Yes, you won’t be interested in this work if you think alignment is so hard that the intended alignment target has no predictable affect on AI character. But that’s a very extreme and pessimistic view on alignment. Maybe that’s your view? Here are some paths to impact for this work that rely on slightly more optimistic assumptions:
- There’s s 20% that we solve alignment. In those worlds, the alignment target (the AI character we aimed for) has a massive impact on how good the future is. p(ai takeover)=80% is a pretty extreme doomer view, and it still leaves lots of value for this work.
- We don’t solve alignment but there’s still some predictable relationship between the alignment target we were aiming for and the resultant AI’s character. By choosing a better alignment target, we improve the worlds conditional on AI takeover.
So i’m really not seeing that we’re “taking an answer for granted” vs just having moderate credence on alignment being solvable or (if not) the alignment target have some preditable influence on AI character.

(I’ll reply to your point 2 in another comment!)

AI character is a big deal

wdmacaskill and Tom Davidson

23 Mar 2026 16:36 UTC

34 points

27 comments12 min readLW link

(www.forethought.org)

Tom Davidson 18 Mar 2026 10:03 UTC
2 points
0
in reply to: Ihor Kendiukhov’s comment on: On The Independence Axiom
But I don’t think EE would give different answers! EE would give the answer that would maximise the time-averaged geometric growth rate. That would be the same answer that would maximise the expected final log wealth after a long series of identical decisions.

Tom Davidson 18 Mar 2026 9:54 UTC
2 points
0
in reply to: Ihor Kendiukhov’s comment on: On The Independence Axiom
Thanks for the detailed reply! I want to push back on the claim that the EE agent and the log-utility maximizer make different choices

You say that in the additive case, the EE agent is exactly risk-neutral while the log-utility maximizer remains risk-averse. But consider what a log-utility maximizer actually does when facing a long sequence of independent additive bets.

For a single additive bet with payoff x, the contribution to log wealth is ln(W + x) - ln(W) ≈ x/W for small x relative to W. So the log-utility maximizer treats each bet approximately linearly. And as the number of independent additive bets grows, each individual bet becomes smaller relative to total wealth (because wealth is growing via the accumulated positive-EV bets), making the linear approximation increasingly exact.

In the limit of infinitely many independent additive bets, the log-utility maximizer’s per-bet behavior converges to exact risk-neutrality — which is exactly what EE prescribes. So in the regime where the EE prescription is most cleanly motivated (many repeated bets, which is the whole setup for time-average reasoning), the two agents converge.

Where they diverge is for large additive bets relative to current wealth. But here I think the log-utility maximizer is actually more faithful to the “maximize time-average growth rate” objective than the EE agent is. If you’re risk-neutral about an additive bet that could cost you half your wealth, you’re exposing yourself to ruin risk, which destroys your time-average growth rate. The identity mapping tells you the additive process is ergodic, but being ergodic doesn’t mean you should be risk-neutral about large bets within it — not if your goal is to maximize the growth rate of your own single trajectory.

So it seems to me that either (a) the bets are small relative to wealth, and the log-utility maximizer behaves (approximately, converging to exactly) like the EE agent, or (b) the bets are large relative to wealth, and the log-utility maximizer is arguably more correct about maximizing time-average growth. In neither case do the two agents clearly diverge in a way that supports the independence violation argument.

What am I missing?

If I’m right then again it doesn’t seem like the EE agent is really violating independence, given that its behaviour is replicated by a vnm agent that just values its final log wealth

Tom Davidson 16 Mar 2026 17:55 UTC
LW: 2 AF: 1
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
Thanks, that’s helpful.

What seems clear to me is that our world is the result of fairly simple laws of physics, and our creators wanted to know how those simple laws would play out. They’re saying “if there was a universe with these laws, what would happen”. (This is what I’d meant by “simulation”)

I agree it’s less clear that they’re doing this bc they think those laws also describe a real-world process (somewhere in the multiverse) and they want to predict the outcome of that process. (This is what you meant by “simulation” and I think your def is better.)

So I understand where you’re coming from better now. Thanks!

But I still think we’re in a simulation, in your stronger sense of the word! Why? Bc:
1. other civs will reasonably believe our laws of physics describe part of the multiverse,
2. this gives them a strong instrumental reason to simulate this,
3. absent 1 and 2 there aren’t comparably strong reasons to run vivariums like our world.

Tom Davidson 15 Mar 2026 10:45 UTC
15 points
2
on: On The Independence Axiom
Nice post!
I want to push on one thing, though. I’m sceptical of the claim that the ergodicity economics agent violates independence.
As I understand it, the EE agent has a fixed objective: maximize the time-average growth rate of wealth, which is equivalent to maximizing expected log terminal wealth. When the stochastic environment changes — say from multiplicative to additive dynamics — the optimal per-bet policy changes. In the multiplicative case, you Kelly-bet (which looks like log utility applied locally). In the additive case with many independent bets, you behave roughly linearly with each bet (because log is approximately linear for small additive increments relative to total wealth).
But is this actually a violation of independence? Independence says: if you prefer lottery A to lottery B, then mixing both with a common lottery C at the same probability shouldn’t reverse that preference. It’s a constraint on your ranking of probability distributions over outcomes.
What the EE agent is doing seems different. They have a fixed preference over (distributions over) outcomes (log terminal wealth, or equivalently, time-average growth rate). When the dynamics change, the mapping from available actions to outcome distributions changes, so the optimal action changes. But the preference ordering over final outcomes hasn’t changed — the agent still prefers higher log wealth to lower log wealth. It’s the decision problem that’s different, not the preferences.
To put it another way: an EU maximizer with log utility would make exactly the same choices as the EE agent in every case you describe. They’d Kelly-bet in multiplicative environments and behave more linearly in additive ones, because that’s what maximizing expected log wealth requires in each setting. But the EU maximizer with log utility satisfies independence by construction. So how can the EE agent be violating independence while making identical choices?
I think the thing that looks like a context-dependent utility function is really a context-dependent policy derived from a fixed utility function under different dynamics. These seem importantly different, and I’m not sure the independence axiom is violated by the latter.

Tom Davidson 12 Mar 2026 11:15 UTC
LW: 4 AF: 1
2
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
Ok but if we’re on a computer then isn’t it clear we’re a simulation, not a vivarium, bc clearly it was designed to simulate the behavior of a pre-agi civ?

Tom Davidson 6 Mar 2026 11:12 UTC
LW: 5 AF: 3
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
In our case, if we are ultimately running on a computer, then wouldn’t that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization.

Could we be running on a computer and also be a vivarium?

Tom Davidson 6 Mar 2026 11:10 UTC
LW: 4 AF: 3
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
But is there any active selection for that amongst civilizations who survive and grow?

Tom Davidson 6 Mar 2026 1:48 UTC
LW: 4 AF: 3
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal

Personally I think the world as we know it is more likely to be in a vivarium than a simulation,

Why?

Tom Davidson

AI char­ac­ter is a big deal

AI character is a big deal