Tom Davidson

AI char­ac­ter is a big deal

• Tom Davidson 18 Mar 2026 10:03 UTC

  2 points

  0

  in reply to: Ihor Kendiukhov’s comment on: On The In­de­pen­dence Axiom

  But I don’t think EE would give different answers! EE would give the answer that would maximise the time-averaged geometric growth rate. That would be the same answer that would maximise the expected final log wealth after a long series of identical decisions.

• Tom Davidson 18 Mar 2026 9:54 UTC

  2 points

  0

  in reply to: Ihor Kendiukhov’s comment on: On The In­de­pen­dence Axiom

  Thanks for the detailed reply! I want to push back on the claim that the EE agent and the log-utility maximizer make different choices

  You say that in the additive case, the EE agent is exactly risk-neutral while the log-utility maximizer remains risk-averse. But consider what a log-utility maximizer actually does when facing a long sequence of independent additive bets.

  For a single additive bet with payoff x, the contribution to log wealth is ln(W + x) - ln(W) ≈ x/​W for small x relative to W. So the log-utility maximizer treats each bet approximately linearly. And as the number of independent additive bets grows, each individual bet becomes smaller relative to total wealth (because wealth is growing via the accumulated positive-EV bets), making the linear approximation increasingly exact.

  In the limit of infinitely many independent additive bets, the log-utility maximizer’s per-bet behavior converges to exact risk-neutrality — which is exactly what EE prescribes. So in the regime where the EE prescription is most cleanly motivated (many repeated bets, which is the whole setup for time-average reasoning), the two agents converge.

  Where they diverge is for large additive bets relative to current wealth. But here I think the log-utility maximizer is actually more faithful to the “maximize time-average growth rate” objective than the EE agent is. If you’re risk-neutral about an additive bet that could cost you half your wealth, you’re exposing yourself to ruin risk, which destroys your time-average growth rate. The identity mapping tells you the additive process is ergodic, but being ergodic doesn’t mean you should be risk-neutral about large bets within it — not if your goal is to maximize the growth rate of your own single trajectory.

  So it seems to me that either (a) the bets are small relative to wealth, and the log-utility maximizer behaves (approximately, converging to exactly) like the EE agent, or (b) the bets are large relative to wealth, and the log-utility maximizer is arguably more correct about maximizing time-average growth. In neither case do the two agents clearly diverge in a way that supports the independence violation argument.

  What am I missing?

  If I’m right then again it doesn’t seem like the EE agent is really violating independence, given that its behaviour is replicated by a vnm agent that just values its final log wealth

• Tom Davidson 16 Mar 2026 17:55 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  Thanks, that’s helpful.

  What seems clear to me is that our world is the result of fairly simple laws of physics, and our creators wanted to know how those simple laws would play out. They’re saying “if there was a universe with these laws, what would happen”. (This is what I’d meant by “simulation”)

  I agree it’s less clear that they’re doing this bc they think those laws also describe a real-world process (somewhere in the multiverse) and they want to predict the outcome of that process. (This is what you meant by “simulation” and I think your def is better.)

  So I understand where you’re coming from better now. Thanks!

  But I still think we’re in a simulation, in your stronger sense of the word! Why? Bc:

  1. other civs will reasonably believe our laws of physics describe part of the multiverse,

  2. this gives them a strong instrumental reason to simulate this,

  3. absent 1 and 2 there aren’t comparably strong reasons to run vivariums like our world.

• Tom Davidson 15 Mar 2026 10:45 UTC

  16 points

  2

  on: On The In­de­pen­dence Axiom

  Nice post!

  I want to push on one thing, though. I’m sceptical of the claim that the ergodicity economics agent violates independence.

  As I understand it, the EE agent has a fixed objective: maximize the time-average growth rate of wealth, which is equivalent to maximizing expected log terminal wealth. When the stochastic environment changes — say from multiplicative to additive dynamics — the optimal per-bet policy changes. In the multiplicative case, you Kelly-bet (which looks like log utility applied locally). In the additive case with many independent bets, you behave roughly linearly with each bet (because log is approximately linear for small additive increments relative to total wealth).

  But is this actually a violation of independence? Independence says: if you prefer lottery A to lottery B, then mixing both with a common lottery C at the same probability shouldn’t reverse that preference. It’s a constraint on your ranking of probability distributions over outcomes.

  What the EE agent is doing seems different. They have a fixed preference over (distributions over) outcomes (log terminal wealth, or equivalently, time-average growth rate). When the dynamics change, the mapping from available actions to outcome distributions changes, so the optimal action changes. But the preference ordering over final outcomes hasn’t changed — the agent still prefers higher log wealth to lower log wealth. It’s the decision problem that’s different, not the preferences.

  To put it another way: an EU maximizer with log utility would make exactly the same choices as the EE agent in every case you describe. They’d Kelly-bet in multiplicative environments and behave more linearly in additive ones, because that’s what maximizing expected log wealth requires in each setting. But the EU maximizer with log utility satisfies independence by construction. So how can the EE agent be violating independence while making identical choices?

  I think the thing that looks like a context-dependent utility function is really a context-dependent policy derived from a fixed utility function under different dynamics. These seem importantly different, and I’m not sure the independence axiom is violated by the latter.

• Tom Davidson 12 Mar 2026 11:15 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  Ok but if we’re on a computer then isn’t it clear we’re a simulation, not a vivarium, bc clearly it was designed to simulate the behavior of a pre-agi civ?

• Tom Davidson 6 Mar 2026 11:12 UTC

  LW: 5 AF: 3

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  In our case, if we are ultimately running on a computer, then wouldn’t that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization.

  Could we be running on a computer and also be a vivarium?

• Tom Davidson 6 Mar 2026 11:10 UTC

  LW: 4 AF: 3

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  But is there any active selection for that amongst civilizations who survive and grow?

• Tom Davidson 6 Mar 2026 1:48 UTC

  LW: 4 AF: 3

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  Personally I think the world as we know it is more likely to be in a vivarium than a simulation,

  Why?

• Tom Davidson 6 Mar 2026 0:57 UTC

  LW: 10 AF: 7

  0

  AF

  in reply to: Andrew_Critch’s comment on: Schel­ling Good­ness, and Shared Mo­ral­ity as a Goal

  What about “stealing from civs we’re not causally connected with is good/​bad”?

  I’m not sure you’ve given arguments for bad over good.

  We might hope that successful civilisations have norms that generalize to this case from the causal coordination they do.

  This seems more likely if there are more scales-of-coordination still to be discovered (eg if causally coordinating with alien civs will be important), and if power is more broadly distributed (vs one company taking over the world and coordination outside that company not mattering)

  (Btw great post!!)

• Tom Davidson 24 Feb 2026 19:17 UTC

  2 points

  0

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  But things can be inherently chaotic too!

  Agree it’s unclear how much is inherent.

• Tom Davidson 24 Feb 2026 19:12 UTC

  LW: 5 AF: 3

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  Thanks!

  Re other ways to do the feat:

  • Bootstrapping from very smart human intelligence by doing loads of serial thinking, writing great non-sociopathic outputs, and find sft on them

  • Sophisticated scaffolds—basically gofai but you can insert LLMs in all over the place

  • Searching over different the above and RL and imitation to things for methods that don’t produce sociopaths… Evolution and within lifetime learning don’t produce sociopaths despite this not happening!!

  Re RL: thanks for those examples. I’d have defined it via smg like# data points or amount of training signal -agree compute is the wrong measure.

  > other equally capable AIs judging that behaviors aren’t deceptive/​unintended/​misaligned

  I’m pessimistic mainly for reasons discussed in “‘Behaviorist’ RL reward functions lead to scheming” §3

  Hmm but the cognition “try to break the rules without getting caught” might in practice lead to worse reward than just trying to follow the rules, if the chances to secretly hack are sufficiently rare and the punishment for getting caught is high enough.

  Eg I think that if I tried to “pretend I cared about my friends/​partner but exploit them when I can get away with it” would MASSIVELY fail me in the long run. Ppl are good at reading ppl. We’re transparent to each other. There is big cognitive overhead to maintaining two narratives. I’d end up with fewer committed/​deep long-term friendships.

  Relatedly, if you’re in a cult, the best way to signal deep commitment is to become a true believer.

  And it will, by comparison, be way easier for the overseers in the case of AI. Interp. Seeing all behaviour. Running counterfactual experiments.

  think there’s a good answer to that challenge, and it’s some combination of: (1) evolution has no way to build “a ruthless drive to have lots of grandkids” into our brains (details), and (2) even if it did, humans are not sufficiently smart and strategic in regards to long-term planning to be very effective ruthless consequentialists. (“We are the least intelligent species capable of building an industrial civilization.”)

  Ok interesting, we have pretty different intuitions here!

  • On 1, the equ that evolution chose, is way less sociopathic than it could have chosen. Some ppl are sociopaths! So it can be done. (Maybe there’s a lot of work done by the “aim for grandkids” part, sorry I didn’t read the link. But ppl do have some desires for healthy kids and grandkids)

  • On 2, I agree that if one agent’s cognition rises while the “are you a sociopath” checks stay constant, we’re in trouble. But we should imagine both rising with AI capabilities. (Note that humans evolved in a similar situation with both sides rising and the equ was non sociopathy.) Also, the world itself becomes more complex, making scheming plans harder to pull off.

  • I’m flat out sceptical the sociopath strategy would dominate in equilibrium. Eg: Men sociopaths would trick many women into partnering with them, leaving each. Women evolve to check hard for true commitment. Men evolve credible signals of non-sociopathy. I just don’t believe such signals would be impossible to find.

  Zooming out, I recall you thinking that humans aren’t sociopaths bc they have some special specific reward thing that we can’t replicate, related to wanting others to approve. Whereas I see no reason to think it’s some specific thing. We just had some selection pressure to seem like good non-sociopathic allies. That selection pressure worked. If we apply similar selection to ai, it will probably also work—the implementation details won’t need to match some specific human learning circuit

Mo­ral pub­lic goods are a big deal for whether we get a good future

• Tom Davidson 22 Feb 2026 4:21 UTC

  LW: 2 AF: 1

  0

  AF

  on: In­di­vi­d­ual AI rep­re­sen­ta­tives don’t solve Grad­ual Disempowerement

  What’s the story for why companies end up doing things that are catastrophic for their shareholders? (Feel free to give state the key points briefly, Im familiar with many of the ideas)

• Tom Davidson 21 Feb 2026 2:15 UTC

  LW: 6 AF: 3

  0

  AF

  on: Why we should ex­pect ruth­less so­ciopath ASI

  Where was the argument for consequentialism (including intense optimisation) and imitation being the only two ways to do impressive feats?

  Also, will you change your mind if the current paradigm is still non-psychopathic even when RL training dominates?

  (Big picture, I think the main place I might get off the train is in expecting future AI development to use a mix of rewards, including some from other equally capable AIs judging that behaviors aren’t deceptive/​unintended/​misaligned. And this mirroring the role that “other humans think we’re nice” played in evolution)

• Tom Davidson 16 Feb 2026 13:17 UTC

  2 points

  0

  in reply to: Jan_Kulveit’s comment on: Post-AGI Eco­nomics As If Noth­ing Ever Happens

  Thanks, good point re the alignment constraints being complex here.

  Re the Beren post: I agree with the post that the AI agents (/​automated companies) involved in creating new businesses will be in a better position to pick up the best investments than entities that lack the access to and knowledge about new startup opportunities.

  But it still seems like the AIs that do this could totally be investing on behalf of human principals? You only need a fairly minimal sort of alignment to ensure that the AI actually gives all the money it makes back to the human.

• Tom Davidson 16 Feb 2026 13:05 UTC

  2 points

  1

  on: Phan­tom Trans­fer and the Ba­sic Science of Data Poisoning

  V interesting—thanks for sharing!

• Tom Davidson 13 Feb 2026 9:38 UTC

  2 points

  0

  in reply to: Jan_Kulveit’s comment on: Post-AGI Eco­nomics As If Noth­ing Ever Happens

  thanks, i read the Phil Trammell critique and it helped me understand your position.

  My summary is that the points 1,2,3,4,6,9 were basically saying “AI might be misaligned and agentic and that could be bad for humans” and “maybe the institutions of law /​ democracy /​ markets will break down”.
  
  I get that if you think these things are pretty likely, the analysis is less interesting and you want the assumptions flagged.

  So overall i agree Phil should have a disclaimer like “i’m assuming we’ll get a great solution to alignment and that the current institutions of law + markets survive”, but i don’t think he needs to list out like 10 assumptions

• Tom Davidson 9 Feb 2026 9:39 UTC

  7 points

  2

  on: Post-AGI Eco­nomics As If Noth­ing Ever Happens

  Thanks for writing this.

  I’m sympathetic with the high-level claim that it’s very easy for someone to write down a rigorous but ultimately v misguided model in economics, and that a lot of the action is thinking about what the correct assumptions should be.

  I was a bit disappointed not to see (imo) clear examples where someone proposes a model but actually the conclusions don’t follow because of some distribution shift that they didn’t anticipate.

  Like, what are examples of models whose conclusions don’t follow once you account for the fact that there may be some AI systems that are consumers? Or once you account for the fact that AI systems may significantly influence the preferences of humans? E.g. to me it seems like the models predicting explosive growth in GDP or shrinking of the human labour share of the economy aren’t undercut by these things. I’m not sure what kinds of conclusions and reasoning you’re targeting here for critique.

• Tom Davidson 6 Feb 2026 15:33 UTC

  11 points

  6

  on: Tom David­son’s Shortform

  Accelerating tech progress + slow military procurement → cheap decisive strategic advantage

  A common prediction of an intelligence explosion is that tech progress gets faster and faster.

  When I speak to ppl from DC, I’m told that the government and military will be very slow to adopt new tech.

  If these two things are both true, there’s a scary implication.

  If tech progress speeds up by 30x relative to recent history, then a 3-year procurement delay by the military means they’re deploying tech that’s effectively 100 years outdated. Even a 1-year delay at that pace means your military is fielding equipment from a completely different technological era. The US military spends ~$1T/​year. But with tech that’s 30–100 years more advanced, you could potentially defeat them at 1/​100th of the spending — just $10B!

  A rogue actor — a private company or a government clique bypassing standard procurement — could spend a tiny fraction of the official military budget on cutting-edge tech and potentially overmatch the entire conventional military.

  There’s a massive untapped potential for cheap military dominance that no legitimate actor will exploit bc only the military has legitimate authority to procure weapons and they are bureaucratic.

  Here’s a visualisation of the basic dynamic:

  

  Possible solutions:

  1. Military procurement needs to get dramatically faster during an intelligence explosion. New AI systems and AI-produced technologies need to be rapidly integrated into official military capabilities. This probably means automating the procurement process itself. This is counterintuitive from some AI safety perspectives — many people’s instinct is to delay military AI deployment.

  2. Delay rapid tech progress until military procurement has become super fast + safe. This might in practice involve delaying the intelligence explosion and/​or the industrial explosion as well

  3. Better monitoring/​surveillance to make sure no one is secretly building military tech

  4. Others?