I’m an Assistant Professor at Carnegie Mellon’s Machine Learning Department. I’m also a core faculty member in CMU’s Neuroscience Institute, and hold a courtesy appointment in the Robotics Institute.
My lab works at the intersection of neuroscience & AI to reverse-engineer animal intelligence and build the next generation of autonomous agents, responsibly and safely.
Learn more here: https://cs.cmu.edu/~anayebi
As promised, I’ve updated the blogpost with our new Contravariance theory paper regarding linear mapping complexity being at most sufficient for representational alignment between two systems under hard tasks. Putting the links here as well for convenience:
Paper: https://arxiv.org/abs/2607.08561
Substack series (which goes into more of the conceptual details & intuitions but without all the hairy math!): https://danyamins.substack.com/p/the-theory-of-contravariance-part0
Thanks so much, Bogdan! Excellent questions.
The preprint about the linear mapping complexity should be out in the next couple weeks. The proofs are all done, but we’ve been working hard to make the manuscript as clear as possible, since the mathematics of it is genuinely nontrivial (at least to us!), even though the proofs are based on empirical intuitions Dan & I developed over the years. I’ll update this blogpost with the arXiv link once it’s out :)
The implications of such a result for safety is that you can identify sufficient conditions (specifically, regularities in the pre-nonlinear thresholds at each layer) such that if two networks (or a brain and a network) solve the same tasks within
I discuss sample complexity on pg. 4 (cf. Remark 1) of the UAI paper. It’s additionally worth noting that the world model recovery bounds in the fully observed case of Theorem 1, and partially observed case of Theorem 3, are explicitly dependent on the number of goals (
I do think AI will be super useful for accelerating this type of research. One thing though, at least with current reasoning models like GPT-5.5-Pro, is that while I have yet to see them prove untrue theorems (like I did with o1/o3 once or twice a year ago), they can prove true but relatively trivial theorems, in that strong assumptions are smuggled in. That’s something that for now needs human experts to discern and continually interact with, “unwind”, and elicit from the models.
Thank you for the engagement! I do think that blocking a system to think about itself or its own cognition would actually reduce its task performance drastically, as it can’t effectively plan. This is why I think the whole capability --> {world models, etc} direction in Arrow 1 is important, because world models enable you to plan, and they implicitly entail a self-model, because you need to model yourself to be able to be a good estimator of the transition kernel
The point being made in the “On Functionalism” section is more fundamental than that: namely, thermodynamic bounds like the Bekenstein bound rule out infinite-information hypercomputation in physical systems.
Then Turing universality matters because once the relevant causal dynamics are finite/computable, the same organization can in principle be realized across substrates. No extra observer-relative “simulation” semantics is doing the work.
My recent selection theorems may be of interest, and I explicate a bit more how they may relate to questions of AI sentience/welfare: https://www.lesswrong.com/posts/SD9jayFvEctW82Duk/what-capable-agents-must-know-why-ai-consciousness-may-be-an-2
You actually don’t need an observer (human or otherwise!) to define computation—really what is meant there is finiteness.
Thank you for the kind words! I would take a look at the section at the end “On Functionalism” (https://www.lesswrong.com/posts/SD9jayFvEctW82Duk/what-capable-agents-must-know-why-ai-consciousness-may-be-an-2#On__Functionalism_) for the connection between consciousness and computation, as well as my 2014 Minds & Machines article linked therein for why these finite physical processes are Turing computations: https://arxiv.org/abs/1210.3304
You might find the arguments here, spelling out why anti-functionalism isn’t scientifically defensible under the currently well-accepted laws of physics, of potential interest: https://www.lesswrong.com/posts/SD9jayFvEctW82Duk/what-capable-agents-must-know-why-ai-consciousness-may-be-an-2#On__Functionalism_
Thanks for this great post! You may be interested in recent work of mine on corrigibility guarantees here: https://www.lesswrong.com/posts/M5owRcacptnkxwD2u/from-barriers-to-alignment-to-the-first-formal-corrigibility-1
The conclusions are consistent with your intuitions:
Aligning to all human values is intractable (even for computationally unbounded agents!)
Corrigibility is therefore a reasonable value set we can mostly all agree on that avoids the intractability in 1)
Corrigibility cannot be guaranteed by a single objective (as is currently done in RLHF & Constitutional AI), which is what prior proposals considered which failed
Corrigibility can instead be formally guaranteed via a small number of objectives, all of which have a higher lexicographic priority over the task objective, thereby making it a tractable safety target
I frankly think calling the Eon video any sort of “upload” is quite misleading and exaggerated. There are at least two fundamental reasons for this:
@Aurelia, as you, Ken, and even Eon (later on in their blog post) correctly point out, this was a leaky integrate-and-fire (LIF) model built from the fly connectome. So it’s not even close to the full brain of the fruit fly: no neurotransmitters, no synaptic weights, no synaptic dynamics from the fruit fly. We are not even faithfully simulating its brain in silico.
Not only is the central nervous system not a true upload, but the motor system isn’t either. What is instead used as a mapping between this LIF model and motor outputs is a policy that is hard-coded (not even imitation-learned via RL, though later on they & others do this) from fly behaviors from the NeuroMechFly team at EPFL. So the LIF model they use is neither necessary nor sufficient for the generated behavior: the fly policy can walk on its own without any additional inputs, as the EPFL team already demonstrated.
Great post! Very much agree about the conservatism.
This is why I find it useful to do economic analyses where the variables and factors are exposed, such as in my recent AI UBI analysis, so that one needn’t assume values for those variables, but can try out a multitude of scenarios and see how the predictions change, along with having an understanding of what factors matter more than others through the derived analytic form.
For example, one thing I found was that having a Scandinavian ownership amount of AI profits (~33%), drastically reduces the AI capability needed to be productive enough to fund a UBI. As a policy, this then seems very reasonable and attainable.
The full paper with all the cited sources can be found here: https://arxiv.org/abs/2505.18687
Pretraining doesn’t evade the lower bound: a “pointer” is just a compressed index into a large hypothesis space, and constructing it already requires resolving the same M-way ambiguity during pretraining. The lower bound applies regardless of where the bits are paid.
You can certainly put it in U2 instead (U2 is just a special case of U4 with one auxiliary), but putting it in U4 already ensures it’s suboptimal to preserve the switch & defer yet “kill all humans”, because it collapses many future intervention and recovery options simultaneously. In other words, it’s a hard constraint in effect — U4 enforces it as a global irreversibility invariant, whereas U2 is only needed for narrow single-channel invariants like switch reachability.
That’s correct, it can be naturally folded into U4 as one of its auxiliary utilities, in the same manner as we do for off-switch preservation.
Thanks! I really appreciate this, and I think your natural-latents framing fits nicely with the Part I point about needing to compress D down to a small set of crisp, structured latents. On the lexicographic point: it’s worth noting that even though Theorem 3 writes the full objective as a discounted sum, the safety heads U1-U4 aren’t long-horizon objectives — they’re local one-step tests whose optimal action doesn’t depend on future predictions. For example, U1 is automatically satisfied each round by waiting (and once the human approves the proposed action, the agent simply executes it, thereby engaging U2-U5), and U4 is a one-step reversibility check against an inaction baseline, not a long-run impact estimate. The only head with genuine long-horizon structure is U5, which sits below the safety heads, so discounting never creates optimization pressure on them. This makes the whole scheme intentionally deontic and “natural-latent–friendly”, exactly matching the tractable regime suggested by the large-D lower bounds of Part I.
I very much agree regarding corrigibility being the most tractable safety target for alignment. I discuss some formal results proving this claim, in case it’s of interest here: https://www.lesswrong.com/posts/M5owRcacptnkxwD2u/from-barriers-to-alignment-to-the-first-formal-corrigibility-1