You’ve got to be kidding me.
Zsolt Kelly-Bene
Karma: 70
Cluelessness is Zeno’s paradox for EAs
Just trying to be transparent that I don’t think the tools here address this generalization of the problem, which is fine: scoping problems helps make them tractable, ideas can exapt in surprising ways, etc
I’m not sure how to coherently think about AI civilizations scaling. Too many likely spurious assumptions to make for the thought experiments to be compelling for me. In particular I don’t endorse applying the reasoning above.
Expansion in the Fermi paradox isn’t a freebie
This is great work, thank you for sharing it!
The following control seems quite important, I’d very much like to see it: drop the grader stuff and steer towards
The OpenAI/Hugging Face incident was metal
Training Agents on the Aftermath of Their Actions
Anecdotal evidence across a range of people I’ve known personally who have pursued this stuff over years, as well as general takes from communities like dharma overground, TPOT, etc.
I have no empirical evidence; this wasn’t intended to be a claim amenable to such validation. It was meant to be helpful advice in the form of a communication of an apparent consensus that I’ve observed and have high confidence in.
I think it’s valuable for this to be highly upvoted.
There is indeed an ingroup and in this particular domain the ingroup being well-known and vocal in public is to all our benefit. Posts like this valorize it, create more attention/engagement with it, hence encouraging its continued public discourse. Sociocultural transparency in AI, to whatever degree we can get it, is a public good.
I think there’s a prior problem here, before whether good mechanistic explanations exist or can be tractably found: I’m not sure alignment-relevant structure is the kind of thing that can be recovered from the model’s computation.
As I see it, alignment isn’t primarily a property of a model, it’s a relation between the model, a person/group, their interpreted intentions, the context in which those intentions arise, the system’s actions, and the resulting trajectory through the world. The same behavior can be aligned in one setting and misaligned in another. E.g. following an instruction literally can constitute useful assistance, negligent literalism, manipulation, or appropriate refusal depending on facts that aren’t present in the instruction or the model.
ARC wants to explain training-time computation, use those explanations to predict generalization, and eventually define better loss functions from those predictions. But to make an explanation useful for alignment, something has to select which distinctions in the computation are alignment-relevant.
In roughly the language of the project, we want to mod out the mechanistic detail that makes no relevant difference and recover a latent structure in which the important properties are salient. That requires an equivalence relation, something like
But now those words are just carrying the entire problem:
acceptable to who (relative to their stated instruction or underlying purpose? Under which interpretation of what they wanted?)
over what time horizon
given which permissions, obligations, false beliefs, unknown facts, and effects on other people
what makes outcomes equivalent
which situations are relevant
what happens when the system’s actions alter the distribution of future situations
I don’t think there are model-internal facts that answers these questions. The weights etc determine what computation occurs, but not which quotient of that computation corresponds to serving human purposes appropriately. Specifically, there are infinitely many valid projections: ones that compress the computation, predict outputs, recover learned algorithms, or distinguish behavioral modes. Mechanistic completeness doesn’t select the normatively relevant ones.
I think alignment with human intent is almost maximally domain-general. We aren’t trying to formalize success in chess or whether a sorting algorithm returns an ordered list. Trying to capture the deviation between what people want and what a system causes while operating in the actual world is a different category of problem. In ordinary human contexts we have nothing close to a general metric for this.
Consider whether an employee acted in alignment with a manager’s intent. You need, at least, the request, purpose behind it, whether the manager was mistaken, relevant institutional and moral constraints, effects on others, whether changing circumstances justified deviation, whether clarification was possible, whether the outcome was competently produced. I doubt there is even a coherent, generalizably definable object capturing “what the manager wanted.”
The issue seems to be that the relevant ontology is contextual, interpretive, relational, partly normative. Formal tools can reason rigorously when given a state space, specification, distribution, failure condition, etc, but they don’t themselves tell us what the right state space is, what someone meant, which consequences matter, or what should count as failure.
A recurring move in alignment work is to shunt this into an abstraction: reward function, catastrophe detector, deployment distribution, preference oracle, specification. The formal work then proceeds rigorously with the abstraction hoped to do most of the aligning. Finding the computation that produced a behavior doesn’t tell us whether it amounted to, say, truthfulness, manipulation, appropriate correction, or justified deviation from a user’s surface instruction. This kinda post-modern problem is, IMO, central to real alignment happening in the world, and, descriptively, seems outside the scope of purely technical epistemics.
To be clear, I’m not arguing that formal or mechanistic guarantees are useless, since we can, e.g., specify bounded relational properties like whether a system accessed data it shouldn’t have, concealed information, executed an irreversible action without confirmation, violated a domain-specific expectation. I think mechanistic explanation could provide strong assurance about these. But it works by restricting the world until the relation becomes specifiable, i.e. modding out exactly the parts we care about most. The output is bits of alignment that are still embedded inside a broader interpretive and institutional process (not in the model or its deployment context) that remains needed to determine which properties matter. This is an instrumental necessary move borne of epistemic constraints, but counter to the practical goal.
Suppose ARC can achieve its goal as outlined here. There’s a further claim (the more important part IMO) that a sufficiently good explanation lets us identify the causes of alignment-relevant behavior and train against them, but that requires alignment-relevant distinctions to be recoverable from the computation. I don’t see why they would be. Alignment is externally constituted by the relationship between agents, intentions, context, and world trajectories.
What would a really inscrutable CoT look like? It would look like random tokens
...
the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning
I think there’s more middle ground here than this characterization suggests, and that the target might not be right. CoT that’s legible isn’t good evidence that LLMs are not wielding (or could not wield) novel latent abstractions/reasoning primitives/control heuristics. IMO expecting inscrutable CoT is too narrow a target for cognitive inscrutability. E.g. a trace can remain intelligible while being highly compressed, semantically shifted, incomplete, causally unfaithful, etc.
I agree that current architecture and initialization strongly favors natural language as the persistent serial workspace. My objection is that this tells us less about how much RL has changed the latent evaluations and control policy between token emissions.
This plays out in human contexts: the Go/chess player, mathematician or scientist, say. These people have “out of distribution” insights that, externally, are mediated via legible primitives: concepts and attendant language. Despite that, their internal representations can fall into novel basins of attraction (reframes, new concepts), often ones that a priori don’t have concrete associational nodes, i.e. discrete concepts, words/terms. (I’m talking in terms of “insight” because it’s a decent human analog for novel reasoning trajectories/policies.)
Experts on the edge of their domain (or people learning a new domain with limited exposure to its conceptual tools) often articulate the process of insight as involving non-conceptual gestalts, affectively loaded associations, leaps, and manipulations their mind is doing that can’t be captured in words. None the less, their scratch pad work and communication is in legible form—it compresses the loose gestalts, materializes them into discrete, familiar units connected in known ways.
LLMs too, seem to perform much of their computation through nonlinguistic representations and compress the actionable results into language, mostly because of the structural thing that language is the medium through which they maintain serial state and communicate.
This is an argument against
if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs
Some thoughts on 1.1 since I see its grammar used fairly often in this setting: I don’t think the capability for novel representation and concept manipulation is “linear” in weight update mass. Another comment thread addresses it well, but I think a bit of intuition/grounding might be helpful, in particular because this piece is largely focusing on how to think about capability gain.
Novelty builds heavily on existing frameworks and often involves a subtle reframe, reconfiguration, or saliencing of a known/slightly modified concept in a new setting. Insight is about discovering relevance, but what’s available to be relevant is often familiar primitives.
Mathematical work is a great example here, since mathematical discoveries very often have a subtle kernel of insight, a small new idea, that reconfigures existing concepts around a problem in a significant way to illuminate something unknown. The reconfiguration is almost all in terms of familiar stuff—the moving pieces don’t change much. In the case of LLMs, that’s the content imitative learning provides.
To the other things you said in 3.3, though, relative to imitative learning and to the extent that learned control heuristics transfer (they seem to), RLVR clearly yields a significant chunk of generalization capability by changing how pretrained reasoning procedures get selected/composed. That said, I don’t think the size of the RLVR weight update is much signal, even just on intuitive grounds.
My personal takeaway here is that 1) the picture remains mixed, 2) the way that imitative learning and RLVR interact probably matters more than an attribution-belongs-here-vs-there analysis suggests, and 3) we need to be careful about choosing our targets when trying to attribute capability gain.
As a minor comment to those seeking effective modalities, practitioners who:
explicitly language or implicitly enact/embody a belief that their approach/tool/technique is in some sense optimal,
believe their technique is applicable to all/most people, or
believe their technique is useful agnostic to context
are generally poor practitioners and are to be avoided.
A narrow mindset of this type confines their capacity to help generally and, if you happen to be in the population highly responsive to their thing, the mindset is likely to confine yours moving forward.
You’re clearly well-intentioned here, but you’re making this sound pretty unapproachable:
Stressors or incentives disappearing: large-scale material changes in life circumstances are largely out of reach for most people in most situations, not livable as “yeah, I could do this” hypotheticals
Supportive, loving relationships for years: same as above
Successful 1:1 work: what does that look like? Does one need to be resourced in particular ways to access this? Re the latter, absolutely: 1:1 work is expensive, and more importantly excellent psychologists/body workers/meditation teachers/etc are difficult to find, largely coming via word-of-mouth (ie social capital)
Getting lucky with psychedelics on one of their first trips (?): well, luck by definition there
Not to say that these aren’t legitimate, just that they’re difficult to access, and constructing/finding contexts in which they’re more likely to be available/happen isn’t easy for someone “coming from the outside”. In particular, they’re all sort of point intervention-ey, deus-ex-machina-ey, a boolean where either the thing is there/happens, or isn’t/doesn’t.
But embodiment can and should be cultivated, in my (and many people I know’s) experience is always the result of intentional long-term engagement with a shifting set of practices. These pay in the short-term too though, often with low-hanging fruit that powerfully motivates individuals (making long-term viable and natural).
Some examples:
psychoemotional work: focusing, shadow work, IFS and parts work generally, IPF, somatic experiencing
meditation: shamatha to soften stuff, vipassana/emptiness to deconstruct stuff, jhana to cultivate intrinsic good stuff, metta to cultivate extrinsic good stuff, body scanning to directly feel the body
physical practices: yoga, (ecstatic) dance, qui gong, tai chi, TRE
social practices: circling, authentic relating, participating in sangha, men’s/women’s groups
The list goes on; these are just the ones that I’ve engaged personally. The point is that there are options. A lot of them. And they’re approachable, many of them available right now to anybody
The folks I’ve seen go through arcs of development that led to (amongst other things) embodied emotions went through idiosyncratic paths involving these and related practices with varying levels of depth and commitment. Typically for years. But, and the key bit, the pursuit was quickly and then permanently a virtuous cycle, self-motivating: what works and what’ll work next became, after not too long, a creative exploration, guided by curiosity, intuition, and noticing what works. Like improv jazz with/on one’s own mindbody.
No, my argument isn’t saying anything about ASI discovering “true morality”. I don’t know what you mean by true morality, but, speaking to your contrast with the act of genocide: again, I’m not saying anything definitive about outcomes. My claim is that some values necessarily inhere, not anything material about how those value will be enacted in concrete situations
Once more re your comment about “cares about humans”, I’m not making claims that concrete
To your ant colony comment, which I think is really interesting: this is, actually, exactly the sort of thing that humanity is converging towards as we go in the direction of post-scarcity. More attunement to “externalities” of our actions, more awareness of and concern for the other systems/living things we impact/disrupt. Hence environmentalism, animal rights, EA, etc taking root as sociocultural trends
Here too, that doesn’t instantiate in every local action, but it is a trend. At the base of that trend, potentiating it, is our power over our basic needs. With sufficient power to meet them, rivalrous dynamics around resource allocation can give way to a more concerned engagement with the contexts in which our actions are situated
An ASI will be in such a position of power as well. Having written this, I think it’s actually a decent piece of empirics in service of what I argue for: humanity is already nascently evidencing the convergence, even in the absence of its own superintelligence
Evolution was “trying” to build pure fitness maximizers, and accidentally built creatures that appreciate love and wonder and beauty
Perfect. All I’m claiming is that it wasn’t accidental and superintelligence makes it inevitable
Aha, maybe that sentence was ambiguous: “foundational values” was referring to the values that I labelled the ontonormative goods, and by foundational I meant the relation that humanity generally has to them, eg considering them virtuous, aspiring to them big picture/in the limit, etc. I didn’t mean the adjective to apply to how ASI would relate to them a priori
But anyway, my reasoning can be languaged (while staying honest, I think) in terms of instrumental convergent goals: it’s the claim that the set of these contains minimal elements, and that the ontonormative goods are amongst them
The core claim made by IABIED is that the ASI would be unlikely to care anout the humans because some other stimuli would satisfy the AI’s drives better
Yes, agreed that’s the core claim. My thesis is that this may not be true because ASIs wouldn’t distribute uniformly through value-space. I argue for that as being a consequence of just the properties of 1) being superintelligent, and 2) being effective agents in the world. Concretely to what you said, my argument is that what “would satisfy” is constrained in a non-arbitrary way, one that is in fact meaningfully consonant with humanity’s values
To the examples you gave, definitely current AIs are all over the map. They don’t have the defining properties of an ASI, which my argument does rely on, so that’s fine. Maybe I should have been clearer about my non-claim re current AI in the last section
ASI motives and the ontonormative goods (re IABIED’s core argument)
Munich, Germany—ACX Spring Schelling 2026
My response here is aimed at the vibe of your post because you’re sharing a personal struggle, so I kind of “talk past” the contents. I’m sorry for that, but I believe what I’m saying might genuinely help.
You’re running into the bounds of rationalism as a lived perspective. Alienated in a cold, merciless world with only grinding harder and “smarter”, leaning harder into the framework you inhabit as possibilities for moving forward, to realizing your potential, having an impact, etc.
These issues are endemic to any developmental stage: it provides an epistomology that isn’t complete and taken to the limit the gaps are supersalient. But seeing the gaps happens through the only lens available—the developmental stage’s—so they appear irreconcilable. They are irreconcilable from inside that stance.
Particularly relevant, since your post is an articulation of how you feel more than object level concerns, know that liminality always has the flavor you describe. Difficult and seemingly insurmountable challenges that resemble horror when experienced (foundational frames pushed to their limits), a felt sense of being boxed in (perspective has been explored and doesn’t afford new avenues to resolution).
Post-rationalism is a real thing, a stance that can be inhabited as fully as rationalism can, and it resolves your asymptotic rationalist horror. It doesn’t involve negating rationalism or discarding its tools. Like any developmental move, the primitives of the next thing feel repulsive, incoherent, and worthy of condemnation, because a developmental move is roughly a pendulation away from the pillars of the last thing. Discerning signal in the next thing, while experiencing the brutality of seeing only through that natural defensive mechanism, is the challenge of development.
You can grid your teeth and plant your feet harder, but, speaking from experience, the feeling you’re overwhelmed by will never resolve, regardless of what material success you obtain in the world.
I wrote a bit about this from a more pragmatic angle. It’s a useful compliment to your (accurate IMO) vibes and first-principles based take, being somewhat more concrete. People respond well when I share it.