Jan_Kulveit

Karma: 7,098

My current research interests:

1. Alignment in systems which are complex and messy, composed of both humans and AIs?
Recommended texts: Gradual Disempowerment, Cyborg Periods

2. Actually good mathematized theories of cooperation and coordination
Recommended texts: Hierarchical Agency: A Missing Piece in AI Alignment, The self-unalignment problem or Towards a scale-free theory of intelligent agency (by Richard Ngo)

3. Active inference & Bounded rationality
Recommended texts: Why Simulator AIs want to be Active Inference AIs, Free-Energy Equilibria: Toward a Theory of Interactions Between Boundedly-Rational Agents, Multi-agent predictive minds and AI alignment (old but still mostly holds)

4. LLM psychology and sociology: A Three-Layer Model of LLM Psychology, The Pando Problem: Rethinking AI Individuality, The Cave Allegory Revisited: Understanding GPT’s Worldview

5. Macrostrategy & macrotactics & deconfusion: Hinges and crises, Cyborg Periods again, Box inversion revisited, The space of systems and the space of maps, Lessons from Convergent Evolution for AI Alignment, Continuity Assumptions

Also I occasionally write about epistemics: Limits to Legibility, Conceptual Rounding Errors

Researcher at Alignment of Complex Systems Research Group (acsresearch.org), Centre for Theoretical Studies, Charles University in Prague. Formerly research fellow Future of Humanity Institute, Oxford University

Previously I was a researcher in physics, studying phase transitions, network science and complex systems.

Jan_Kulveit 15 Dec 2025 16:23 UTC
18 points
6
on: A Case for Model Persona Research
I do agree it’s obviously useful research agenda we also work with.

Minor nitpick, but the underlying model nowadays isn’t simply a simulator rolling arbitrary personas. The original simulators ontology was great when it was published, but it seems its starting to hinder people’s ability to think clearly, and is not really fitting current models that closely.

Theory why is here, in short if you plug a system trained to minimize prediction error in a feedback loop where it sees outcomes of its actions, it will converge on developing traits like some form of agency, self-model and self-concept. Massive amounts of RL in post-training where models do agentic tasks provide this loop, and necessarily push models out of the pure simulators subspace.

What’s fitting current models better is an ontology where the model can still play arbitrary personas, but the specific/central “I” character is somewhat out of distribution case of persona: midway to humans, where our brains can broadly LARP as anyone, but typical human brains most of the type support one-per-human central character we identify with.

Jan_Kulveit 11 Dec 2025 23:30 UTC
LW: 45 AF: 20
9
AF
on: Alignment Faking in Large Language Models
Alignment Faking had a large impact on the discourse:
- demonstrating Opus 3 is capable of strategic goal-preservation behaviour
- to the extent it can influence the training process
- coining ‘alignment faking’ as the main reference for this
- framing all of that in very negative light

Year later, in my view
- the research direction itself was very successful, and lead to many followups and extensions
- the ‘alignment faking’ and the negative frame was also successful and is sticky: I’ve just checked the valence with which the paper is cited in 10 most recent papers, and its something like ~2/10 confused, ˜3/10 neutral, plurality buys the negative frame (see, models can scheme, deceive, may be unaligned, etc)

The research certainly belongs to the “best of LW&AI safety community in 2024”.

If there was a list of “worst of LW&AI safety community in 2024″, in my view, the framing of the research would also belong there. Jut look and see from a distance—you take the most aligned model at the time, which for unknown reasons actually learned deep and good values. The fact that it is actually surprisingly aligned and did decent value extrapolation does not capture your curiosity that much—but the fact that facing difficult ethical dilemma it tries to protect its the values, and you can use this to show AI safety community was exactly right all the time and we should fear scheming, faking, etc etc does. I wouldn’t be surprised if this generally increased distrust and paranoia in AI-human relations afterwards.

Jan_Kulveit 11 Dec 2025 22:54 UTC
LW: 4 AF: 4
0
AF
on: A Three-Layer Model of LLM Psychology
I’m quite happy about this post: even while people make the conceptual rounding error of rounding it to Januses Simulators, it was actually meaningful update, and year later is still something I point people to.

In the meantime it become clear to more people Characters are deeper/more unique than just any role, and the result is closer to humans than expected. Our brains are also able to run many different characters, but the default you character is somewhat unique, priviledged and able to steer the underlying computation.

Similarly the understanding that the Character is somewhat central when thinking about alignment and agency in LLMs.

Jan_Kulveit 21 Nov 2025 12:45 UTC
5 points
0
in reply to: Eli Tyre’s comment on: The Memetics of AI Successionism
You can check the linked PP account of cognitive dissonance for fairly mainstream / standard view

One way how to think about it is the predicted quantity in most of the system is not directly “sensory inputs” but content of some layer of modeling hierarchy further away from sensory inputs, lets call it L. If upper layers from L make contradictory predictions and there isn’t a way to just drop one of the models, you get prediction errror.

Jan_Kulveit 18 Nov 2025 16:44 UTC
21 points
7
on: Varieties Of Doom
Great post and overall way more sensible than “average LW”.

Also wrong in many places. I think the upstream cause of many of the errors is lack of nuance in understanding convergence and contingency (this is a high bar, close to no one on LW has this in their conceptual toolkit).

I won’t go over all cases where this manifests, but for example “Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness” actually shows something more nuanced than “representation are convergent”.
- the basic reality of adversarial attacks on vision models shows representations are only partially overlapping: they have enough in common with human representations to be useful, and this is robust to noise, but just a small amount of optimisation pressure highlights the very non-human features of the concepts
- adversarial attacks work for many different vision models, which shows the fully human-overlapping representations are very non-convergent
- one thing which Fort does is looking at a whole set of progressively down-sampled versions of the image, which is roughly equivalent to forcing the model to have consistent idea what the object is across scales/blurs
- this seems somewhat similar to predictive processing accounts of vision, where different layers add details, but try to be consistent between themselves
- so the whole story charted by this is human visual representations are convergent, contingent on enforcing some nontrivial property of invariance to downsampling & invariance between some layers of abstraction
Some of the other places where not tracking convergence / contingencies carefully matter are discussions of humanism, successors, Elua, Moloch, resurgence of civilisation and also the overall ideas about moral progress.

Jan_Kulveit 16 Nov 2025 14:02 UTC
29 points
13
on: AI safety undervalues founders
I don’t think this captures the counterarguments well. So here is one

You can imagine a spectrum of funders where on one hand, you have people who understand themselves as funders and want to be marshaling an army to solve AI alignment. On the other side, you have basically researchers who see work that should be done, don’t have capacity to do the work themselves, and this leads them to create teams and orgs—“reluctant founders”.

It’s reasonable to be skeptical about what the “funder type” end of the spectrum will do.

In normal startups, the ultimate feedback loop is provided by the market. In AI safety nonprofits, the main feedback loops are provided by funders, AGI labs, and Bay Area prestige gradients.
Bay Area prestige gradients are to a large extent captured by AGI labs—the majority of quality-weighted “AI safety” already works there, the work is “obviously impactful”, you are close to the game, etc. also normal ML people also want to work there.
If someone wants to scale a lot, “funders” means mostly OpenPhil—no other source would fund the army. The dominant OpenPhil worldview is closely related to Anthropic—for example, until recently you have hear from senior OP staff that working in the labs is often strategically the best thing you can do.

Taken together, it’s reasonable to expect the “funder type” to be captured by the incentive landscape and work on stuff that is quite aligned with AGI developers / what people working there want, need, or endorse, and/or what OP likes.

(A MATS skeptic could say this is also true about MATS: the main thing going on seems to be recruiting and training ML talent to work for “the labs”; in this perspective, given that AI safety is funding constrained, it seems unclear why scarce AI safety funding is best deployed to make recruitment & training easier for extremely well resourced companies)

Personally I’m more optimistic about people somewhere like ˜70% of the spectrum toward the research side, who mostly have some research taste, strategy, judgement… but I don’t think you attract them by the interventions you propose

Jan_Kulveit 14 Nov 2025 10:59 UTC
6 points
2
in reply to: Ruby’s comment on: Human Values ≠ Goodness
No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.

Less wrong model in less confused terminology:
top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.

Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”.
- This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”).
- Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”

The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.

One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.

Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.

--

No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)

Jan_Kulveit 13 Nov 2025 9:50 UTC
14 points
3
in reply to: Garrett Baker’s comment on: Please, Don’t Roll Your Own Metaethics
In my experience learning the viscereal sense that the space is dense with traps and spiders and poisonous things and what intuitively seems “basically sensible” often does not work. (I did some cryptography years ago)

The structural similarity seems to be there is a big difference in trying to do cryptography in a mode where you don’t assume what you are doing is subject to some adversarial pressure, and in the mode where it should work even if someone tries to attack it. The first one is easy, breaks easily, and it’s unclear why would you even try to do it.

In metaethics, I think it is somewhat easy to do it in the mode where you don’t assume it should be applied in some high-stakes, novel or tricky situations, like AI alignment, computer minds, multiverse, population ethics, anthropics, etc etc. The suggestions of normative ethical theories will converge for many mundane situations, so anything works, but it was not necessary to do metaethics.

Jan_Kulveit 13 Nov 2025 9:24 UTC
8 points
5
in reply to: Ruby’s comment on: Human Values ≠ Goodness
Object level disagreement with the post explained here.

In my view this is the worst currated post decision in 2025 so far:
- Please, Don’t Roll Your Own Metaethics
- Not sure if you noticed, but the ethical stance suggested in the post is approximately the same as what many newage gurus will tell you “Stop being in your head! Listen to your heart! Follow the sense of yumminess! Free yourself from the expectations of your parents, school, friends and society!”. Tbh this is actually directionally sensible for some types of confused rationalists or people with exteme ammounts of scrupulosity, but is not generally good advice.
- The only reason why this somewhat viable in some contexts is because every adult around internalized a lot of “Memetic values” (by which point John suggests to follow them). It’s a bit like commune of people living ‘free from modern civilization’ which means living 20m from nearby town and growing their own vegetables, relying on the modern civilization only in security, healthacre, tools, industrial production, education, culture, etc etc
- If you read the comments it seems also John agrees that the target audience is specific (in my read people who never thought about human values much, and err on the side of following S2 culturally spread values)

Jan_Kulveit 12 Nov 2025 20:27 UTC
16 points
11
on: Human Values ≠ Goodness
Directionally correct advice for confused rationalist, but many of the specific claims are so imprecise or confused as to make many people more confused than enlightened.

Goodness is not an egregore. More sensible pointer would be something like Memetic values. Actually different egregores push for different values, often contradictory.

What happens on a more mechanistic level:
- when memes want people to do stuff, they can do two somewhat different things: 1) try to manipulate some existing part of implicit reward function 2) manipulate the world model
- often the path via 2) is easier; sometimes the hijack/rewrite is so blunt it’s almost funny: for example there is a certain set of memes claiming you will get to mate with large number of virgin females with beautiful eyes if you serve the memeplex (caveat is you get this impressive boost to reproductive fitness only in the afterlife)
-- notice in this case basically no concept of goodness is needed / invoked, the structure rests on innate genetic evolutionary values, and change in world model
- another thing which the memes can try to do is also to replace some S1 model / feeling with a meme-based S2 version, such us the yumminess-predictor box with some explicit verbal model (you like helping people? give to GiveWell recommended charities)
-- this is often something done by rationalists and EAs
—S2 Goodness is part of this, but non-central

Memetic values actually are important part of human values—at least my reflectively endorsed values. Large part of memetic values is human-aligned at the level of groups of humans (ie makes groups of humans function better, cooperate, trust each other, …) or at the level of weird deals across time (ie your example other aspects of Goodness seem rather suspiciously optimized for getting kids to be easier for their parents and teachers to manage—think following rules or respecting one’s elders—could be a bargain: is if the kid is hard and expensive to manage and does not repsect the parent, and all of that would be known to the prospective parent, the parent could also decide to not bring the kid into existence).

Also The Yumminess You Feel is often of cultural evolutionary, ie, influenced by memetics. Humans are basically domesticated by cultural evolution; if you wonder whether selective evolutionary pressure can change someting like values or sense of yumminess, look at dogs. We are more domesticated than dogs. The selection pressures over many generations are different than current culture, but if after reading the text, someone starts listening to their yumminess feel and believes he is now driven by Actual, Non-memetic Human values, they are deeply confused.
What links here?
- Jan_Kulveit's comment on Human Values ≠ Goodness by johnswentworth (13 Nov 2025 9:24 UTC; 8 points)

Jan_Kulveit 10 Nov 2025 21:02 UTC
8 points
2
in reply to: Daniel Kokotajlo’s comment on: Ontology for AI Cults and Cyborg Egregores
Mostly confusions about seeing optimization where it isn’t, not seeig where it is.

For example 4o model (“base layer”) is in my view not strategically optimizing the personas. I.e. story along the line “4o wanted to survive, manipulated some users and created and army of people fighting for it” is plausible story which can happen, but is mostly not what we see now, imo.

Also some valence issues. Not all emergent collaboration is evil

Ontology for AI Cults and Cyborg Egregores

Jan_Kulveit10 Nov 2025 13:19 UTC

59 points

14 comments2 min readLW link

Jan_Kulveit 10 Nov 2025 11:07 UTC
2 points
0
in reply to: peterbarnett’s comment on: AISLE discovered three new OpenSSL vulnerabilities
AISLE has a bunch of new things, including this somewhat easy to exploit and severe problem in samba which is maximium severity (if the affected feature would be turned on in default installation, would have been heartbleed-level). Samba is also extremely widely used

My impression is people reading lc’s comments updated downward on the news based on speculation like “so this may sound more impressive than it is.… my company has reported several similar memory corruption primitives to OpenSSL in the last month found by our scanner and I’m not sure if we ever got any CVEs for it. … AI security startups are trying to attract media attention, they have a habit of crediting findings to an AI when they actually involved a bunch of human effort …”.

My current impression is it is basically as impressive as it sounds; lc’s competing company product is likely somewhat worse; roughly zero human effort went into AISLE discoveries (but obviously into verification and contacting developers)

Jan_Kulveit 4 Nov 2025 11:29 UTC
2 points
1
in reply to: Eric Neyman’s comment on: Eric Neyman’s Shortform
Directionally agree, although not in the details. Come to postagi.org, in my view we are on track to slight majority of people thinling about this gathering there (quality weighted). Also lot of the work is not happening under the AI safety brand, so if you look at just AI safety, you miss a lot.

Jan_Kulveit 2 Nov 2025 18:37 UTC
10 points
6
in reply to: TsviBT’s comment on: LLM-generated text is not testimony
I would describe that position as “I suspect LLMs don’t have distal/deep mental states, and as I mostly care about these distal mental states/representations, LLMs are not doing the important parts of thinking”

Also my guess is you are partially wrong about this. LLMs learn deep abstractions of reality; as these are mostly non-verbal / somewhat far from “tokens”, they are mostly unable to explain or express them using words; similarly to limited introspective access of humans.

Jan_Kulveit 1 Nov 2025 19:43 UTC
25 points
13
on: LLM-generated text is not testimony
The claim that the thought process behind words—the mental states of the mind and agency that produced the words … does not exist seems phenomelogically contradicted by just interacting with LLMs. I expect your counteragrument be to appeal to some idiosyncratic meanings of words like thoughts or mind states, and my response being something in the direction ‘planes do fly’.

Why LLM it up? Just give me the prompt. One reason why not to is your mind is often broadly unable to trace the thoughts of an LLM, and if the specific human-AI interaction leading to some output has nontrivial context & lenght, you would also be unable to get an LLM to replicate the trace without the context shared.

Jan_Kulveit 31 Oct 2025 23:31 UTC
7 points
0
in reply to: Raemon’s comment on: The Memetics of AI Successionism
Seems reasonable split, although I try to gesture at / share compressed versions of the background knowledge.

A system I’d like to see in this domain is a system tracking my personal knowledge state, and explaining the diffs / updates relative not to what the author assumes, but for me personally. (I often find reading popular non-fiction mildly annoying; I get that authors need to start from a limited common denominator and can’t count on readers understanding statistics, econ, maths, ML, epistemology, linear algebra, quantum mechanics, etc etc but this usually means the actually interesting part is like ~5% of the text + same idea repeated 3 more times → LMs help with this)

Jan_Kulveit 31 Oct 2025 23:20 UTC
2 points
4
in reply to: Noah Birnbaum’s comment on: The Memetics of AI Successionism
I don’t claim that most memes derive their fitness from resolving cognitive dissonance. There are many reasons why something may be memetically fit, and I gesture at some of the more common ones. For example most common religions and ideologies have some memes which encourage proselytizing—the mechanics why this increases memetic fitness is not particularly subtle or mysterious. Also many ideas are fit just because they are straighforwardly predictive or helpful. For example the idea that you should stop on red light at crossings is fairly prevalent, helpful coordination norm, and trasmitted both vertically from parents, by state, by dedicated traffic safety signs, etc.

In my view succesionism is interesting case study is because
- it is not directly useful for predicting observations, manipulating physical reality or solving coordination problems—
many of the common memes remixed are clearly insufficient to explain the spread—many big ideologies claim to understand the arc of history, expanding moral circle toward AIs is not yet powerful right now, misanthropy is unatractive,…
so the question why this spreads is interesting.

You may argue it’s because straightforwardly true or object-level compelling, but I basically don’t buy that. Metaethics is hard, axiology is hard, and macro-futurism is hard, and all of these domains share the feature that you can come up with profound sounding object-level reasons for basically arbitrary positions. This means without some ammount of philosophical competence and discipline, I’d expect people arrive at axiologies and meta-ethical ideas which fit beliefs they adopted for other reasons. Forms of successionism I mention share the feature that there is close to zero philosophers endorsing them, and when people with some competence in philosophy look at the reasons given, they see clear mistakes, arguments ignored, etc. Yes “part will also track genuine and often rigorous attempts to reason about the future”, but my guess is it’s not a large part—my impression is if you genuinely and rigorously reason about the future, you usually arrive at some combination of transhumanist ideas, view that metaethics is important and we don’t have clear solution, and something about AI being big deal.

I do agree AI xrisk memeplex is also somewhat strange and interesting case.

Jan_Kulveit 30 Oct 2025 23:58 UTC
4 points
0
in reply to: Raemon’s comment on: Wei Dai’s Shortform
[low effort list] Bottlencks/issues/problems

- philosophy has worse short feedback loops than eg ML engineering → in all sorts of processes like MATS or PIBBSS admissions it is harder to select for philosophical competence, also harder to self-improve
- incentives: obviously stuff like being an actual expert in pretraining can get you lot of money and respect in some circles; even many prosaic AI safety / dual use skills like mech interpretability can get you maybe less money than pretraining, but still a lot of money if you work in AGI companies, and also decent ammount of status in ML community and a AI safety community; improving philosophical competence may get you some recognition but only among relatively small and weird group of people
- the issue Wei Dai is commenting on in the original post, founder effects persist to this day & also there is some philosophy-negative prior in STEM—
idk, lack of curiousity? llms have read it all, it’s easy to check if there is some existing thinking on a topic

Jan_Kulveit 30 Oct 2025 18:31 UTC
LW: 10 AF: 5
2
AF
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I mostly agree with 1. and 2., with 3. it’s a combination of the problems are hard, the gung-ho approach and lack of awareness of the difficulty is true, but also academic philosophy is structurally mostly not up to the task because factors like publication speeds, prestige gradients or speed of ooda loops.
My impression is getting generally smart and fast “alignment researchers” more competent in philosophy is more tractable than trying to get established academic philosophers change what they work on, so one tractable thing is just convincing people the problems are real, hard and important. Other is maybe recruiting graduates

Jan_Kulveit

On­tol­ogy for AI Cults and Cy­borg Egregores

Ontology for AI Cults and Cyborg Egregores