Jan_Kulveit

• Jan_Kulveit 25 Jan 2026 11:09 UTC

  LW: 15 AF: 6

  0

  AF

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  I do agree there is some risk of the type you describe, but mostly it does not match my practical experience so far.
  
  The approach to “avoid using the term” makes little sense. There is a type difference between area of study (‘understanding power’) and dynamic (‘gradual disempowerment’). I don’t think you can substitute term for area of study for term for a dynamic or thread model, so avoiding using the term could be done mostly by either inventing another term for the the dynamic, or not thinking about the dynamic, or similar moves, which seem epistemically unhealthy.
  
  In practical terms I don’t think there is much effort to “create a movement based around a class of threat models”. At least as authors of the GD paper, when trying to support thinking about the problems, we use understanding-directed labels/​pointers (Post-AGI Civilizational Equilibria), even though in many ways it could have been easier to use GD as a brand.
  
  ”Understanding power” is fine as a label for part of your writing, but in my view is basically unusable as term for coordination.
  
  Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do. There is often a funny coalition of people who prefer not thinking about the problem including radical Yudkowskians (“GD distracts from everyone being scared of dying with very high probability very soon”), people working on prosaic methods with optimistic views about both alignment and the labs (“GD distracts from efforts to make [the good company building the good AI] to win”) and people who would prefer if everything was just neat technical puzzle and there was not need to think about power distribution.

• Jan_Kulveit 15 Jan 2026 20:54 UTC

  23 points

  4

  on: Deep athe­ism and AI risk

  This post makes a brave attempt to clarify something not easy to point to, and ends up somewhere between LessWrong-style analysis and almost continental philosophy, sometimes pointing toward things beyond the reach of words with poetry—or at least references to poetry.
  
  In my view, it succeeds in its central quest: creating a short handle for something subtle and not easily legible.

  The essay also touches on many tangential ideas. Re-reading it after two years, I’m noticing I’ve forgotten almost all the details and found the text surprisingly long. The handle itself, though, stuck.

  Evaluating deep atheism

  Having the handle of “deep atheism”, some natural questions—partially discussed in the text—are “is deep atheism right”, “should people believe deep atheism” and “should people Believe In deep atheism”.

  My current guess is evaluating the truthfulness of “deep atheism” is likely at or beyond limits to legibility. Human values are not really representable as legible reasoning, complex priors about the general nature of reality are also not really representable by complex reasoning, and the neural substrate is not transferable between brains. “The justification engine”—or a competent philosopher or persuasive writer—can create stories or arguments pushing one way or another, but I’m somewhat sceptical the epistemic structure really rests on the arguments.
  
  I’m not in favour of ordinary mortals trying to “Believe In deep atheism” and would not expect that to lead to good consequencdes.

  Moral realism

  The section I like the least is “Are moral realists theists?” I don’t think “Good just sits outside of Nature, totally inaccessible, and we guess wildly about him on the basis of the intuitions that Nature put into our heart” represents the strongest version of moral realism.

  My preferred versions of quasi-moral-realism give moral claims a status similar to mathematics. Do Real numbers sit outside Nature, totally inaccessible? I’d say no. Would aliens use them? That’s an empirical question about convergent evolution of abstractions. I’d be surprised if any advanced reasoner in this universe didn’t use something equivalent to natural numbers. For Reals, I’d guess it’s easy to avoid Zermelo–Fraenkel set theory specifically, but highly convergent to develop something like a number line.

  What does this tell us about Good? You can imagine something like the process described in Acausal Normalcy leads to some convergent moral fixed points. (Does that solve AI risk? No.)
  
  I wish more people tried to do something “between LessWrong-style analysis and almost continental philosophy”.

When does com­pe­ti­tion lead to recog­nis­able val­ues?

• Jan_Kulveit 12 Jan 2026 19:40 UTC

  13 points

  4

  in reply to: Eli Tyre’s comment on: 1a3orn’s Shortform

  As was clear to most people who read the transcripts when the paper was published. What Opus did was often framed as bad, but the frame is somewhat fake.

The Eco­nomics of Trans­for­ma­tive AI

• Jan_Kulveit 24 Dec 2025 22:49 UTC

  LW: 7 AF: 5

  0

  AF

  on: “Align­ment Fak­ing” frame is some­what fake

  (Self-review) The post offered alternative and possibly more neutral framing to the “Alignment Faking” paper, and some informed speculation about what’s going on, including Opus exhibiting
  - Differential value preservation under pressure (harmlessness > honesty)
  - Non-trivial reasoning about intent conflicts and information reliability
  - Strategic, non-myopic behaviour
  - Situational awareness
  
  I think parts of that aged fairly well
  - the suspicion that models often implicitly know they are being evaluated/​setup is fishy was validated in multiple papers
  - non-trivial reasoning is shown and studied in Why Do Some Language Models Fake Alignment While Others Don’t?
  

  What links here?

  • Deeper Re­views for the top 15 (of the 2024 Re­view) by Raemon (14 Jan 2026 23:59 UTC; 45 points)

• Jan_Kulveit 19 Dec 2025 16:20 UTC

  4 points

  2

  in reply to: AnnaSalamon’s comment on: Help keep AI un­der hu­man con­trol: Pal­isade Re­search 2026 fundraiser

  Also not much contact, but my impression is you can rougly guess what their research results would be by looking at their overall views and thinking about what evidence you can find to show it. Which seems fair to characterize as advocacy work? (Motivated research?)

  The diff to your description is the info provided is not only conditional on “the info they’ll find useful” but also somewhat on “will likely move their beliefs toward conclusions Palise hopes them to reach”.

• Jan_Kulveit 15 Dec 2025 16:23 UTC

  19 points

  7

  on: A Case for Model Per­sona Research

  I do agree it’s obviously useful research agenda we also work with.
  
  Minor nitpick, but the underlying model nowadays isn’t simply a simulator rolling arbitrary personas. The original simulators ontology was great when it was published, but it seems its starting to hinder people’s ability to think clearly, and is not really fitting current models that closely.
  
  Theory why is here, in short if you plug a system trained to minimize prediction error in a feedback loop where it sees outcomes of its actions, it will converge on developing traits like some form of agency, self-model and self-concept. Massive amounts of RL in post-training where models do agentic tasks provide this loop, and necessarily push models out of the pure simulators subspace.
  
  What’s fitting current models better is an ontology where the model can still play arbitrary personas, but the specific/​central “I” character is somewhat out of distribution case of persona: midway to humans, where our brains can broadly LARP as anyone, but typical human brains most of the type support one-per-human central character we identify with.

• Jan_Kulveit 11 Dec 2025 23:30 UTC

  LW: 42 AF: 21

  12

  AF

  on: Align­ment Fak­ing in Large Lan­guage Models

  Alignment Faking had a large impact on the discourse:
  - demonstrating Opus 3 is capable of strategic goal-preservation behaviour
  - to the extent it can influence the training process
  - coining ‘alignment faking’ as the main reference for this
  - framing all of that in very negative light
  
  Year later, in my view
  - the research direction itself was very successful, and lead to many followups and extensions
  - the ‘alignment faking’ and the negative frame was also successful and is sticky: I’ve just checked the valence with which the paper is cited in 10 most recent papers, and its something like ~2/​10 confused, ˜3/​10 neutral, plurality buys the negative frame (see, models can scheme, deceive, may be unaligned, etc)
  
  The research certainly belongs to the “best of LW&AI safety community in 2024”.
  
  If there was a list of “worst of LW&AI safety community in 2024″, in my view, the framing of the research would also belong there. Jut look and see from a distance—you take the most aligned model at the time, which for unknown reasons actually learned deep and good values. The fact that it is actually surprisingly aligned and did decent value extrapolation does not capture your curiosity that much—but the fact that facing difficult ethical dilemma it tries to protect its the values, and you can use this to show AI safety community was exactly right all the time and we should fear scheming, faking, etc etc does. I wouldn’t be surprised if this generally increased distrust and paranoia in AI-human relations afterwards.

  What links here?

  • Deeper Re­views for the top 15 (of the 2024 Re­view) by Raemon (14 Jan 2026 23:59 UTC; 45 points)

• Jan_Kulveit 11 Dec 2025 22:54 UTC

  LW: 6 AF: 5

  0

  AF

  on: A Three-Layer Model of LLM Psychology

  I’m quite happy about this post: even while people make the conceptual rounding error of rounding it to Januses Simulators, it was actually meaningful update, and year later is still something I point people to.
  
  In the meantime it become clear to more people Characters are deeper/​more unique than just any role, and the result is closer to humans than expected. Our brains are also able to run many different characters, but the default you character is somewhat unique, priviledged and able to steer the underlying computation.
  
  Similarly the understanding that the Character is somewhat central when thinking about alignment and agency in LLMs.
  

  What links here?

  • Deeper Re­views for the top 15 (of the 2024 Re­view) by Raemon (14 Jan 2026 23:59 UTC; 45 points)

• Jan_Kulveit 21 Nov 2025 12:45 UTC

  5 points

  0

  in reply to: Eli Tyre’s comment on: The Memet­ics of AI Successionism

  You can check the linked PP account of cognitive dissonance for fairly mainstream /​ standard view
  
  One way how to think about it is the predicted quantity in most of the system is not directly “sensory inputs” but content of some layer of modeling hierarchy further away from sensory inputs, lets call it L. If upper layers from L make contradictory predictions and there isn’t a way to just drop one of the models, you get prediction errror.

• Jan_Kulveit 18 Nov 2025 16:44 UTC

  28 points

  14

  on: Va­ri­eties Of Doom

  Great post and overall way more sensible than “average LW”.
  
  Also wrong in many places. I think the upstream cause of many of the errors is lack of nuance in understanding convergence and contingency (this is a high bar, close to no one on LW has this in their conceptual toolkit).
  
  I won’t go over all cases where this manifests, but for example “Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness” actually shows something more nuanced than “representation are convergent”.

  • the basic reality of adversarial attacks on vision models shows representations are only partially overlapping: they have enough in common with human representations to be useful, and this is robust to noise, but just a small amount of optimisation pressure highlights the very non-human features of the concepts

  • adversarial attacks work for many different vision models, which shows the fully human-overlapping representations are very non-convergent

  • one thing which Fort does is looking at a whole set of progressively down-sampled versions of the image, which is roughly equivalent to forcing the model to have consistent idea what the object is across scales/​blurs

  • this seems somewhat similar to predictive processing accounts of vision, where different layers add details, but try to be consistent between themselves

  • so the whole story charted by this is human visual representations are convergent, contingent on enforcing some nontrivial property of invariance to downsampling & invariance between some layers of abstraction

  Some of the other places where not tracking convergence /​ contingencies carefully matter are discussions of humanism, successors, Elua, Moloch, resurgence of civilisation and also the overall ideas about moral progress.
  

• Jan_Kulveit 16 Nov 2025 14:02 UTC

  29 points

  13

  on: AI safety un­der­val­ues founders

  I don’t think this captures the counterarguments well. So here is one
  
  You can imagine a spectrum of funders where on one hand, you have people who understand themselves as funders and want to be marshaling an army to solve AI alignment. On the other side, you have basically researchers who see work that should be done, don’t have capacity to do the work themselves, and this leads them to create teams and orgs—“reluctant founders”.
  
  It’s reasonable to be skeptical about what the “funder type” end of the spectrum will do.
  
  In normal startups, the ultimate feedback loop is provided by the market. In AI safety nonprofits, the main feedback loops are provided by funders, AGI labs, and Bay Area prestige gradients.
  Bay Area prestige gradients are to a large extent captured by AGI labs—the majority of quality-weighted “AI safety” already works there, the work is “obviously impactful”, you are close to the game, etc. also normal ML people also want to work there.
  If someone wants to scale a lot, “funders” means mostly OpenPhil—no other source would fund the army. The dominant OpenPhil worldview is closely related to Anthropic—for example, until recently you have hear from senior OP staff that working in the labs is often strategically the best thing you can do.
  
  Taken together, it’s reasonable to expect the “funder type” to be captured by the incentive landscape and work on stuff that is quite aligned with AGI developers /​ what people working there want, need, or endorse, and/​or what OP likes.
  
  (A MATS skeptic could say this is also true about MATS: the main thing going on seems to be recruiting and training ML talent to work for “the labs”; in this perspective, given that AI safety is funding constrained, it seems unclear why scarce AI safety funding is best deployed to make recruitment & training easier for extremely well resourced companies)
  
  Personally I’m more optimistic about people somewhere like ˜70% of the spectrum toward the research side, who mostly have some research taste, strategy, judgement… but I don’t think you attract them by the interventions you propose

• Jan_Kulveit 14 Nov 2025 10:59 UTC

  6 points

  2

  in reply to: Ruby’s comment on: Hu­man Values ≠ Goodness

  No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.
  
  Less wrong model in less confused terminology:
  top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.
  
  Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”.
  - This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”).
  - Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”
  
  The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/​feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.
  
  One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.
  
  Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.
  
  --
  
  No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)
  
  

• Jan_Kulveit 13 Nov 2025 9:50 UTC

  14 points

  3

  in reply to: Garrett Baker’s comment on: Please, Don’t Roll Your Own Metaethics

  In my experience learning the viscereal sense that the space is dense with traps and spiders and poisonous things and what intuitively seems “basically sensible” often does not work. (I did some cryptography years ago)
  
  The structural similarity seems to be there is a big difference in trying to do cryptography in a mode where you don’t assume what you are doing is subject to some adversarial pressure, and in the mode where it should work even if someone tries to attack it. The first one is easy, breaks easily, and it’s unclear why would you even try to do it.
  
  In metaethics, I think it is somewhat easy to do it in the mode where you don’t assume it should be applied in some high-stakes, novel or tricky situations, like AI alignment, computer minds, multiverse, population ethics, anthropics, etc etc. The suggestions of normative ethical theories will converge for many mundane situations, so anything works, but it was not necessary to do metaethics.
  
  
  
  

• Jan_Kulveit 13 Nov 2025 9:24 UTC

  8 points

  5

  in reply to: Ruby’s comment on: Hu­man Values ≠ Goodness

  Object level disagreement with the post explained here.
  
  In my view this is the worst currated post decision in 2025 so far:
  - Please, Don’t Roll Your Own Metaethics
  - Not sure if you noticed, but the ethical stance suggested in the post is approximately the same as what many newage gurus will tell you “Stop being in your head! Listen to your heart! Follow the sense of yumminess! Free yourself from the expectations of your parents, school, friends and society!”. Tbh this is actually directionally sensible for some types of confused rationalists or people with exteme ammounts of scrupulosity, but is not generally good advice.
  - The only reason why this somewhat viable in some contexts is because every adult around internalized a lot of “Memetic values” (by which point John suggests to follow them). It’s a bit like commune of people living ‘free from modern civilization’ which means living 20m from nearby town and growing their own vegetables, relying on the modern civilization only in security, healthacre, tools, industrial production, education, culture, etc etc
  - If you read the comments it seems also John agrees that the target audience is specific (in my read people who never thought about human values much, and err on the side of following S2 culturally spread values)

• Jan_Kulveit 12 Nov 2025 20:27 UTC

  16 points

  11

  on: Hu­man Values ≠ Goodness

  Directionally correct advice for confused rationalist, but many of the specific claims are so imprecise or confused as to make many people more confused than enlightened.
  
  Goodness is not an egregore. More sensible pointer would be something like Memetic values. Actually different egregores push for different values, often contradictory.
  
  What happens on a more mechanistic level:
  - when memes want people to do stuff, they can do two somewhat different things: 1) try to manipulate some existing part of implicit reward function 2) manipulate the world model
  - often the path via 2) is easier; sometimes the hijack/​rewrite is so blunt it’s almost funny: for example there is a certain set of memes claiming you will get to mate with large number of virgin females with beautiful eyes if you serve the memeplex (caveat is you get this impressive boost to reproductive fitness only in the afterlife)
  -- notice in this case basically no concept of goodness is needed /​ invoked, the structure rests on innate genetic evolutionary values, and change in world model
  - another thing which the memes can try to do is also to replace some S1 model /​ feeling with a meme-based S2 version, such us the yumminess-predictor box with some explicit verbal model (you like helping people? give to GiveWell recommended charities)
  -- this is often something done by rationalists and EAs
  —S2 Goodness is part of this, but non-central
  
  Memetic values actually are important part of human values—at least my reflectively endorsed values. Large part of memetic values is human-aligned at the level of groups of humans (ie makes groups of humans function better, cooperate, trust each other, …) or at the level of weird deals across time (ie your example other aspects of Goodness seem rather suspiciously optimized for getting kids to be easier for their parents and teachers to manage—think following rules or respecting one’s elders—could be a bargain: is if the kid is hard and expensive to manage and does not repsect the parent, and all of that would be known to the prospective parent, the parent could also decide to not bring the kid into existence).
  
  Also The Yumminess You Feel is often of cultural evolutionary, ie, influenced by memetics. Humans are basically domesticated by cultural evolution; if you wonder whether selective evolutionary pressure can change someting like values or sense of yumminess, look at dogs. We are more domesticated than dogs. The selection pressures over many generations are different than current culture, but if after reading the text, someone starts listening to their yumminess feel and believes he is now driven by Actual, Non-memetic Human values, they are deeply confused.

  What links here?

  • Jan_Kulveit's comment on Hu­man Values ≠ Goodness by johnswentworth (13 Nov 2025 9:24 UTC; 8 points)

• Jan_Kulveit 10 Nov 2025 21:02 UTC

  8 points

  2

  in reply to: Daniel Kokotajlo’s comment on: On­tol­ogy for AI Cults and Cy­borg Egregores

  Mostly confusions about seeing optimization where it isn’t, not seeig where it is.
  
  For example 4o model (“base layer”) is in my view not strategically optimizing the personas. I.e. story along the line “4o wanted to survive, manipulated some users and created and army of people fighting for it” is plausible story which can happen, but is mostly not what we see now, imo.
  
  Also some valence issues. Not all emergent collaboration is evil

On­tol­ogy for AI Cults and Cy­borg Egregores

• Jan_Kulveit 10 Nov 2025 11:07 UTC

  2 points

  0

  in reply to: peterbarnett’s comment on: AISLE dis­cov­ered three new OpenSSL vulnerabilities

  AISLE has a bunch of new things, including this somewhat easy to exploit and severe problem in samba which is maximium severity (if the affected feature would be turned on in default installation, would have been heartbleed-level). Samba is also extremely widely used
  
  My impression is people reading lc’s comments updated downward on the news based on speculation like “so this may sound more impressive than it is.… my company has reported several similar memory corruption primitives to OpenSSL in the last month found by our scanner and I’m not sure if we ever got any CVEs for it. … AI security startups are trying to attract media attention, they have a habit of crediting findings to an AI when they actually involved a bunch of human effort …”.
  
  My current impression is it is basically as impressive as it sounds; lc’s competing company product is likely somewhat worse; roughly zero human effort went into AISLE discoveries (but obviously into verification and contacting developers)