Jeremy Gillen

• Jeremy Gillen 31 Aug 2026 23:01 UTC

  70 points

  62

  in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

  Sorry to add to a pile-on, but just for the record:

  • I think the positive case for you joining OpenAI is weak-to-non-existent. Having a marginally better model of RSI isn’t clearly valuable. More detail and higher confidence parameters aren’t very important for governance or other interventions.

  • The case for negative impact is strong. You’ve said you know that people inside OpenAI want to use your work to accelerate RSI.

  • My primary model of your behavior is that you’re optimizing for importance and people paying attention to you (and maybe secondarily money), rather than primarily trying to do good. To the extent you consider me a friend, I hereby pressure you to quit.

  I appreciate you writing this list of takes, I’ve heard most of them but it’s good to put things online and some of them are interesting. I disagree with some (particularly 7 and 8) for standard reasons we’ve discussed.

• Jeremy Gillen 24 Aug 2026 16:16 UTC

  8 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  What were your previous go-tos?

• Jeremy Gillen 17 Aug 2026 19:37 UTC

  2 points

  −1

  in reply to: Thomas Kwa’s comment on: leogao’s Shortform

  I think there’s a natural bias toward overestimating positive impact and underestimating capability externalities. Overall this could easily lead to a community with overall negative impact, and I’d guess it has.
  If so, it might be optimal to try to add an explicit corrective bias against this. For this reason I prefer Leo’s statement.

• Jeremy Gillen 14 Aug 2026 21:58 UTC

  2 points

  0

  in reply to: Satya Benson’s comment on: Jeremy Gillen’s Shortform

  Where in the paper does Chaitin use that definition?

  I think there’s some sense in which the optimality assumption will be implicit if we do it like that, but I do like the idea of not stacking up a log for every merge.

• Jeremy Gillen 14 Aug 2026 19:00 UTC

  2 points

  0

  in reply to: Satya Benson’s comment on: Jeremy Gillen’s Shortform

  I haven’t seen conditional K complexity defined that way before, is there a textbook you’re getting it from? I’m not sure I’d want to drop the interpretation of conditional complexity as the length of a program, since that fits nicely with the latent string model definition, but maybe it’s a good idea anyway.

• Jeremy Gillen 14 Aug 2026 3:23 UTC

  14 points

  0

  on: Jeremy Gillen’s Shortform

  Algorithmic condensation

  I needed a more detailed sketch of algorithmic condensation so I could build on it, so here’s what I’ve got. I’ve done a proof for two concrete cases of the objectivity theorem, which I find much easier to understand than keeping track of the indices in the fully general proof.

  Latent String model

  A latent string model is a set of observations and a set of latent strings , and a Turing machine , such that prints

  1. appropriately separated, then

  2. a mapping from each index to subsets of (in some specified format, e.g. “<numeral for j>:<ordered list of ts>,...”), which defines the contribution function . We guarantee that the first latent contributes to all observations.

  In addition, if is given any -upward closed subset of the latents, it prints roughly the same output, except we only guarantee that:

  1. Each is correct if the set of latents contains all latents that contribute to .

  2. The printed contribution function is correct on all given latents. Additionally, it needs to be able to overestimate the contribution of unseen latents. We’ll call this overestimating version of the contribution function , and guarantee that for all unseen , . We also guarantee that if then .

  Almost perfect

  We’ll say that this latent string model is an -almost perfect condensation if

  1. (Reconstruction) For all , if then

     • If contains more than a fraction of the observations that contribute to , then we can reconstruct from that set of observations (given the larger context latents).

  2. (Markov) for all -upward closed sets , we have

     • If we set up the strings in a directed graph where a node is a child if its contribution is a subset of a parent (so there are never any grandparents, only parents), we have analogous conditional independence properties as a Bayes net, approximately.

       • A Bayes net has a local Markov property that each variable is conditionally independent of its non-descendants given its parents. don’t have shared descendants, and all shared ancestors are in their intersection. So d-separation can be used to show that this Markov property is implied by the Bayes net local Markov property. In the reverse direction, we can recover the local Markov property for any given node by taking any node , then for every node that isn’t a descendant, we have that ’s parents gives us independence from those nodes.

  3. (Well-separated) for all distinct , either we have or

     • Any two latents will contribute to sufficiently different observation sets, such that the two latents can be easily distinguished.

     • When is a smallest parent of , we assume that we still have (using the overestimating contribution function rather than the true contribution function).

  Objectivity for almost perfect

  Given two models, and , which are both -almost perfect condensations of the same data , we want to sketch how the objectivity result works.

  For concreteness, let’s assume looks like:

  

  1: {1,2,3,4,5,6,7,8,9,10,11,12}
  2,3,4:
  {1,2,3,4,5,6},{4,5,6,7,8,9},{7,8,9,10,11,12}

  and looks like:

  

  1: {1,2,3,4,5,6,7,8,9,10,11,12}
  2,3,4:
  {2,3,4,5,6},{5,6,7,8,9,10},{8,9,10,11,12,1}

  First let’s bound .

  Here’s the proof structure: First we find observation patches outside of each non- latent. Then we find the latents that contribute to these opposite patches. Then we apply Lemma 5.4 as many times as necessary to isolate .

  Observation patches are , , .

  The relevant latents to each patch are , , .

  Applying Lemma 5.4 to the first two:

  Then combining that resulting set of s with the last set :

  Merging the two equations:

  We can immediately bound each of the mutual information terms by using the Markov condition.

  To bound the first K-complexity, we use the definition of latent string model that allows us to recover subsets of observations from subsets of latents. prints out all the observations and prints out which tells us which ones definitely doesn’t contribute to (so we can rely on the printed observations being accurate). Well-separatedness guarantees that more than enough of these remain such that can be reconstructed using the Reconstruction assumption. By applying the reconstruction assumption we can replace the K-complexity with .

  So overall we have

  where is the number of merges required (usually low, but can be up to ).

  ---

  Now let’s bound .

  The same proof structure as above: First we find observation patches outside of each non- latent. Then we find the latents that contribute to these opposite patches. Then we apply Lemma 5.4 (see proof below) as many times as necessary to isolate .

  Observation patches: ,

  Relevant s: ,

  Apply Lemma 5.4:

  Now it’s slightly trickier to get rid of the K-complexities, because we only have the weaker reconstruction assumption . But we can reconstruct from with only error, so we can do

  .

  This uses the bound

  This overall leaves us with

  Note that the 5 comes from the example and depends on the number of merge steps required, it’ll generally scale up with the number of children.

  A small modification probably gives us and might be considered a more natural bound, but haven’t checked the details.

  ---

  Proof of Lemma 5.4

  ---

  On the complexity metric

  We’re using , which is the length of the shortest prefix-free code for the string .

  We could use complexity, which defines the conditional . To do this, we’d need to assume that all strings in the latent model are maximally compressed (or that we know their maximally compressed length) and that we know the compressed length of every upward closed subset. This isn’t that implausible an assumption, since we usually want a latent variable model to also be (at least approximately) a minimal length model. This would allow every step of our proofs to only have additive constants rather than . The reason I’m choosing to not use Kc is that I want the result to be more robust to approximation, so I don’t want a strong optimality condition on the latent string model.

  ---

  Objectivity theorem

  Missing from this document is how this proof generalizes across all latents in and , and how it generalizes to any latent variable models that satisfies the Reconstruction, Markov and Well-Separated conditions.

  Sam says there’s an objectivity theorem that says that if both and are -almost perfect condensations of the same observation set, then there is a bijection between the latents. I think this might require an extra axiom (something like “every latent is necessary” or “there aren’t additional independences other than the ones implied by the latents”). I’m fairly sure this will hold but haven’t worked through a full proof yet.

• Jeremy Gillen 11 Aug 2026 19:02 UTC

  10 points

  0

  in reply to: David Lorell’s comment on: Re­dux: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  Triggering

• Jeremy Gillen 11 Aug 2026 18:34 UTC

  9 points

  0

  on: Re­dux: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  Copying my reaction from DMs:
  
  I’m impressed! I tried pretty hard to do exactly this with Fable. I spent around two weeks prompting and encouraging it in the background (as I did other work) and it claimed to have a similar result with the large constant. I didn’t trust it so I pushed it to formalize but after a week of “only a couple more sessions to go” I gave up on it entirely, since it’d found flaws and then fixed them dozens of times and it wasn’t clear to me how much of it was real.
  
  I may regret saying this, but if/​when someone gets the small constant I intend to devote however long it takes to understanding the proof.

• Jeremy Gillen 17 Jun 2026 2:49 UTC

  4 points

  0

  on: A Test Suite for Concepts

  I find arc tasks are a good way to generate lots of simple, concrete example concepts.

• Jeremy Gillen 10 Jun 2026 1:40 UTC

  8 points

  0

  in reply to: Sherrinford’s comment on: Sher­rin­ford’s Shortform

  I can’t remember the capability profile in Crystal Society, what are some examples?

• Jeremy Gillen 14 May 2026 18:24 UTC

  3 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Op­ti­mi­sa­tion: Selec­tive ver­sus Predictive

  I’d guess: A predictive optimizer is built out of parts, but its defining feature is that one of the parts is beliefs about external reality. It can contain other parts, like learning or planning algorithms, which can be selective optimizers.

• Jeremy Gillen 6 May 2026 2:09 UTC

  4 points

  0

  in reply to: anaguma’s comment on: anaguma’s Shortform

  What’s great about it?

• Jeremy Gillen 5 May 2026 20:20 UTC

  6 points

  4

  in reply to: Seth Herd’s comment on: Mo­ti­vated rea­son­ing, con­fir­ma­tion bias, and AI risk theory

  Perhaps you’re thinking of cases where one side of the disagreement is making arguments that are pretty obviously bad or irrelevant, so the other side’s confidence is justified?

  Yeah, cases where you actually deeply understand the arguments made on all sides and see the flaws (even if they’re non-obvious flaws).

  I agree that this is rare, and almost all of the time when people think they’re in this situation they are not. But at the same time it’s often worth trying to be in this situation rather than giving up and sticking with uncertainty. It’s possible to succeed and it’s possible at the meta-level to be reasonably confident you’ve succeeded. (It’s rare, and one should be very suspicious every time this happens, but not infinitely suspicious).

  bunch of equally expert people are just being foolish

  I agree that if this should invoke even more suspicion, but the central cases I’m thinking about only involve non-foolish mistakes by experts.

  I think we’re fully in agreement, we both think one should be quite suspicious of oneself if you’re more confident than experts on a controversial question. And I agree that this is the main thing to emphasize. I just think it’s important to nitpick that this isn’t a fully general argument, the amount of suspicion is finite and can occasionally be overcome by object-level considerations.

• Jeremy Gillen 5 May 2026 19:12 UTC

  8 points

  4

  on: Mo­ti­vated rea­son­ing, con­fir­ma­tion bias, and AI risk theory

  I experienced one interesting shift when I started thinking that biases and cognitive limitations were central factors in disagreement: I liked people more.

  Thanks for reminding me of this.

  It’s that everyone’s confidence is probably too high, on most things, most of the time.

  I think this is true, but simultaneously I’m always annoyed at how confidently people dismiss the possibility of confidence. Even though the opposite is more common, there are a bunch of places where it’s possible to be reasonably confident despite strong disagreement between well-meaning experts.

• Jeremy Gillen 4 May 2026 20:33 UTC

  2 points

  0

  in reply to: TAG’s comment on: No Strong Orthog­o­nal­ity From Selec­tion Pressure

  The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.

  I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.

  It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.

  It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.

  I’m not sure what point you’re making here, what are the implications you’re referring to?

• Jeremy Gillen 1 May 2026 5:51 UTC

  10 points

  8

  in reply to: lumpenspace’s comment on: No Strong Orthog­o­nal­ity From Selec­tion Pressure

  i am very well aware the orthodox alignment line is that to maintain aligned goals across ontologies is very difficult!

  Then who’s in the orthogonalist circles you referred to? Or did you make them up?

  but how else would you explain

  When you try to derive someone’s premises from their conclusions, you still have to go and check whether you got it right. When people have different beliefs from you, it’s easy to slip up in this kind of reasoning. In my case it’s explained by me believing that selection isn’t always the main thing determining terminal goals (especially at finite times, or when there are other powerful optimizers interfering with selection).

  there is the significant risk that an agent will reach superintelligence while ultimately continuing to pursuit a valueless goal

  I endorse this statement. But as per this yud tweet, it might be useful to disentangle the orthogonality thesis from the chance of misalignment, because misalignment involves a stack of additional arguments. It’d be better to directly engage with the strong form of the orthogonality thesis as described in the second sentence of the wiki page and with the arguments for it, rather than making them up your own versions of these.

• Jeremy Gillen 1 May 2026 3:17 UTC

  33 points

  30

  on: No Strong Orthog­o­nal­ity From Selec­tion Pressure

  This post reads to me as if you’ve mostly extrapolated the beliefs and arguments of “orthodox alignment theorists” from small snippets and ended up with a wildly oversimplified strawman. Then you’ve re-derived mostly orthodox rat beliefs and arguments and presented them as a devastating counterargument.

  I think I’m fairly orthodox as lesswrongers go, and I agree with most of the statements and arguments you made in this post. I only have one or two disagreements toward the end of the post.

  One example of many, because I found this one particularly funny:

  There is an assumption, in orthogonalist circles, that these cycles are completely costless for the agent in question.

  The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview. You must be just making up an opposing worldview here? Where are the orthogonalist circles who say this?

• Jeremy Gillen 7 Apr 2026 22:33 UTC

  8 points

  0

  on: Jeremy Gillen’s Shortform

  A way to think about Condensation in the algorithmic setting

  These are note that describe a (somewhat flawed) variation of algorithmic condensation. It covers similar ground as this post by Abram, with slightly more concrete detail. I’m publishing it because a) it’s simple and relatively concrete and still works on most test cases I’ve worked through, and b) it helps motivate the more general approach.

  We want a generative model whose output is a set of observations. We want the hypothesis space to be the space of programs, and we want it to factorize naturally into components such that each component “contributes to” some subset of the observations.

  Observations: A set of tuples, where each tuple contains some content and a location. For example, the content might be a pixel r,g,b and the location might be the coordinates of that pixel x,y. Or the content might be a bit b and location as the index of that bit inside a large bitstring.

  To make the factorization work we need to impose some structure on the program. We’ll have one top program which can access (via built-in instructions) a List of other strings. The List must have an interface that satisfies some properties:

  1. It must be possible to retrieve elements based on their content, or their index in the list.

  2. Item retrieval from the List is such that any items other than the one that actually ends up being retrieved can be removed from the List without changing the behavior of the program.

  For example, we could have a function lookup that scores each item on the list and returns the highest scoring one.

  The top program, when executed, must output the full set of observations (and no others).

  Scoring subsets

  A condensation has a score for every subset A of observation variables.

  Conditioned^[1] score: Delete as many strings as possible from the List such that the subset of observations A is still recovered correctly by running the program. Sum the length of these programs. This gives the conditioned score. “Recovered correctly” means that the program outputs a set that contains all the correct pixels, and contains no more than one pixel for each location. It’s allowed to output incorrect pixels that aren’t part of A (because forcing it to output an exact subset would add complexity).

  Some more detail about List

  We want to make sure the program can’t hide complexity in unexpected ways using the List data structure. Kaarel pointed out one way of doing this in long lists, where you can sometimes save complexity by adding a bit to several items in the list. Items that contain the bit are located such that their index conveys useful information.

  To avoid this and similar hacks, we’ll assume that the List memory layout is automatically and perfectly managed behind the scenes, but the additional data required for this (e.g. storing the lengths of each item in the list) counts as part of the complexity of the top program. This stops us from adding a bit to an item in the list as a means of encoding its index, because that forces the List to remember which item has a unique length.

  Why include a location label in every observation?

  This is a simple way to force the program to keep track of which bit it’s generating at a given time, even when most parts of the program (or observations) aren’t present.

  Example: Two red squares

  Imagine our data is an image of two red squares on a black background. Remember each pixel also contains its location. Ideally, we want the model to contain a string representing the concept of “red square”, two strings for the individual squares, a string for the background color and a top string that holds information about how to tie it all together.

  We can guess the conditional scores of important subsets:

  • A single pixel in the background only requires the top string to generate. This is a longer string than what it would take to describe the background color, so it’s not a great score.

  • A single pixel in one of the red squares requires three strings to generate (top, red square, specific red square). Not great score.

  • The set of pixels that cover a red square require the same three strings as a single pixel. The score should be near-optimal.

  • The set of all pixels requires all strings, and its score is near-optimal (this is an ~optimal compression of the data).

  Example: 70% 1s random bitstring

  We want this condensation to have the following structure: One top string that captures the fact that 70% of the observations, and a string associated with each observation that captures the unique information in that bit.

  However, this won’t work. Because each string can hold a minimum of 1 bit, the total length of all strings is going to be much larger than the optimal compression of the observations.

  So the condensation model could go in two directions: Either it’ll drop the 70% information, or it’ll partition the observations into small groups in exchange for a poor score on the small subsets.

  This flaw motivates a better approach

  I think Sam Eisenstat has a (maybe unfinished, unpublished) approach that helps with this issue: Have the top latent generate a probability distribution over the other latents, and then score the other latents according to their nlogprob in this distribution, rather than their length.

  We can see this as taking the List memory management to its logical conclusion. My description above requires that the top latent tracks the length of each latent string (which is equivalent to specifying a uniform distribution over bitstrings of fixed length). Sam’s approach just allows this distribution to be non-uniform.

  This is cleaner theoretically, but I find it somewhat harder to think about. It makes it hard to look at an example and work out a set of programs the would score well. It also requires a bunch of machinery (around handling the distributions) that isn’t usually needed (because most useful latent concepts are larger than a bit), so it’s good to be able to drop it.

  1. ^

     Simple score:

     If we’re scoring using the simple score, we don’t need to allow access to other strings, so there’s no List. Simply have a set of programs, each of which outputs a set of observations. The simple score of A is the sum of lengths of programs such that the intersection of their output sets correctly pins down the observations in A.

• Jeremy Gillen 21 Feb 2026 18:10 UTC

  12 points

  4

  on: If you don’t feel deeply con­fused about AGI risk, some­thing’s wrong

  I like the advice you’ve given. I recently wrote a message of final advice to the PIBBSxILIAD fellowship. It seems fairly closely related.

  Final Advice from Jeremy

  I have some final advice for your future careers as alignment researchers:

  Keep your eye on the real problems and a full pathway to solving them. Don’t be distracted by the short term proxies of success. Don’t trust your employers, mentors, colleagues or funders to do this for you, they won’t. Always strive to understand the entire stack of motivations for your work and how it fits into a full solution to the problem. If you discover that some part of the stack isn’t as valid as you thought, switch research topic. In most fields your current set of skills is the most important factor in deciding what to work on. This is a field where catching up to the frontier of a subfield is relatively easy, so usually you should prioritize solving important problems over how well the problem fits with your current skills.

  The real problem is building a superintelligence that you understand at a very deep level, such that you know it will act as you intended. You need to understand things like: How its goals are stored and why they will stay the same as its world model updates. How much it can rely on its world model. When and how you can specify goals that have easy-to-predict and safe consequences. How it might be motivated to improve itself, and why you can trust it to do this.

  Your work will almost always be several steps upstream of this goal, and only solve a small subproblem. This is fine, as long as the connection is known and clearly communicated to others, so that the research community can prioritize necessary but neglected problems.

  Have ambition. Hold yourself to a higher standard than the current field exemplifies. Your work kinda only matters if it makes significant advances. So take risks. Avoid low value experiments done just for publication. Think about the deep conceptual questions and allow them to motivate your research.

  This attitude is somewhat associated with becoming a crackpot. To balance this out: Break down your research plans into small steps. Take care to communicate your ideas clearly and frequently. Work on small problems and help advance other people’s research, but treat this as training for the real work.

• Jeremy Gillen 2 Feb 2026 16:36 UTC

  2 points

  0

  in reply to: RogerDearnaley’s comment on: The cor­rigi­bil­ity basin of at­trac­tion is a mis­lead­ing gloss

  2 is close enough. Extrapolating the results of safe tests to unsafe settings requires a level of theoretical competence we don’t currently have. Steve Brynes just made a great post that is somewhat related, I endorse everything in that post.