Jeremy Gillen

• Jeremy Gillen 10 Sep 2026 23:36 UTC

  2 points

  0

  in reply to: Jeremy Gillen’s comment on: Re­sults of a small ZBiotics RCT

  Xavier Roberts-Gaal sent me a Bayesian analysis of this data that I like more than my own:

  https://​​drive.google.com/​​file/​​d/​​1OucvxNv6q4fTB3Qm-u5923xhD2Ei8mPZ/​​view?usp=sharing

  

• Jeremy Gillen 9 Sep 2026 6:35 UTC

  4 points

  0

  in reply to: Brendan Long’s comment on: Re­sults of a small ZBiotics RCT

  Not in my data:

  

• Jeremy Gillen 9 Sep 2026 6:14 UTC

  7 points

  2

  on: Re­sults of a small ZBiotics RCT

  I replicated this study with a slightly larger treatment group (13) and 11 controls. The headline result is the same, no detectable effect improving hangovers (in fact they seem marginally worse). Here are the graphs:

  

  

  

  I was much worse than Nikola at collecting responses, the followup survey was collected mostly on Sunday but some filled it out the next day from memory.

  Here’s some more visualizations of the detailed data in a Claude artifact, and the (semi-anonymized) raw data at the end: https://​​claude.ai/​​code/​​artifact/​​419a852c-74b9-4f4d-a5be-01aa443cdf1f

  Check the raw data for your initials if you want to know whether you got the control or treatment group.

  I plan to keep testing it on myself (not blinded), but at this point I’d bet most of the apparent effect is placebo.

• Jeremy Gillen 3 Sep 2026 18:51 UTC

  6 points

  0

  in reply to: Luke Stebbing’s comment on: Re­s­olu­tion has a new Agent Foun­da­tions team

  Customized theorem proving scaffold seems clearly in the first category while making datasets to improve conceptual reasoning seems to me squarely in the second category. Finetuning to memorize relevant background knowledge might be a grey area.

  I don’t know a simple methodology for separating these things. We’ll discuss each case and try to game out potential impacts, which will involve guesswork. There’s a council inside Resolution that will decide where to draw the boundary in ambiguous cases. I trust them to take this seriously and reason about it carefully.

Re­s­olu­tion has a new Agent Foun­da­tions team

• Jeremy Gillen 31 Aug 2026 23:01 UTC

  85 points

  77

  in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

  Sorry to add to a pile-on, but just for the record:

  • I think the positive case for you joining OpenAI is weak-to-non-existent. Having a marginally better model of RSI isn’t clearly valuable. More detail and higher confidence parameters aren’t very important for governance or other interventions.

  • The case for negative impact is strong. You’ve said you know that people inside OpenAI want to use your work to accelerate RSI.

  • My primary model of your behavior is that you’re optimizing for importance and people paying attention to you (and maybe secondarily money), rather than primarily trying to do good. To the extent you consider me a friend, I hereby pressure you to quit.

  I appreciate you writing this list of takes, I’ve heard most of them but it’s good to put things online and some of them are interesting. I disagree with some (particularly 7 and 8) for standard reasons we’ve discussed.

• Jeremy Gillen 24 Aug 2026 16:16 UTC

  8 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  What were your previous go-tos?

• Jeremy Gillen 17 Aug 2026 19:37 UTC

  2 points

  −1

  in reply to: Thomas Kwa’s comment on: leogao’s Shortform

  I think there’s a natural bias toward overestimating positive impact and underestimating capability externalities. Overall this could easily lead to a community with overall negative impact, and I’d guess it has.
  If so, it might be optimal to try to add an explicit corrective bias against this. For this reason I prefer Leo’s statement.

• Jeremy Gillen 14 Aug 2026 21:58 UTC

  2 points

  0

  in reply to: Satya Benson’s comment on: Jeremy Gillen’s Shortform

  Where in the paper does Chaitin use that definition?

  I think there’s some sense in which the optimality assumption will be implicit if we do it like that, but I do like the idea of not stacking up a log for every merge.

• Jeremy Gillen 14 Aug 2026 19:00 UTC

  2 points

  0

  in reply to: Satya Benson’s comment on: Jeremy Gillen’s Shortform

  I haven’t seen conditional K complexity defined that way before, is there a textbook you’re getting it from? I’m not sure I’d want to drop the interpretation of conditional complexity as the length of a program, since that fits nicely with the latent string model definition, but maybe it’s a good idea anyway.

• Jeremy Gillen 14 Aug 2026 3:23 UTC

  14 points

  0

  on: Jeremy Gillen’s Shortform

  Algorithmic condensation

  I needed a more detailed sketch of algorithmic condensation so I could build on it, so here’s what I’ve got. I’ve done a proof for two concrete cases of the objectivity theorem, which I find much easier to understand than keeping track of the indices in the fully general proof.

  Latent String model

  A latent string model is a set of observations and a set of latent strings , and a Turing machine , such that prints

  1. appropriately separated, then

  2. a mapping from each index to subsets of (in some specified format, e.g. “<numeral for j>:<ordered list of ts>,...”), which defines the contribution function . We guarantee that the first latent contributes to all observations.

  In addition, if is given any -upward closed subset of the latents, it prints roughly the same output, except we only guarantee that:

  1. Each is correct if the set of latents contains all latents that contribute to .

  2. The printed contribution function is correct on all given latents. Additionally, it needs to be able to overestimate the contribution of unseen latents. We’ll call this overestimating version of the contribution function , and guarantee that for all unseen , . We also guarantee that if then .

  Almost perfect

  We’ll say that this latent string model is an -almost perfect condensation if

  1. (Reconstruction) For all , if then

     • If contains more than a fraction of the observations that contribute to , then we can reconstruct from that set of observations (given the larger context latents).

  2. (Markov) for all -upward closed sets , we have

     • If we set up the strings in a directed graph where a node is a child if its contribution is a subset of a parent (so there are never any grandparents, only parents), we have analogous conditional independence properties as a Bayes net, approximately.

       • A Bayes net has a local Markov property that each variable is conditionally independent of its non-descendants given its parents. don’t have shared descendants, and all shared ancestors are in their intersection. So d-separation can be used to show that this Markov property is implied by the Bayes net local Markov property. In the reverse direction, we can recover the local Markov property for any given node by taking any node , then for every node that isn’t a descendant, we have that ’s parents gives us independence from those nodes.

  3. (Well-separated) for all distinct , either we have or

     • Any two latents will contribute to sufficiently different observation sets, such that the two latents can be easily distinguished.

     • When is a smallest parent of , we assume that we still have (using the overestimating contribution function rather than the true contribution function).

  Objectivity for almost perfect

  Given two models, and , which are both -almost perfect condensations of the same data , we want to sketch how the objectivity result works.

  For concreteness, let’s assume looks like:

  

  1: {1,2,3,4,5,6,7,8,9,10,11,12}
  2,3,4:
  {1,2,3,4,5,6},{4,5,6,7,8,9},{7,8,9,10,11,12}

  and looks like:

  

  1: {1,2,3,4,5,6,7,8,9,10,11,12}
  2,3,4:
  {2,3,4,5,6},{5,6,7,8,9,10},{8,9,10,11,12,1}

  First let’s bound .

  Here’s the proof structure: First we find observation patches outside of each non- latent. Then we find the latents that contribute to these opposite patches. Then we apply Lemma 5.4 as many times as necessary to isolate .

  Observation patches are , , .

  The relevant latents to each patch are , , .

  Applying Lemma 5.4 to the first two:

  Then combining that resulting set of s with the last set :

  Merging the two equations:

  We can immediately bound each of the mutual information terms by using the Markov condition.

  To bound the first K-complexity, we use the definition of latent string model that allows us to recover subsets of observations from subsets of latents. prints out all the observations and prints out which tells us which ones definitely doesn’t contribute to (so we can rely on the printed observations being accurate). Well-separatedness guarantees that more than enough of these remain such that can be reconstructed using the Reconstruction assumption. By applying the reconstruction assumption we can replace the K-complexity with .

  So overall we have

  where is the number of merges required (usually low, but can be up to ).

  ---

  Now let’s bound .

  The same proof structure as above: First we find observation patches outside of each non- latent. Then we find the latents that contribute to these opposite patches. Then we apply Lemma 5.4 (see proof below) as many times as necessary to isolate .

  Observation patches: ,

  Relevant s: ,

  Apply Lemma 5.4:

  Now it’s slightly trickier to get rid of the K-complexities, because we only have the weaker reconstruction assumption . But we can reconstruct from with only error, so we can do

  .

  This uses the bound

  This overall leaves us with

  Note that the 5 comes from the example and depends on the number of merge steps required, it’ll generally scale up with the number of children.

  A small modification probably gives us and might be considered a more natural bound, but haven’t checked the details.

  ---

  Proof of Lemma 5.4

  ---

  On the complexity metric

  We’re using , which is the length of the shortest prefix-free code for the string .

  We could use complexity, which defines the conditional . To do this, we’d need to assume that all strings in the latent model are maximally compressed (or that we know their maximally compressed length) and that we know the compressed length of every upward closed subset. This isn’t that implausible an assumption, since we usually want a latent variable model to also be (at least approximately) a minimal length model. This would allow every step of our proofs to only have additive constants rather than . The reason I’m choosing to not use Kc is that I want the result to be more robust to approximation, so I don’t want a strong optimality condition on the latent string model.

  ---

  Objectivity theorem

  Missing from this document is how this proof generalizes across all latents in and , and how it generalizes to any latent variable models that satisfies the Reconstruction, Markov and Well-Separated conditions.

  Sam says there’s an objectivity theorem that says that if both and are -almost perfect condensations of the same observation set, then there is a bijection between the latents. I think this might require an extra axiom (something like “every latent is necessary” or “there aren’t additional independences other than the ones implied by the latents”). I’m fairly sure this will hold but haven’t worked through a full proof yet.

• Jeremy Gillen 11 Aug 2026 19:02 UTC

  10 points

  0

  in reply to: David Lorell’s comment on: Re­dux: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  Triggering

• Jeremy Gillen 11 Aug 2026 18:34 UTC

  9 points

  0

  on: Re­dux: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  Copying my reaction from DMs:
  
  I’m impressed! I tried pretty hard to do exactly this with Fable. I spent around two weeks prompting and encouraging it in the background (as I did other work) and it claimed to have a similar result with the large constant. I didn’t trust it so I pushed it to formalize but after a week of “only a couple more sessions to go” I gave up on it entirely, since it’d found flaws and then fixed them dozens of times and it wasn’t clear to me how much of it was real.
  
  I may regret saying this, but if/​when someone gets the small constant I intend to devote however long it takes to understanding the proof.

• Jeremy Gillen 17 Jun 2026 2:49 UTC

  4 points

  0

  on: A Test Suite for Concepts

  I find arc tasks are a good way to generate lots of simple, concrete example concepts.

• Jeremy Gillen 10 Jun 2026 1:40 UTC

  8 points

  0

  in reply to: Sherrinford’s comment on: Sher­rin­ford’s Shortform

  I can’t remember the capability profile in Crystal Society, what are some examples?

• Jeremy Gillen 14 May 2026 18:24 UTC

  3 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Op­ti­mi­sa­tion: Selec­tive ver­sus Predictive

  I’d guess: A predictive optimizer is built out of parts, but its defining feature is that one of the parts is beliefs about external reality. It can contain other parts, like learning or planning algorithms, which can be selective optimizers.

• Jeremy Gillen 6 May 2026 2:09 UTC

  4 points

  0

  in reply to: anaguma’s comment on: anaguma’s Shortform

  What’s great about it?

• Jeremy Gillen 5 May 2026 20:20 UTC

  6 points

  4

  in reply to: Seth Herd’s comment on: Mo­ti­vated rea­son­ing, con­fir­ma­tion bias, and AI risk theory

  Perhaps you’re thinking of cases where one side of the disagreement is making arguments that are pretty obviously bad or irrelevant, so the other side’s confidence is justified?

  Yeah, cases where you actually deeply understand the arguments made on all sides and see the flaws (even if they’re non-obvious flaws).

  I agree that this is rare, and almost all of the time when people think they’re in this situation they are not. But at the same time it’s often worth trying to be in this situation rather than giving up and sticking with uncertainty. It’s possible to succeed and it’s possible at the meta-level to be reasonably confident you’ve succeeded. (It’s rare, and one should be very suspicious every time this happens, but not infinitely suspicious).

  bunch of equally expert people are just being foolish

  I agree that if this should invoke even more suspicion, but the central cases I’m thinking about only involve non-foolish mistakes by experts.

  I think we’re fully in agreement, we both think one should be quite suspicious of oneself if you’re more confident than experts on a controversial question. And I agree that this is the main thing to emphasize. I just think it’s important to nitpick that this isn’t a fully general argument, the amount of suspicion is finite and can occasionally be overcome by object-level considerations.

• Jeremy Gillen 5 May 2026 19:12 UTC

  8 points

  4

  on: Mo­ti­vated rea­son­ing, con­fir­ma­tion bias, and AI risk theory

  I experienced one interesting shift when I started thinking that biases and cognitive limitations were central factors in disagreement: I liked people more.

  Thanks for reminding me of this.

  It’s that everyone’s confidence is probably too high, on most things, most of the time.

  I think this is true, but simultaneously I’m always annoyed at how confidently people dismiss the possibility of confidence. Even though the opposite is more common, there are a bunch of places where it’s possible to be reasonably confident despite strong disagreement between well-meaning experts.

• Jeremy Gillen 4 May 2026 20:33 UTC

  2 points

  0

  in reply to: TAG’s comment on: No Strong Orthog­o­nal­ity From Selec­tion Pressure

  The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.

  I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.

  It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.

  It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.

  I’m not sure what point you’re making here, what are the implications you’re referring to?