johnswentworth

• johnswentworth 10 Sep 2025 3:33 UTC

  6 points

  2

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  Why don’t we live in a world where wealth can buy a society defenses against such egregores?

  I would point to the non-experts can’t distinguish true from fake experts problem. That does seem to be a central phenomenon which most parasitic egregores exploit. More generally, as wealth becomes more abundant (and therefore lots of constraints become more slack), inability to get grounded feedback becomes a more taut constraint.

  That said… do you remember any particular evidence or argument which led you toward the story at top of thread (as opposed to away from your previous understanding)?

• johnswentworth 9 Sep 2025 23:43 UTC

  8 points

  8

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  Without commenting on the specifics, I agree with a lot of the gestalt here as a description of how things evolved historically, but I think that’s not really the right lens to understand the problem.

  My current best one-sentence understanding: the richer humans get, the more social reality can diverge from physical reality, and therefore the more resources can be captured by parasitic egregores/​memes/​institutions/​ideologies/​interest-groups/​etc. Physical reality provides constraints and feedback which limit the propagation of such parasites, but wealth makes the constraints less binding and therefore makes the feedback weaker.

• johnswentworth 9 Sep 2025 21:35 UTC

  LW: 7 AF: 5

  0

  AF

  in reply to: johnswentworth’s comment on: john­swent­worth’s Shortform

  Proof

  Specifically, we’ll show that there exists an information throughput maximizing distribution which satisfies the undirected graph. We will not show that all optimal distributions satisfy the undirected graph, because that’s false in some trivial cases—e.g. if all the $Y$’s are completely independent of $X$, then all distributions are optimal. We will also not show that all optimal distributions factor over the undirected graph, which is importantly different because of the $P\left[X\right]>0$ caveat in the Hammersley-Clifford theorem.

  First, we’ll prove the (already known) fact that an independent distribution $P\left[X\right]=P\left[X_1\right]P\left[X_2\right]$ is optimal for a pair of independent channels $(X_1\to Y_1,X_2\to Y_2)$; we’ll prove it in a way which will play well with the proof of our more general theorem. Using standard information identities plus the factorization structure $Y_1-X_1-X_2-Y_2$ (that’s a Markov chain, not subtraction), we get

  $MI(X;Y)=MI(X;Y_1)+MI(X;Y_2|Y_1)$

  $=MI(X;Y_1)+\left(MI\right(X;Y_2)-MI(Y_2;Y_1)+MI(Y_2;Y_1|X\left)\right)$

  $=MI(X_1;Y_1)+MI(X_2;Y_2)-MI(Y_2;Y_1)$

  Now, suppose you hand me some supposedly-optimal distribution $P\left[X\right]$. From $P$, I construct a new distribution $Q\left[X\right]:=P\left[X_1\right]P\left[X_2\right]$. Note that $MI(X_1;Y_1)$ and $MI(X_2;Y_2)$ are both the same under $Q$ as under $P$, while $MI(Y_2;Y_1)$ is zero under $Q$. So, because $MI(X;Y)=MI(X_1;Y_1)+MI(X_2;Y_2)-MI(Y_2;Y_1)$, the $MI(X;Y)$ must be at least as large under $Q$ as under $P$. In short: given any distribution, I can construct another distribution with as least as high information throughput, under which $X_1$ and $X_2$ are independent.

  Now let’s tackle our more general theorem, reusing some of the machinery above.

  I’ll split $Y$ into $Y_1$ and $Y_2$, and split $X$ into $X_{1-2}$ (parents of $Y_1$ but not $Y_2$), $X_{2-1}$ (parents of $Y_2$ but not $Y_1$), and $X_{1\cap 2}$ (parents of both). Then

  $MI(X;Y)=MI(X_{1\cap 2};Y)+MI(X_{1-2},X_{2-1};Y|X_{1\cap 2})$

  In analogy to the case above, we consider distribution $P\left[X\right]$, and construct a new distribution $Q\left[X\right]:=P\left[X_{1\cap 2}\right]P\left[X_{1-2}|X_{1\cap 2}\right]P\left[X_{2-1}|X_{1\cap 2}\right]$. Compared to $P$, $Q$ has the same value of $MI(X_{1\cap 2};Y)$, and by exactly the same argument as the independent case $MI(X_{1-2},X_{2-1};Y|X_{1\cap 2})$ cannot be any higher under $Q$; we just repeat the same argument with everything conditional on $X_{1\cap 2}$ throughout. So, given any distribution, I can construct another distribution with at least as high information throughput, under which $X_{1-2}$ and $X_{2-1}$ are independent given $X_{1\cap 2}$.

  Since this works for any Markov blanket $X_{1\cap 2}$, there exists an information thoughput maximizing distribution which satisfies the desired undirected graph.

• johnswentworth 9 Sep 2025 20:41 UTC

  37 points

  4

  in reply to: Buck’s comment on: The Case Against AI Con­trol Research

  It’s been about 8 months since this post and Buck’s comment above.

  At the time, I didn’t bother replying to Buck’s comment because it didn’t really say much. My post basically said “this control agenda doesn’t seem to address anything important”, illustrated with a slew of specific examples of ways-things-go-wrong which seem-to-me to account for far more probability mass than scheming in early transformative AGI. Buck’s response was basically “yeah, those are real limitations, but IDK man scheming seems intuitively important?”. There’s a substantive argument to be had here about why I expect scheming of early transformative AGI to be either unlikely or easy to fix (relative to other problems), whereas Buck expects the opposite, and we haven’t had that debate.

  Anyway, I’m leaving this comment now because I think some people saw that I had a critique, that Buck had responded, and that I hadn’t responded back, and therefore assumed that the ball was in my court and I wasn’t engaging. That’s not my understanding of what’s going on here; I think Buck has basically not argued back substantively against the core of the critique, but also I haven’t argued strongly against his core crux either. We’ve identified a crux, and that’s the state of things. (And to be clear that’s fine, not every debate is worth having.)

• johnswentworth 8 Sep 2025 17:20 UTC

  LW: 5 AF: 3

  0

  AF

  on: john­swent­worth’s Shortform

  Does The Information-Throughput-Maximizing Input Distribution To A Sparsely-Connected Channel Satisfy An Undirected Graphical Model?

  [EDIT: Never mind, proved it.]

  Suppose I have an information channel $X\to Y$. The X components $X_1,...,X_m$ and the Y components $Y_1,...,Y_n$ are sparsely connected, i.e. the typical $Y_i$ is downstream of only a few parent X-components $X_{pa\left(i\right)}$. (Mathematically, that means the channel factors as $P\left[Y|X\right]={\prod }_{i}P\left[Y_i|X_{pa\left(i\right)}\right]$.)

  Now, suppose I split the Y components into two sets, and hold constant any X-components which are upstream of components in both sets. Conditional on those (relatively few) X-components, our channel splits into two independent channels.

  

  E.g. in the image above, if I hold $X_4$ constant, then I have two independent channels: $(X_1,X_2,X_3)\to (Y_1,Y_2,Y_3,Y_4)$ and $(X_5,X_6,X_7)\to (Y_5,Y_6,Y_7,Y_8)$.

  Now, the information-throughput-maximizing input distribution to a pair of independent channels is just the product of the throughput maximizing distributions for the two channels individually. In other words: for independent channels, we have independent throughput maximizing distribution.

  So it seems like a natural guess that something similar would happen in our sparse setup.

  Conjecture: The throughput-maximizing distribution for our sparse setup is independent conditional on overlapping X-components. E.g. in the example above, we’d guess that $P\left[X\right]=P\left[X_4\right]P[X_1,X_2,X_3|X_4]P[X_5,X_6,X_7|X_4]$ for the throughput maximizing distribution.

  If that’s true in general, then we can apply it to any Markov blanket in our sparse channel setup, so it implies that $P\left[X\right]$ factors over any set of X components which is a Markov blanket splitting the original channel graph. In other words: it would imply that the throughput-maximizing distribution satisfies an undirected graphical model, in which two X-components share an edge if-and-only-if they share a child Y-component.

  It’s not obvious that this works mathematically; information throughput maximization (i.e. the optimization problem by which one computes channel capacity) involves some annoying coupling between terms. But it makes sense intuitively. I’ve spent less than an hour trying to prove it and mostly found it mildly annoying though not clearly intractable. Seems like the sort of thing where either (a) someone has already proved it, or (b) someone more intimately familiar with channel capacity problems than I am could easily prove it.

  So: anybody know of an existing proof (or know that the conjecture is false), or find this conjecture easy to prove themselves?

• johnswentworth 8 Sep 2025 4:19 UTC

  2 points

  0

  in reply to: denkenberger’s comment on: Trans­porta­tion as a Constraint

  Tacking itself was one of the most important technological developments of transportation. IIUC there were occasional examples early on but it didn’t really catch on as a common method until the Renaissance. After that, IIUC it was one of the major drivers of Europe’s dominance of the seas.

• johnswentworth 7 Sep 2025 20:39 UTC

  3 points

  0

  in reply to: Lucius Bushnaq’s comment on: Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

  Yup, that is correct.

  If the the theorems approximately hold under approximate agreement on observables.

  Yeah, there is still the issue that the theorems aren’t always robust to approximation on the Agreement on Observables condition, though the Solomonoff version is and there’s probably other ways to sidestep the issue.

• johnswentworth 5 Sep 2025 21:29 UTC

  4 points

  0

  in reply to: Daniel C’s comment on: Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

  Yes, though I’ll flag that we don’t have robustness with respect to approximation on the agreement condition (though we do have other ways around that to some extent, e.g. using the Solomonoff version of natural latents), and those sorts of updates are the kind of thing which I’d expect to run into that robustness problem.

• johnswentworth 5 Sep 2025 19:02 UTC

  2 points

  0

  in reply to: MalcolmMcLeod’s comment on: Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

  @Lorxus translated the proofs here into categorical language IIRC (and found these deterministic versions much nicer than the older stochastic versions).

• johnswentworth 4 Sep 2025 16:40 UTC

  18 points

  1

  on: If I imag­ine that I am im­mune to ad­ver­tis­ing, what am I prob­a­bly miss­ing?

  A few parts:

  • Ads create common knowledge (as RamblinDash’s answer points out)

  • Creation of common knowledge is the main bottleneck to creating or moving Schelling points

  • Most social symbols/​signals of group identity or role identity are Schelling points

  As an example, let’s consider an oversimplified scenario with two social identities: the MBA types, and the artsy counterculture types. The MBA types want to signal that they’re MBA types, definitely not artsy counterculture types. The artsy counterculture types want to signal they’re artsy counterculture types, not MBA types. So this is a pure cooperative game: everyone wants clear signals, and everyone is incentivized to use those signals honestly.

  But that still leaves a degree of freedom in which symbols signal which group membership. In our world, converse hightops signal artsy counterculture, while MBA types debate the relative merits of oxfords vs brogues. But one could easily (and somewhat amusingly) imagine a different world in which the middle managers wear hightops, and the artsy types debate oxfords vs brogues. The choice of which symbols signal which things is a Schelling problem: everyone wants to coordinate on the choice of symbolic meaning.

  Now enter the Converse marketing team. The main thing they care about is that their brand have some strong group identity signal, so that there’s some group of people who will buy their overpriced sneakers. Early on, their customer base accidentally happens to have relatively few MBA types, so they lean into that and launch a marketing campaign to establish common knowledge that converse hightops signal artsy counterculture. Their product becomes a profitable part of the symbolic meaning Schelling point.

  Notably: at this point, the association between converse hightops and artsy counterculture group membership becomes a real feature of the world; converse hightops have real predictive power about the social behavior of people who wear them (and their friends). The ads aren’t just manipulating people into buying the product; the ads helped to create a real predictive feature in the environment. So it would be epistemically suboptimal to be “immune to” the outputs of the advertising.

• johnswentworth 4 Sep 2025 15:41 UTC

  20 points

  6

  in reply to: Thomas Kwa’s comment on: Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

  My main opinionated view relative to current discourse is that if someone is trying to apply any of this directly to LLMs, then they are probably very deeply confused about natural abstractions and also language and also agency/​intelligence/​etc in general.

  The path we’re optimizing for right now is to figure out the whole damn core of the theory of agency, get it across the theory-practice gap, and then not be so clueless about everything. See The Plan. Possibly with AI acceleration of the research at some point; our decisions right now are basically the same regardless of whether the research will be accelerated by AI later.

• johnswentworth 4 Sep 2025 15:34 UTC

  4 points

  0

  in reply to: Simon Pepin Lehalleur’s comment on: Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

  The appendices make heavy use of additivity across independent variables (and across factorizations more generally), which is the main thing I’d expect to need to work around in order to use other divergences/​metrics.

Nat­u­ral La­tents: La­tent Vari­ables Stable Across Ontologies

• johnswentworth 3 Sep 2025 14:38 UTC

  8 points

  0

  in reply to: Jeremy Gillen’s comment on: Re­sam­pling Con­serves Re­dun­dancy (Ap­prox­i­mately)

  It sure does feel like a powerful technique! We haven’t explored much how to generalize it yet, though.

  At the time, we were thinking about the optimization problem “max (the one error) subject to (constraint on other errors)”, and what the curve looks like which gives the max value as a function of the constraint errors. One (of many) angles I tried was to consider ways of transforming a latent, which would move it from one point in the feasible set to another point in the feasible set. And once I asked that question, basically the first thing I tried was the transformation in the proof which just scales down all the errors.

  At that point we had already done the Hellinger distances thing (also among many other things), on the general principle of “try it in the second order regime before trying to prove globally”, so it was just a matter of connecting the pieces together.

• johnswentworth 3 Sep 2025 14:31 UTC

  6 points

  0

  in reply to: Jeremy Gillen’s comment on: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  How did you work out you could use pareto-optimality? I’m guessing you got it from looking at properties of optimized and unoptimized empirical latents?

  No, actually. The magic property doesn’t always hold for pareto optimal latents; the resampling step is load bearing. So when we numerically experimented with optimizing the latents, we often got latents which didn’t have the structure leveraged in the proof (though we did sometimes get the right structure, but we didn’t notice that until we knew to look for it).

  We figured it out by algebraically playing around with the first-order conditions for pareto optimality, generally trying to simplify them, and noticed that if we assumed zero error on one resampling condition (which at the time we incorrectly thought we had already proven was a free move), then it simplified down a bunch and gave the nice form.

  It is that you can use just use the variables themselves as the latent and quickly compute mediation error?

  Yup.

• johnswentworth 3 Sep 2025 14:30 UTC

  2 points

  0

  in reply to: Jeremy Gillen’s comment on: (∃ Stochas­tic Nat­u­ral La­tent) Im­plies (∃ Deter­minis­tic Nat­u­ral La­tent)

  How did you work out you could use pareto-optimality? I’m guessing you got it from looking at properties of optimized and unoptimized empirical latents?

  No, actually. The magic property doesn’t always hold for pareto optimal latents; the resampling step is load bearing. So when we numerically experimented with optimizing the latents, we often got latents which didn’t have the structure leveraged in the proof (though we did sometimes get the right structure, but we didn’t notice that until we knew to look for it).

  We figured it out by algebraically playing around with the first-order conditions for pareto optimality, generally trying to simplify them, and noticed that if we assumed zero error on one resampling condition (which at the time we incorrectly thought we had already proven was a free move), then it simplified down a bunch and gave the nice form.

  It is that you can use just use the variables themselves as the latent and quickly compute mediation error?

  Yup.

• johnswentworth 3 Sep 2025 2:18 UTC

  2 points

  0

  in reply to: Dweomite’s comment on: When Both Peo­ple Are In­ter­ested, How Often Is Flir­ta­tious Es­ca­la­tion Mu­tual?

  I was also confused by that, and resolved it by adding some more salt on my already-salt-heavy view of the paper. I do not trust that these authors can do basic arithmetic, nor that they had a large enough sample for the numbers to be particularly meaningful anyway, but I do believe the large qualitative findings.

  So this is saying that actual female flirting has zero effect on whether her partner perceives her as flirting.

  Yup, I actually had a few sentences talking about that (I do find it very plausible, in the 10-minute-conversation setup used). I cut those sentences because I do not trust the numbers enough to update on something that precise.

[Question] When Both Peo­ple Are In­ter­ested, How Often Is Flir­ta­tious Es­ca­la­tion Mu­tual?

• johnswentworth 2 Sep 2025 5:08 UTC

  3 points

  0

  in reply to: the gears to ascension’s comment on: Dat­ing Roundup #7: Back to Basics

  Sounds like you might be able to provide a data point on the main question I’m curious about: roughly what fraction of people (or people of your preferred gender) mutually escalate at all?

• johnswentworth 2 Sep 2025 3:55 UTC

  4 points

  0

  in reply to: Vlad Loweren’s comment on: Dat­ing Roundup #7: Back to Basics

  That was my previous best guess at the norm! It’s what would make sense on priors, and what matches my observations. And it was what I would have expected if there hadn’t been such unanimous agreement in this thread that, no, mutual escalation is totally a normal thing.

  If that is what people mean when they talk about “the dance of flirtatious mutual escalation”, I wish they would describe it accurately—it’s a dance of “one person escalates and the other passively approves”. That is decidedly not a dance of mutual escalation, it is not a back-and-forth, and it requires both different models and different strategies.