My name is Charles Renshaw-Whitman. I am a physicist (‘symbol gremlin’) by training, currently a MATS scholar studying the connection between the structure of natural data and the structure of learned computations.
Currently training away an aversion to sharing my writing/thoughts publicly—please modulate tone of comments accordingly :)
thanks; “ime” = ‘in my experience’, if that is what you’re referring to? If not, sorry I don’t see it.
As a comment on identity in science: certainly it’s not about ‘personal/agentic’ identity, but I have been thinking a lot about how we draw boundaries between objects—like “when” a group of 3 quarks “is” a proton. Generally this involves specifying a “scale” parameter expressing ~ how much information we’re willing to lose in abstracting away from details—then we take, symbolically, the limit where this parameter goes to infinity. Then you can use perturbation theory to bridge lower-level effects to cash out at the higher-level (eg how quark structure effects proton collision statistics)
For systems of agents, the analogue is, from the perspective of the group-level, ‘coherence’, or ‘incentive-compatibility’ from the sub-agents’ perspective. Unfortunately we don’t really have the tools to do the analogue of perturbation theory in these more complicated cases. It seems like the salient difference is that we can’t really scalarize ‘incoherence’, as there are too many saliently distinct ways for a group to be ‘incoherent’ and no natural commensurability between them.
Yup! A math-ier version of the insight I was trying to convey is to say that, if you imagine dealing with math where you have a handful of these limit-y properties floating around, it may well matter what order you take those limits in! And at the same time that the limit-y-ness of these properties is usually quite “hidden”, so that this becomes a very useful mental motion.
Ahah I have Claude’s system prompt set to default to Chinese so I can practice. Since my speaking/writing abilities suck much worse than my reading, I also told it to nag me to write in Chinese for the sake of practice lol. This works… variably well.
Yup I definitely agree there’s no special role for unicellular attackers—I was eliding the complexity for brevity. I think the asymmetry still broadly holds meaningfully—e.g. multicellular parasites are very complex attackers but have much longer generation-times (I assume?), so they too trade off online vs offline optimization bits. Nonetheless the host organism still has more complexity to draw on for most things with which the immune system is concerned.
Interesting to think about the pareto frontier of offline vs online optimization. The multicellular parasites and unicellular microbes would be paradigm examples. But the microbiome gives lie to this idea—it is complex and organized but highly adaptive still because selection can act on the lower level. Perhaps being ~commensal/mutual instead of adversarial is related? I don’t know.
The linked Claude conversation doesn’t share the markdown file unfortunately. Apologies. Here is a gdrive link https://drive.google.com/file/d/1wpPGI7poP04ZMDoPU_lEh8D1CI8kOtic/view?usp=sharing I read it and it was a good introduction but it did a mediocre job of reframing things ‘as an optimizer’ or even ‘as a control system’.
Note: Skimming, Claude hallucinates what Alon’s ‘periodic table of diseases’ is. He has a pretty good youtube video on it you can watch instead. https://www.youtube.com/watch?v=ZMz_C778WMY&pp=ygUeYWxvbiBwZXJpb2RpYyB0YWJsZSBvZiBkaXNlYXNl
The Immune System as Anti-Optimizer
We have a short list of systems we like to call “optimizers” — the market, natural selection, human design, superintelligence. I think we ought to hold the immune system in comparable regard; I’m essentially ignorant of immunobiology beyond a few YouTube videos (perhaps a really fantastic LW sequence exists of which I am unaware), but here’s why I am thinking this.
The immune system is the archetypal anti-optimizer: it defends a big multicellular organism from rapidly evolving microbiota. The key asymmetry:
Specified once by the genome. It cannot rewrite its own source code between generations. Its adversaries evolve orders of magnitude faster.
Resource asymmetry as counterbalance. Whole organs are devoted to the adaptive immune response. A microbe is one cell; the immune system is a civilization.
This extends to cancer. The immune system typically out-adapts malignant cells despite selection acting far more rapidly on them. Immunosurveillance fails not because it is weak but because cancers occasionally evolve to exploit its specific tolerance mechanisms.
In short: the immune system embodies enough amortized optimization power to defend against online adversarial attacks by natural selection, because these attacks are constrained by the comparative simplicity of the attackers. One optimizer constrains another, faster and more adaptive optimizer, by having more resources.
What makes this especially interesting is that the immune system has no discernible volition. It is complex — probably far more so than I appreciate — but intuitively much more like a thermostat than a scheming eldritch god. It optimizes powerfully, within bounds that feel legible and non-agential.
I will not be so crass as to say “big if true for alignment”, but you are permitted to infer this if it please you. I just think it’s neat. Consider the mere phrase “semiotic immune system” (from, if I recall correctly, Charles Stross’s Accelerando) — suggests a lot at once, eh?
I asked Claude to prepare the following tutorial—which I have not yet read (longa est vita, si uti bene scias...) - developing this theme: https://claude.ai/share/67bb8de3-b73c-4a21-916b-70affba0da43
*Written with slight corrections for conciseness from Opus4.6. Ironically, the em-dashes are mine.
Something which I think highly relevant, and which might inform your GAN discussion, is the difference in performance of W-GAN—basically if you train the GAN using an optimal transport metric instead of an information theoretic one, it seems to have much better robustness properties, and this is probably because shannon entropy doesn’t respect continuity of your underlying metric space (e.g. KL divergence between Delta(x0) and Delta(x0 + epsilon) is infinity for any nonzero epsilon, so it doesnt capture ‘closeness’). I don’t yet know how I think this should tie into the high-probability latent manifold story you tell, but it seems like part of it.
I agree Thom’s work is interesting and relevant here; I’ve seen it bruited about a lot, but likewise haven’t gotten around to seriously studying it. From my perspective, structural stability is extremely important, but I am curious if there is any reason to identify this kind of stability with catastrophes? I’ve always looked at the classification and figured its just one example of this kind of structural stability, and not one I’ve really even seen much of in physics, compared to the I-think-close-but-not-exactly-the-same more general idea of critical phenomena and phase transitions. I’d be very interested if there was some sort of RG-based stability analysis.
The dynamical systems way of framing this I like the most is Koopman analysis; Steve Brunton has a great family of talks on on youtube: https://www.youtube.com/watch?v=J7s0XNT96ag
Tldr is that the Koopman operator evolves functions of the dynamical variables; it’s an infinite dimensional linear operator, thus it has eigenfunctions and eigenvalues. Conserved functions of state have zero-eigenvalue; nearly conserved quantities have ‘small’ eigenvalues, corresponding to a slower time rate of decay. Then you can characterise all functions in terms of the decay rate of their self-correlation. So the dominant koopman eigenvalue of a (not necessarily eigenfunction) is a scalar value corresponding to how “not conserved” it is.
You can also look at the Jordan blocks, which correspond to groups of functions which are predictively closed (you can predict their future values by knowing only the present values of those functions). These are some threads I’ve found interesting in thinking about natural ontology but I do not think they are sufficient—for example they give a very clear picture of intra-realization correlations, but do not have a good way of talking about inter-realization correlations or sensitivity to initial conditions.
strictly weaker
add “than Condorcet” in this sentence since its only implied but not said
I think we agree modulo terminology, with respect to your remarks up to the part about the Krakovna paper, which I had to sit and think a little bit more about.
For the Krakovna paper, you’re right that it has a different flavor than I remembered—it still seems, though, that the proof relies on having some ratio of recurrent vs. non-recurrent states. So if you did something like 1000x the number of terminal states, the reward function is 1000x less retargetable to recurrent-states—I think this is still true even if the new terminal states are entirely unreachable as well?
With respect to the CNN example I agree, at least at a high-level—though technically the theta reward vectors are supposed to be |S| and specify the rewards for each state, which is slightly different than being the weights of a CNN—without redoing the math, its plausible that an analogous theorem would hold. Regardless, the non-shutdown result gives retargetability because it assumes there’s a single terminal state and many recurrent states. The retargetability is really just the ratio (number of terminal states) / (number of recurrent states), which needn’t be greater than one.
Anyways, as the comments from Turntrout talk about, as soon as there’s a nontrivial inductive bias over these different reward-functions (or any other path-dependence-y stuff that deviates from optimality), the theorem doesn’t go through, as retargetability is all based on counting how many of the functions in that set are A-preferring vs. B-preferring—there may be an adaptation to the argument that uses some prior over generalizations and stuff, though—but then that prior is the inductive bias, which as you noted with those TurnTrout remarks, is its own whole big problem :’)
I’ll try and add a concise caveat to your doc, thanks for the discussion :)
Hi thanks for the response :) So I’m not sure what the distinction you’re making between utility and reward functions, but as far as I can tell we’re referring to the same object—the thing which is changed in the ‘retargeting’ process, the parameters theta—but feel free to correct me if the paper distinguishes between these in a way I’m forgetting; I’ll be using “utility function”, “reward function” and “parameters theta” interchangably, but will correct if so.
I think perhaps we’re just calling different objects as “agents”—I mean p(__ | theta) for some fixed theta (i.e. you can’t swap the theta and still call it the same agent, on the grounds that in the modern RL framework, probably we’d have to retrain a new agent using the same higher-level learning process), and you perhaps think of this theta as an input to the agent, which can be changed without changing the agent? If this is the definition you are using, then I believe your remarks are correct. Either way, I think the relevant subtlety weakens the theorems a fair bit from what a first-reading would suggest, and thus is worth talking about.
I’ll outline my meaning more below to try and clarify:
The salient distinction I wanted to make was that this theorem applies over something like ensembles of agents—that if you choose a parameter-vector theta at random, the agent will have a random preference over observations; then because some actions immediately foreclose a lot of observations (e.g., dying prevents you from leaving Moctezuma temple room 1), these actions become unlikely. My point is that the nominal thrust of the theorems is weaker than proving that an agent will likely seek power; it proves that selecting from the ensemble of agents in this way will see agents seek power.
The difference is that this doesn’t super clearly apply to a single agent. In particular, in real life, we do not select these theta uniformly-at-random at all; the model example is RLHF, the point of which is, schematically, to concentrate a bunch of probability mass within this overall space of thetas to ones we like. Given that any selection process we actually use should be ~invariant to option variegation (i.e. shouldn’t favour a family of outcomes more just because it has ‘more members’), this severely limits the applicability of the theorems to practical agent-selection-processes—their premises (I think) basically amount to assuming option-variegation is the only thing that matters.
As a concrete example of what I mean, consider the MDP where you can choose either 0 to end the game, or 1 to continue, up to a total of T steps. Say the universe of possible reward functions is the integers 1 to T, representing the one timestep at which an agent gets reward for quitting, getting 0 otherwise. Also ignore time-discounting for simplicity. Then the average optimal policy will be to go to T/2 (past which point half of all possible rewards favor continuing, and half would have ended it earlier) - however this says nothing about what a particular trained agent will do. If we wanted to train an RL agent to end at, say, step 436, even if T is 10^6, the power-seeking theorems aren’t a meaningful barrier to doing so because we specify the reward function rather than selecting it from an ensemble at random. Similarly, even though a room with an extra handful of atoms has combinatorially many more states than one without those extra atoms, we wouldn’t reasonably expect an RL agent to prefer the former to the latter solely on the grounds that having more states means that more reward functions will favor those states.
That said, the stronger view that individual agents trained will likely seek power isn’t without support even with these caveats—V. Krakovna’s work (which you also list) does seem to point more directly in the direction of particular agents seeking power, as it extends the theorems in the direction of out-of-distribution generalization. It seems more reasonable to model out-of-distribution generalization via this uniform-random selection than the overall reward-function selection, even as this still isn’t a super-duper realistic model of the generalization, since it still depends on the option-variegation. I expect that if the universe of possible reward functions doesn’t scale with the number of possible states (as it would not if you used a fixed-architecture NN to represent the reward function), this theorem would not go through in the same way.
Let me know your thoughts :)
Remarks on the Slow Boring post about “Kritiks” in debate, copied here for a friend who wanted to reference them, and lightly edited to fit the shortform format:
In the format of debate I do (“British Parliamentary” (BP), also my favorite, having also done “Public Forum”), this sort of thing is sorta-kinda explicitly not allowed. Ironically, I think BP is somewhere where this would be most appropriate—because you don’t have the ‘motion’ until 15 minutes prior to start, focus on ground-facts is low (only relatively, compared to other research-heavy styles); you spend a lot of time arguing about “moral weighing-criteria” and “framing” and things like this.
We do have a pretty clear line where you can, say, go full-Marxist in saying some particular impact is the most important—you have to defend this framing, not just assert “Marxism says X”—but you definitely can’t tell the motion to go fuck itself. (A late-debate reframing is called a “squirrel” and only allowed if the opening-half was batshit insane with the framing).
Specifically, you pretty much can get away with saying “zoos are anthropocentric and that’s bad because...”, but then you have to justify that and “weigh it out” against the other cases people bring. That is, you can win with a case that would be Kritik-adjacent (i.e. radical but you still can’t challenge the motion), but you then have to argue why that’s the relevant framing of the debate, why it’s the most important thing, etc.
I only know EU debate for BP, so it’s possible its just that US debate as a whole is weird, but I don’t think so since I’ve seen e.g. Princeton compete. In the slow-boring article they note that its worst in “Policy” debate which sounds reasonable since that’s always been the most Goodharted format (I’ve heard of classes where they teach you to speak at 350 WPM). Also worth noting that I never debated at an especially high level, and only in the Netherlands, but I’ve watched a few world-finals debates and some online workshops and nothing Kritik-flavored has come up.
I find BP is actually one of the best antidotes to Twitter hot-takes, in part I’d guess because framing etc. is an integral part of the debate anyways, so what would be Kritik “flip-the-table” BS in Policy can be (is required to be) much more productive in BP. Perhaps this is the structure of the format, but it may also just be the community.
Yeah indeed I think my engagement profile was different from the more competitive attitudes you’d expect high-powered teams/unis to have—our debating club was a small one at a school self-consciously focused on engineering, so it was much more hobbyist. I was always in it for self-growth rather than winning, so it served as a good check on the kind of intellectual hubris that accrued as I was levelling-up in physics. We were fortunate to have a handful of emerita members that could pass on a lot of their knowledge without expecting a big commitment to competition performance. But I’m not surprised (and sorry to hear!) that there are suckier steady-states.