My name is Charles Renshaw-Whitman. I am a physicist (‘symbol gremlin’) by training, currently a research fellow with Iliad. I study the foundations of interpretability with the aim of building better ontologies of intelligence.
My personal website is here:
https://charlesr-w.github.io/crw-blog/
CarolusRenniusVitellius
Karma: 227
Some early mixed-genre thoughts on plasticity and crtical periods in neural networks, and as a motif in agent-ology. Compiled in part based on work done at MATS 9.1 under the mentorship of Richard Ngo, and the Iliad fellowship, under the mentorship of Dmitry Vaintrob.
Plasticity and Critical Periods
Plasticity dynamics, especially ‘critical periods’, seem to be important in neural networks, and they happen generically enough [1] that it feels to me like an instance of a more general phenomenon. In animals, in NNs, and, if you’re sufficiently Quigley-pilled, institutions and civilizations. [2]
Roughly, one needs high-plasticity periods in order to explore many configurations — to try out many candidate circuits, strategies, or institutional arrangements — but then one needs to tune down plasticity in order to hone in on good solutions. [3]
In Markets and Coalitional Agents
One way of thinking about markets is as information aggregators. Profits are accrued as a result of a history of correct predictions or useful actions. Geometric Rationality makes this more literal by showing that Bayesian updates can be regarded as Kelly betting of credibility-points among hypotheses.
Real-life markets typically allow for a state of “bankruptcy”: a threshold level of performance below which agents are permanently excluded from participation. [4] This is a problem to the extent that one wishes to maintain diverse information inputs to the market. The question of plasticity is then the question of how to allow for such sensitivity while excluding true nonsense participants. In practice, such a mechanism will always involve some kind of subsidy paid to underperformers, in the hope that some of them will one day make good. I’m not aware of a great analogy for critical periods within markets, and would be interested to hear any thoughts.
In real life, agents can maintain inconsistent beliefs. This in part stems from the fact that small inconsistencies may not merit the computational cost of full belief repropagation. But plasticity provides an interesting alternative perspective: one might prefer to subsidize conflicting families of beliefs in the hope that this diversity will cash out as an overall ability to assimilate novel information.
That is, in the absence of perfect Bayesian updating, one might like to have one’s belief network exhibit some amount of inconsistency in order to more readily integrate surprising observations. On this perspective, competition between belief-families-qua-agents is counterbalanced by an inconsistency subsidy, so as to maintain the whole network at a more adaptable equilibrium. [5] I haven’t yet thought enough about critical periods in belief networks to say what this might correspond to—certainly one wants a minimum amount of consistency, but it’s not clear that this should correspond to a critical-period in the time-dynamics of the belief network.
Coda on Neural Darwinism
I have been thinking a lot about neural Darwinism as an ontology for neural networks over the past few months, and I think “network plasticity as a hedge against Knightian uncertainty” is a more versatile extension of that idea, one which does not require having particular units of neural selection. I’ll be exploring this in the coming days; I’m excited by some interesting connections to HTSR theory and more schematically, to lazy-vs-rich learning viewed through kernel dynamics (since the kernel is dual to the Fisher-information matrix, a proxy for plasticity). Hopefully more to come.
- ↩︎
I have been reading, inter alia, these papers: 1711.08856, 2210.04643, and 2308.12221.
- ↩︎
Doubtless there is no monopoly on this model; Quigley is just the one who comes to mind. Samo Burja is another impressive thinker in this genre.
- ↩︎
This is a fully generic bias-variance tradeoff argument; there is a more precise story to be had, but I am still thinking my way to it.
- ↩︎
There are two different reasons to dislike bankruptcy. The first is as a non-ideality in the market, which I think rational agents can simply route around by Kelly betting. The more interesting reason is Knightian: in practice, market-like mechanisms do not effectively account for very rare events. Cf. the tangential but excellent ACX post, “Heuristics That Almost Always Work”.
- ↩︎
Cf. the fact that biology much prefers opponent-process models to binary switches.
- ↩︎
Conditioning on the arisal of highly-capable long-term coherent planning (i.e., ‘superintelligence’), I think the only difference between these perspectives is where one expects the effective ‘locus of desires’ to be—in the weights or in the activations.
In human-alignment, we might contrast the brain-qua-architecture with the actual brain of a particular person: the brain-qua-architecture has lots of goals (eating, etc.), and so too people have particular goals as instantiations of that architecture (money, power, etc.). The former architecture-level misalignment has solutions which look like Confucius, Hobbes, etc., while the latter ‘persona’-level alignment looks like making said person read Cicero until they know what True Virtue means.
I think a putative LLM-based superintelligence could have its locus of desires either baked-in or instantiated as a persona, but the causes and remediations to either of these as engineering problems must be quite different.
Perhaps a deconfusion would be to dissolve the notion of personae into clouds of correlated beliefs and action—there is a face beneath the mask only to the extent that behaviour deviates from that desired by the ‘mask’ while serving that of the ‘face’. But I don’t think this is a complete description—incoherence or coexistence of different loci of desires seems important. Very Janusian.
Some misc. thoughts on how compute constraints might inform a belief hierarchy perspective on agents:
I think it might be helpful to think of self-/other-modeling as solving an optimization problem under computational constraints. Specifically, if you have an amount of compute
This gets you naturally into talking about things like Q-learning or actor-critic architectures : if the hierarchy has to bottom out somewhere, the bottom layer should call some low-resolution proxy for what self/other would do—so the belief hierarchy might well be implemented by something that looks like an analogue of AlphaGo’s MCTS + value-backup training. I don’t expect this analogy to be very good for describing a real ideal agent, but I do expect that some of the fundamental problems of RL may be shadows of more platonic constraints. Two examples:
-
RL-like learning rules tend to be unstable—stabilizing them looks like slowing down some updates over others with some kind of ‘ballasting’ function like an exponential-moving-average. This may be the shadow of something like a pressure to have some slow-changing “identity” which stabilizes the recursive update. This would explain, e.g. an evolutionary pressure towards having a well-defined sense of identity and an aversion to changing it too quickly.
-
RL-like algorithms also have to deal with the computational cost of propagating updates when those updates are individually too unimportant to justify the expense of completely revising the value function. The natural solution is some kind of N-step backup or TD-
To the best of my understanding, an actions-as-predictions paradigm should have to face the same fundamental dynamic—it’s not an RL-exclusive pathology. C.f. things like AMP for Bayesnets and LIR for PDGs for cases where one faces similar constraints in a more probability-theory-y setting. I think my point is that it may be valuable to think about this sort of technicalia as representing shadows of more fundamental agent-design constraints, and to see if one can pin down what things are fundamental vs. engineering-/regime-dependent.
-
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.
These are notes from my research undertaken as part of the Iliad Fellowship under the mentorship of Dmitry Vaintrob.
I sat down and thought silently for 2h about what needs to happen in order for us to understand neural networks. Summary below:
-
We must understand structure in terms of the neural network itself, not by trying to project this structure onto things we can interpret. If it is interpretable, then the structure should tell us so—spectral gaps, isolated/sparse circuits, etc. A better understanding of NN structure doesn’t have to mean better interpretability—they’re correlated only to the extent that the actual structure is made up of interpretable units; if NNs are made up of circuits, we should see them, if not, we should not see them. NNs obviously have sufficient structure that we can understand them better—build a mental model robust enough to predict the median ICML ‘neural networks sure are weird arent they’ paper. Module criticality and ‘transformers as painters’ are my measuring-stick for this kind of mental model building.
-
We want meso- and macro-ontologies (e.g., circuits, personae, ‘capabilities’). Strong-opinion-weakly-held: if it were possible to construct these ontologies well without starting from the micro-scale, we would have succeeded already. We need robust handles for thinking about behaviour on the micro-scale—we will have to abstract away from bare-parameter/activation reasoning bit by bit. It is best if we can justify these ontic units as being somehow ‘isolated’ or ‘discrete’; the archetype I have in mind is the BBP spike transition—it is sufficiently robust that one can plausibly define signal deteciton/recovery as in reference to BBP, and this definition will serve well in some regime perturbed around BBP. If we are careful when we construct these ontic units, they will come labeled with a asymptotic regime of validity. This tells us where to look for the next level of abstraction. For example, if BBP gives us a definition of “detected, localized signal”, the breakdown of BBP in the many-signal regime tells us where to look; hopefully, this breakdown, construed as a phase transition, gives us another ontic unit to work with, and lets us proceed. (my pet theory for the last three days, perhaps this breakdown of localized signals is precisely the ‘delocalization’ of knowledge inherent in going from memorized patterns to a connected world model).
-
Noise and training dynamics seem fundamental, not incidental. I hypothesise that, at equal loss, an NN trained under full-batch gradient-flow will be essentially useless at extrapolating OOD (i.e., would suck at being finetuned to a downstream task—not merely generalizing to unseen data from the same distribution). Two perspectives clash here: one is data-essentialist, that an NN has finite capacity, and there’s a particular optimal way to represent that data given fixed capacity, and that way happens to be sparse/hierarchical/factored representations, perhaps on account of some sort of ‘simplicity’ prior—the ‘golden path hypothesis’ is in this genre. Second, there’s a structure-essentialist view, wherein the dynamics of training are the main thing which determine the nature of representations learned, with a secondary role for the data. I lean towards the structure-essentialist view—the neurons just want to learn man. My working mental model is much closer ‘neural Darwinism’, natural selection acting on competing species (competing ‘circuits’) - the abiotic environment (the data) matters, but the biotic environment is shaped ‘more’ by its own internal logic of competition and ecology. For example, perhaps a globally-shared ‘world-model’ is selected for not by compression, but by being so universally-used in computations that even mediocre performance would make any gradient more harmful than useful.
-
I also expect a notion of plasticity and critical-periods will be important—nature didn’t implement these in humans for fun; literature seems to point to early/middle/late layers as corresponding to “feature-extraction / generic computation / task-specific”, with the early and late layers deplastifying early in training (I’m unsure of this interpretation). It seems like some kind of plasticity control is important, and that we just mostly never realized this on account of NNs having figured out how to half-assedly implement this on their own - ‘divine benevolence’ and all that. Possible relation here to EGOP/AGOP/NFA, but I don’t know.
-
A huge percentage of brain machinery seems to be dedicated towards managing stacks of different regulatory loops, and it is probably the case that enhancing intelligence, cognition, etc. corresponds more to precisely tuning an interlocking system of regulatory loops than to having any difference in the basic machinery of neural function. I think there are convergent reasons to expect this to be true of NNs (even aside form the critical-period literature) - probably most of training is best thought of as ‘tuning’. This also makes model addition/Polyak averaging make a bit more sense—if
-
(Naive) weight-based analysis should be extremely hard, probably usually impossible. the natural units of analysis may instead be weights-in-context—pairs
-
I’m less sure about this, but I think we should only rarely think in terms of ‘moment bias’ - I wrote down an EFT formalism where increasing data let one ‘resolve’ progressively higher moments of a data distribution—a la ‘sliding down the stairs’ - then I stared at this and realized there’s absolutely no way one could use this to ever find a ‘cat’ concept without making assumptions tantamount to solving the problem. Asserting that high level semantic concepts should live in higher moments of the data-distribution is vacuously true—iirc M’th order moments take exp(M) samples to estimate. Insofar as NNs seem to have some universality in the things they learn, I’m fairly sure one could not predict this universality from the data-distribution/geometry, short of training NNs on it—I think it comes down much more to “inductive bias”. Obviously, caveats to this for fine-tuning a trained neural network.
What specific next-steps are implied by this?
Things I need to learn/read more: NNMFT, plasticity and critical-periods in biology and ML, Liquid-state machine model of brain. Log-linearity and supposed approximate equivalence between fine-tunes/system-prompts/steering-vectors, etc.
Think more about: Quasispecies models as ways to operationalize ‘units of selection’; different sources of ‘noise’, when they are equivalent, and when should this have anything to do with ‘linearity’ in transformers? Relationship between evolved noise-robustness and plasticity?; BBP transitions and Saxe-dynamics.
I think the most promising things to think about are phase transitions of every shape and flavour, as these are the most actionable, and the things for which I can most readily evaluate validity. This may entail learning MFT. The ‘plasticity’ / ‘critical period’ idea features more heavily in my model than I had realized, and I should try to understand this better too. My heart says these must be deeply related to ‘structured noise’ (e.g., one module becoming robust to ‘noise’ from another == loss of plasticity?). Hand-wavily, ‘critical periods’ seem like a good thing to mine for my beloved phase-transitions.
-
I’ve been thinking today about problems of self-reference in agent foundations. I wanted to share a toy model I came up with for the emergence of coherent identity in an agent (of course, for values of those words which make them correspond to the things I’m about to define, so caveat emptor). Roughly, factoring the world into (agent) x (environment) induces a very natural way to divide patterns of behaviour into so-called ‘motivational orbits’, and particular such orbit is selected for in the long-time limit. The math maps cleanly onto natural selection acting between these ‘motivational orbits’ - thus they are a very natural way to divide an ‘agent’ into competing subagents. Further, there is a natural way to assign a value-function-like object to each of these subagents; the long-term-selected orbit then has a single coherent value-function.
As ever, I’m more motivated by the mathematical archetype and less with having the model produce a particular phenomenon—its one of those things which I think would make a good starting point for an ontology in a more complete theory. Other than the standard RL terminology, you can look at quasispecies for the inspiration to this model.
Example in words: Bob is a good Calvinist who tends to wake up energized at 4am, work until 7pm, and then eat a plain dinner and pray until bedtime at 8pm. One day, he sleeps very poorly because of a thunderstorm or something—he wakes up ill-rested and demoralized, watches TV for a bit, and generally hangs around for the rest of the day. He sleeps well, and is ‘back at it’ the next day.
Bob as an entity may want or feel different things throughout this process, both on his good days and his bad days. He may want to work, or to watch TV, or to pray. However there is a sense in which the ‘routine’ days are more ‘stable’, and to the extent that he regularly returns to his routine after some perturbation, we might be justified in considering them more ‘representative’ of Bob as a person. For our purposes “Bob on a good day” and “Bob on a bad day” are subagents, and they ‘compete’ to be expressed more frequently (i.e., over a long enough period of time, we see more of the one which is more stable).
Below I present a model where something like certain ‘routines’ are the natural joints at which to carve the world; so carved, these ‘routines’ undergo a natural selection process until only the dominant one is left. I present this as a toy model for thinking about the emergence of a coherent identity. I think its worth considering because the quasispecies idea is a cool-and-maybe-useful joint-carving motif.
We consider a markov chain (not an MDP!) defined over the cross product (internal state, external state), call these
These
So if you as an external observer have some initial prior over the agent-environment joint state
If you want to go a bit further and predict the agent’s internal state, you can take this distribution and condition on
Worth noting we can also define, for any prior
GPT-5.5 also claims you can fiddle and multiply the internal-state transition function
I’ll take a look at the ProRL paper later today, thanks for the second.
I agree that RL inefficiency is one problem but I think this can be reasonably factored out in experiments if not in production. The “RL Razor” paper does an experiment where they do SFT on a KL budget and show they get the same ‘reduced forgetting’ effect—I think of this reduced forgetting, in light of the ‘off the principals’ paper, as being something like inertia of learning new representations, or inability to pass through regions of high curvature. That is, there are definitely still qualitative differences at the per-batch level; perhaps this is just an efficiency thing, but it’s plausible to me that it might be more like GD vs SGD in finding different types of solutions with different generalization properties because of the curvature bias. On priors this would surprise me for RL, but I guess I like these papers because they updated me away from that a bit.
As for the milestone self-play results, you’re right that they’ve no place in this story—my semi-cope pro-tem guess is that LLMs operate in a ‘different regime’. Two intuitions for this:
-
for the board games especially, there is no ‘curriculum of representations’ - perhaps learning strategic action is easier for RL than learning to have good ontic chunkings of the world. E.g. for Go, there is no need to learn complex hierarchical representations as table-stakes. The harder Atari games (eg Moctezuma) are counter-evidence to this, except perhaps insofar as their hardness was due to them needing more complex representations? This is an even poorer explanation for something like AlphaStar. But nonetheless, we ended up switching to the pre-training paradigm instead of riding A3C to ASI.
-
The relatively poor performance of things like process supervision for LLM training is still surprising to me and I can’t account for it. If value-learning methods ‘don’t work’ in the LLM regime, is this because there is a structural difference in the data? On priors this just has to be a skill issue, but presumably had someone really solved this and gotten 3-5 OOMs of RLVR efficiency, wouldn’t we be done already? And presumably enough money-effort has been expended that were dramatic success achievable, it’d’ve been done? (To be clear here, by ‘work’ I mean “work so well that per-token gradients aren’t more than a one or two OOMs worse than for pre-training”)
-
Indeed they’re the same—the ‘drift-diffusion’ is the name of the same thing in physics and ML. I’ll add a note, thanks.
Cool work! I am curious if you have in mind any ‘natural kinds’ which you might be able to pull out of the tensor structure? I agree that TNs seem pretty promising for interp due to their tractability, and seem fairly clearly to be in the same ‘universality class broadly construed’ as transformers. My concern in applying this would be that, iiuc, it doesn’t natively incorporate a measure on output space, so for LLMs you’d get a disproportionate fraction of dissimilarity coming from behavioural differences on random strings (I think this is a fundamentally hard problem and you seem clearly aware of it, so no shade thrown). I have been thinking about local kernels like the eNTK (= the cosine similarity of output-gradients as a function of two samples) lately for this reason, but would be super excited if you could use TNs to trade the weight-space-local-but-linear eNTK for a global multilinear TN sorta thing? And then perhaps tensor-decomps would yield some objects with a comparable claim to semantic meaning as the eigenvectors of the eNTK?
A quick argument for why ‘consciousness’ (a central executive function) might be a convergent property of intelligent behaviour (i.e., why Claude might be/come conscious):
5-word summary: Bottom-up hierarchy requires orchestration
Evolution operates at a finite speed and resolution --> most of the structure in complex organisms are not encoded by the genome directly—instead, evolution often follows a motif:
Self-organizing progenitor units : typically the result of a lower level of this process.
Interaction between progenitor units (usually with some dichotomy like ‘excite/inhibit’, ‘activate/suppress’)
Regulatory signals mediating interaction betwen the progenitor units
Differentiation of progenitor units according to interactions + regulatory signals
Example: Neurons --> Neural circuits
Progenitors: Neurons
Competitive/Inhibitory Interactions: Neural signaling + Hebbian learning
Regulatory signals: Nerve growth factors (presumably there are ~infinity other regulators)
Differentiation into neural circuits via winner-take-all dynamics.
At the level of the unit, neurons and their synapses are built in a way that they will create differentiated ‘circuits’ by default via Hebbian learning and random synaptic connections. This is ‘self-organization’. Excitation/Inhibition dichotomy is important because it allows organization to come from ‘balancing’, which is generically more robust: 1 + (-1) = 0 is different from 100 + (-100) = 0 when your noise is O(1). This also makes gating effective, as a small gating signal can now direct a large response, as for transistors.
The genome converts this into useful functionality by tuning this self-organization process. It controls inputs, outputs, and the local ‘rules of the game’ - using proteins connected to signalling networks accessible to the genome. Input optic nerve, output to V2, mix in some (heinously complex pattern of growth signals) --> get V1 visual cortex.
This is ‘facilitated variation’ - imagine evolution had encoded the connectome in its entirety—then every mutation would mess things up, just as randomly flipping bits in your laptop’s RAM would. Bad, but also unlearnable—evolution needs useful mutations to learn. Instead, mutations affect the signalling program, and this generates more ‘meaningful’ variation. Morally, hierarchical organization lets the genome create N levels of structure in O(log N) complexity.
The major difference from computers: every layer in this process has to equilibrate simultaneously with its constituents and its peers—when a circuit updates in response to other circuits, its constituent neurons must instantiate this macro-level update and must re-equilibrate with one another (e.g., maintain a balance of inhibitory and excitatory signalling). This means that tuning a node at level L could in principle cause O(e^L) updating ‘bottom’ - constraining that exponential is presumably an engineering desideratum. This is a model which makes gated development a la Piaget somewhat natural—you do your best at level L, then lock it in before moving to L+1.
If this is ‘bottom-up’, computers are ‘top-down’ - you can design N layers of abstraction in O(log N) effort by tiling ever-larger circuits; the difference is that this structure is specified at design time by engineering away lower-level variation, rather than taming it. There is no adaptation—bit flips are bad with probability ~1. The cost of designing M variants of a circuit is O(M), while evolution does this by default.
Application to cognition: posit the existence of ‘subminds’, the result of lower levels of this process. ‘The Mind Illuminated’ and ‘Internal Family Systems’, i.a., seem to have models of the mind based on this idea. My guess is that these subminds are not well thought of as spatially localized regions of the brain—I have more in mind something like ‘processes competing for thread time allocated by the OS’
I’d sketch as: Progenitor: sub-minds Interactions: ??? Regulatory signal: Conscious/executive attention (or something like that) Differentiation: ??? (the ‘mind’ in this case is the ‘top of the hierarchy’, maybe?)
My point is much less the (absent) particulars, and more “hierarchical organization by encoding regulated self-organization seems to be the default way to generate useful complexity at scale. If you think LLMs trained by SGD learn some kind of hierarchical structure or representations, it’s a small leap to imagine gating mechanisms that look like metacognition.”
Note that this obviously depends on your model of how/what LLMs learn. And note this doesn’t imply valence, moral patienthood—“whereof we cannot speak...” etc. See also the Anthropic paper on ‘emergent introspection’ - I don’t yet have an opinion on if this is evidence for the model I discuss above, but it has informed my thinking.
(This note is based on conversations with Lorxus and Richard Ngo. It sprung from my research in MATS 9.1. To all of whom my thanks.)
Hey commendations on sharing your update.
Another similar line of work I like is Roberts+Yaida’s “Principles of Deep Learning Theory”—this is a similar-in-spirit approach to MFT, but they perturb around a different limit and get feature-learning as a finite-width effect. I haven’t studied MFT to compare the validity of the two; my guess is MFT is the more relevant description. PDLT at least does a very good job modernizing the NTK approach and connecting to the older literature. I’m a fanboy as it was my gateway drug for learning theory lol.
I find the framing in your review is somewhat odd - I think the state of ‘deep learning theory’ is fairly impressive and that its sterility vis-a-vis frontier LLMs is a hint that we are looking in the wrong place. Early-2026 DLT is a major piece of evidence that we need more data-centric theory, precisely because sophisticated theory has had so much trouble connecting to the frontier. If we had worse theory, we would be more uncertain if the relevant complexities were to be located in the data or in the learning process.
Two analogies I have in mind that guide my thinking here:
Understanding the physics of animal neurons is necessary to understanding neuroscience. The complexity of brains exists at a ‘higher level’ than individual neurons, but understanding neurons carves out how much of that complexity can be subcellular vs. being in their larger-scale organization. In the same way, something like LLM behaviour is a product of training process, data, etc., and a good theory of learning lets us ask what part of that complexity belongs where.
Statistical physics as a formalism provides a family of techniques for analyzing physical systems with many degrees-of-freedom. Its great intellectual triumph is the discovery that some things depend on the details how how those degrees-of-freedom interact, while others do not: behaving-like-a-gas is a highly generic property, but material properties like fatigue and fracture-strength can depend quite sensitively on the specifics of the sample in question. The key thing is that we can try to figure out which properties are ‘universal’, and which are not. In DLT, I think it’s much more like a spectrum, as there are many more knobs to tune—data, gross and fine architecture, hyperparameters etc.
Of note to me is that most of the successes of DLT have been at the level of structural depth you’d expect from studying neurons as relate to brain function. E.g., the average neuron firing must activate on average exactly one neuron (lest one die or have a grand mal; comparable to the ‘edge of chaos’ in DLT). These are pretty coarse results more to do with signal processing than with structured computation. It is still illuminating to see that these coarse results hold because they validate our mental models (‘however this neural network is computing stuff, it still has to navigate some kind of signal/noise tradeoff in its activations’).
Insofar as the intelligence of LLMs is the ability to generalize, the no-free-lunch theorems tell us that this generalization has to reflect common structure of the pre-training data and the fine-tuning task (duh!). But our theory of data isn’t yet advanced enough to talk more than proleptically about that structure—e.g. a claim that fine-tuning is ‘conditioning nodes in the common sparse hierarchical latent world model’ is descriptive and substantive, but not enough that it is easily falsifiable. “Write Ruby code” and “Write Python code” are obviously more similar to each other than “design a jet turbine”, but given only a black-box loss-function for each of those 3 tasks, it’s not that clear how we could principledly determine that similarity a priori from the ‘geometry’ of the functions alone.
Yeah indeed I think my engagement profile was different from the more competitive attitudes you’d expect high-powered teams/unis to have—our debating club was a small one at a school self-consciously focused on engineering, so it was much more hobbyist. I was always in it for self-growth rather than winning, so it served as a good check on the kind of intellectual hubris that accrued as I was levelling-up in physics. We were fortunate to have a handful of emerita members that could pass on a lot of their knowledge without expecting a big commitment to competition performance. But I’m not surprised (and sorry to hear!) that there are suckier steady-states.
thanks; “ime” = ‘in my experience’, if that is what you’re referring to? If not, sorry I don’t see it.
As a comment on identity in science: certainly it’s not about ‘personal/agentic’ identity, but I have been thinking a lot about how we draw boundaries between objects—like “when” a group of 3 quarks “is” a proton. Generally this involves specifying a “scale” parameter expressing ~ how much information we’re willing to lose in abstracting away from details—then we take, symbolically, the limit where this parameter goes to infinity. Then you can use perturbation theory to bridge lower-level effects to cash out at the higher-level (eg how quark structure effects proton collision statistics)
For systems of agents, the analogue is, from the perspective of the group-level, ‘coherence’, or ‘incentive-compatibility’ from the sub-agents’ perspective. Unfortunately we don’t really have the tools to do the analogue of perturbation theory in these more complicated cases. It seems like the salient difference is that we can’t really scalarize ‘incoherence’, as there are too many saliently distinct ways for a group to be ‘incoherent’ and no natural commensurability between them.
Yup! A math-ier version of the insight I was trying to convey is to say that, if you imagine dealing with math where you have a handful of these limit-y properties floating around, it may well matter what order you take those limits in! And at the same time that the limit-y-ness of these properties is usually quite “hidden”, so that this becomes a very useful mental motion.
Ahah I have Claude’s system prompt set to default to Chinese so I can practice. Since my speaking/writing abilities suck much worse than my reading, I also told it to nag me to write in Chinese for the sake of practice lol. This works… variably well.
Yup I definitely agree there’s no special role for unicellular attackers—I was eliding the complexity for brevity. I think the asymmetry still broadly holds meaningfully—e.g. multicellular parasites are very complex attackers but have much longer generation-times (I assume?), so they too trade off online vs offline optimization bits. Nonetheless the host organism still has more complexity to draw on for most things with which the immune system is concerned.
Interesting to think about the pareto frontier of offline vs online optimization. The multicellular parasites and unicellular microbes would be paradigm examples. But the microbiome gives lie to this idea—it is complex and organized but highly adaptive still because selection can act on the lower level. Perhaps being ~commensal/mutual instead of adversarial is related? I don’t know.
The linked Claude conversation doesn’t share the markdown file unfortunately. Apologies. Here is a gdrive link https://drive.google.com/file/d/1wpPGI7poP04ZMDoPU_lEh8D1CI8kOtic/view?usp=sharing I read it and it was a good introduction but it did a mediocre job of reframing things ‘as an optimizer’ or even ‘as a control system’.
An image I have found inspiring as a researcher in the age of LLMs:
A samurai orients their entire life towards preparing for a few decisive moments of combat. Awakening at dawn for hours of grueling daily combat training, certainly.
But also an intensive study of calligraphy, classical literature—why?
In that decisive moment of combat, whose movement is the more fluid, who by sense of righteousness is the more unflinching, they are the victors. The fluid breathing of the calligraphy brush, the moral fibre wrought by the classics—in that moment a lifetime of study is vindicated or revealed worthless.
Why study calculus if Claude already knows a hundred or a thousand times more math than I will on my deathbed? Not because I anticipate finding Claude has made a mistake in integrating by parts, but so that my mind and intuition are sharp—so that I know exactly what to ask, what to look for, and exactly what a good answer should look like—to know what kinds of analysis can reasonably yield what kinds of conclusions—to know on a thirty second skim whether Fable 5′s argument is natural or sidestepping the whole problem.
Perhaps this is just ‘taste’, but a large comparative advantage we have now over LLMs is choice in what we think about. LLMs are, if not constrained, then ‘conditioned’, by their starting prompt. Asked to solve a problem in economics, it will likely reason about economics and not biology—or if it does unprompted, any such connection will often-as-not be superficial obsequy, Fable 5 trying very hard to convince you that it is exactly ten IQ points above you. If you want some ex-ante-crazy connection between economics and biology, you have to know what you’re looking for in advance.
Object-level knowledge can be backfilled easily enough—Sol is excellent at writing bespoke tutorials. I think it is highly valuable now to read not for knowledge, but to see and internalize powerful thinking in action—more power to be found reading ‘the wealth of nations’ or ‘on the origin of species’ than an economics or biology textbook with a further century’s developments and pedagogy.
So the greater part of our life now as researchers is be given over to generating more moments of true choice—choice to think, choice what to think about. It’s hard to imagine we will long have truck with professionally-checking-Fable outputs. But I expect the ability to think clearly, grandly, freely, will have at least a longer shelf life. And probably it’s also good for your soul.
In this spirit, I’ll be reading ‘the wealth of nations’ this week. I invite anyone interested to do likewise with a ‘classic’ of their choosing.
Inspired by conversation with @richard_ngo and @Ashe Vazquez Nuñez, to each of whom my thanks.