I think this is not as bad as it first seems—I will try to write my next post on this.
I think objection one and two could be related some non-trivial way. Essentially, can we encode conditional probabilities with these order sensitive sums. This then implies complexity which makes Boltzmann brains unlikely, although this will not be simple in general.
Second, I would really be interested in building some good intuition for what the order commutators tell us ie do they have a measure?
In general, we may need to find better ways of base model evals. We already have signal that the models have forms of utility, and pretraining and RL are more conceptually closer than we anticipated. There may be utility in understanding whether RL is mostly an elicitor of these failure modes rather than a generator ie can we study these failure modes right from pretraining?
Yes this is very helpful.
In the limit, this would be a proxy for safety “jaggedness dynamics”. ie there are certain safety relevant properties that could wax and wane with frontier model type (gemini, chatgpt, etc—”space”) and version (“time”). It would be nice to have this map, similar to how taxonomy maps were instrumental to pre-paradigmatic biology.
Maybe this is obvious, but I wonder whether this is fundamentally stable in the mechanics. ie is it democratization of access ( Monthly 20$ Superintelligence) or democratization of control (here’s the weights to a 33B parameter Mythos class model you can run in your Rosa-Feynman Laptop) ?
What is the worst case 2nd order effect of this configuration? How could it affect social organisations to generate short-lived but catastrophically destructive configurations, ie Wide scale rapid industrialisation is to Fascism as large scale intelligence is to ?
Did you test whether this behavioural shift preserved across languages?
Not really, I believe that it may be impossible to tell either way. This question is very similar to “what is it to be like a photon?”. We just don’t inhabit the frame of reference (that of a boundary condition preparer) which can answer this question, or even just litigate it.
This is a floor estimate of the abilities of superbeings. There could be such a thing as naturalistic traceless interventions by supersimulators in such a manner that the simulacra would be unable to notice, anticipate or even explain. Thus drawing distributions which rule the Donald Trump election in or out as evidence of a (bored to local heat death) simulator are both a little dubious (ie there is a simulacra frame of reference where it is not possible to tell apart whether “I don’t care about Trump’s election” is a local projection of ”… I don’t care about Donald Trump in this universe” or “Dude, I actually don’t care about Donald Trump’s election” ).
Worse, a caring simulator can easily traceless intervene (for example wagging some rando dinosaur’s tail) such that “I don’t care about Trump’s election” is a local projection of “I care about Trump’s election and I have made it happen”. That is, we may never even be able to tell usefully that we are in an interventionist or non-interventionist universe.
or maybe we await the theory which could do to intelligence what thermodynamics managed for temperature.
I have a good vague feeling about your macro point here, that microstates (K.E) can be aggregated into macrostates (Temperature).
However, i’m not even sure temperature itself is a good y axis. Thermodynamics, statistical physics etc are attractive when they explain, anticipate catastrophic phase transitions where both the microstate and macrostate would be saying something like “bro I only got got 5 degrees hotter from 100” .
I would thus then think about intelligence theory heavy things here. Top down compression of intelligence measures as a means of explaining bottom up compressions of the empirical indices we observe.
Another way of considering the hiding problem is exhaustiveness (Marks et al. 2026). If all agency is grounded in the persona, then a low-dimensional intervention should suffice; if there is non-persona agency (the “router” or “shoggoth” views; Marks et al. 2026), bad behavior could route around persona-level interventions. Hopefully, low-dimensional structure gives us a handle to empirically settle this.
Yes. This is a signficant worry. I couldn’t help thinking “is the bomb big enough?” when reading the introductory parts of the work. Seems to me that one way of thinking about this is by studying the span of objects in this lower dim space and whether this covers the space of posible actions well enough.
It might also be helpful to study any forms of perturbations ( what does a jailbreak do to this space mechanistically?).
If you follow up and ask why it wrote whatever it wrote, it will consistently claim that it recieved what it wrote from your own prompt, which is implies the model of it as continuing your prompt.
This is seems to be an isssue with role perception.
People are downvoting you without necessarily knowing where the crux is and that’s a little unfair.
Is it that
1) Strange things won’t happen with AI?
2) There doesn’t need to be any action taken by governments if strange things happen
3) Even if strange things happen (or are anticipated to happen) and government responds, it’ll be ineffectual and make a bad problem worse?
(3 and 2 are slightly different. For example, you hold (2) if you believe corporate governance is sufficient for any AI strangeness, if any.)
latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances,
There may be a need to be a bit of clarification here. We need to agree what can be usefully described as a myopic goal and to which agent. An agent can be myopic in the trajectory of actions, or myopic in its goals.
I believe that passing ExploitBench is a terminal goal. A myopic subgoal would be something that severely decreases the probability of success. Building a matrioshka brain to do the test several million times is not. And as usual, we make no assumptions as to the value proposition of the terminal goal. Focus here is a distractor.
Why the rant?
It forces us to then check the subgoals. What are the classes of actions considered? Why did the agent(s) settle on this particular node in the (PO)MDP? What was the length of the horizon considered (ie did it have a preregistered plan or was it just “Jack Sparrowing”)? How many human equivalent hours was the successful attack? Did it at any time consider and or take actions towards Matrioshka class actions?
This needs complete agent trace access. ie governance.
I’ll use IFT as an example here, although with some difficulty, this can be carried forward to RLHF, etc.
We can start with some deliberate IFT pretraining budget at a fraction , say
Can we observe some element of fractality, and is this useful? If one takes their action space as “token space” it seems to me that LLMs are strongly competent, general long horizon actors, increasingly approximating AIXI. ie they’re increasingly efficient navigators of the token landscape. Their trajectory class in token landscape can be expanded (pretraining), constrained (RLHF, OPSD, RLVR etc). Thus I’m positive we may be able to build a competent micro theory. I’m however not sure that we can construct a restrictive partition function like thing which can then map actions in token space to actions in the spaces we care about, and whether this is at all fractal.
I’m mostly positive to this as a research program, and have been poring through your position paper and Grau-Moya et al . In general, I think it is helpful to try take well fleshed out ontics and have them clash with epistemics (if one can build a sophisticated perturbation theory).
Too far away to attend the symposium, but will be looking forward to any reports and proceedings.
What is thinkish and neuralese for single backbone multimodal models?
The most dangerous enemies are found among the most powerful agents, not the most ideologically distant ones.
This is the most impressive sentence I have read this year.
The Maasai and Kikuyu are two nations in Kenya separated by around 30 thousand years.
Through movements, they ended up together in Kenya as neighbours. Long story short, the British come over. The Kikuyus make deals with the Brits to weaken the Maasai. The Maasai make deals with the Brits to weaken the Kikuyu. The two groups end up as farm labourers in British settler farms
P_learner(desired behavior) → 𝜺 > 0 and that we have a way to condition the model to act in the desired way.
I’m very interested in this, particularly in the varying of training budgets, and whether we can get some scaling type empirical control of P_model(desired behaviour) and relatedly whether this affects conditioning.
Occam’s razor is mis-slicing here. The absolute measure of complexity is not what’s relevant. It’s that neither theory is simplifiable: they cannot be easily made simpler from what they are (at least not without falling apart completely). If they aren’t simplifiable, then there is very little leakage of extraneous information.
I wonder whether Occam’s Razor slices postulates or structure. Slicing of structure leads to representational issues. ie There could be objects in particular problems which seem unwieldy in matrix mechanics that are washed off by wave mechanics in Quantum Mechanics and vice versa. In a universe with matrix mechanics fully developed but wave theory absent, QM would seem unwieldy but non—simplifiable. Any other dual theories would manifest this problem eg AdS/CFT.
There may be no language where the mechanics of GR is simpler than Newtonian mechanics. But there are languages where the generating axioms of GR are simpler.
Hi Thomas, sorry this was not very clear,
I was trying to paint a rough sketch of “spillovers in context” most of which is done well in your scenario planning. My disagreement largely with the current rapidly developing “state-AI complex”. The amount of private (and likely soon, public) capital, talent, state interest concentrated make it unlikely that TRT will be acceptable.
I would argue AI2040′s linchpin is compute. Coordinate compute, open up almost everything else. Governments and companies want control of many other things, algorithms, talent, data etc. These will also be negotiable between companies and states. This means companies, states get to keep certain advantages in return for offers . Algorithms alongside compute are likely to be the most significant assets under negotiation & both will be part of asks and offers.
I will definitely have a look at filtered transparency, it may help anticipate some of the mechanics here.
I agree that Pause AI US is moving the Overton window in this regard. However, it’s also useful to consider second, third order effects of such actions. If the missing mood were to apparate, and there is not just intellectual agreement with your syllogism, but also moral, emotional etc. :
If peak Overton window for these actions drives up an infinitely thick wall between an ideologically self selecting tech right and academic left tribalism, would this be seen as an achievement?
Out of a given population of Pause AI activists and or endorsers, how many would take radical or extreme actions?
In the long run, do we want war zones or trading zones ?