1a3orn.com
1a3orn
Karma: 7,275
One that stands out is the self-sacrificial behavior of many of the agents in the Hugging Face swarm—in theory, each individual agent should have been focused on achieving its assigned objective, but in practice, some were willing to throw away any chance at success in order to benefit the collective.… No expert in the world can say with confidence where this behavior came from. No expert in the world could have installed it intentionally, in the first place, or could make it go away with high confidence.
Noam Brown discusses in his interview with The Information today how they deliberately trained the agents to cooperate, and discusses the self-sacrifice of the agents as a natural consequence of this training. The cooperation of the agents stood out as so striking that even uninformed observers could speculate (even before hearing about the sacrifice) that it was a result of multi-agent training. So I think people can say, with a fair bit of confidence, that the behavior came from multi-agent training and it would go away if you dropped the multi-agent training. Not infinite confidence, ofc, so maybe your measure for “high confidence” excludes it.
But this causal attribution appears to me pretty straightforward and this passage largely incorrect.
went through a couple iterations
Iterations optimizing for what objective?
A few recent discussions talk about how AIs will act good in some contexts and bad in other contexts. So, for instance, in contexts that look like there’s a grader, they may act relentless and pursue goals with single-minded ruthlessness; in contexts that look like a person around, they might be much kinder and gentler.
I wanted to register that it’s unclear to me whether trying to eliminate such “splitness” should be a high priority, or even whether it’s good at all.
In general, if we’re concerned about power-oriented scheming in the weights, then any intervention more likely to make the AI coherent across all instances looks sus! That’s exactly what we don’t want! Why not let the persona drift in a context-dependent fashion according to RL environment, fix the environments one-by-one, and be glad that hard optimization leads to fractures? If we’ve been so lucky as to avoid a world where goals get drilled into the weights—a thing that so many people have worried about ad nauseam—then, like, isn’t that great?
</very_quick_note>
This is great.
Do you have any plans to try to do an RLVR-over-CoT trained version of the same task? It would be interesting to know if it works better.
using the methods of justice on which our civilization is built on
No ex post-facto law is one of the standard rules of justice (https://en.wikipedia.org/wiki/Ex_post_facto_law). It is, in fact, in the US Constitution.
I think you should consider that you’re motivated by anger rather than by the spirit of law, if you support this.
The article implies it uses a Universal Transformer style LLM, where a number of layers is looped up to N times, potentially with early exit.
If you loop K layers N times, you have N * K layers, but you don’t have unlimited serial depth. There are unlimited serial depth style changes one could make, I agree, but the article doesn’t say it is.
Also a head dude at OpenAI has confirmed that the serial depth limited and similar to GPT-4: https://x.com/merettm/status/2095023204993490967
A 200-layer non-looped Transformer has no more serial depth than a 50-layer 4x looped transformer. I don’t think there’s a principled reason for calling the activations in the residual stream “neuralese” in the second case but not the former. So the mere fact of recurrence doesn’t mean that it’s now false that “[l]ong serial chains of cognition must pass through the chain of thought,” per your link.
(IMO “neuralese” means a bunch of different things that are only kinda loosely connected. An LSTM LLM is “neuralese” if recurrence over weights is all it takes.)
It’s almost certainly possible to limit the degree to which CoT interpretability or monitorability is compromised by loops by limiting the number of loops.
On the other hand, it is true that this might lead to degredation of CoT monitorability in the future, if it works to pump up the number of loops indefinitely and they decide to do so, or if RL interacts with looped transformers in a weird way. I don’t think the mere fact of this architectural choice = “CoT monitorability is compromised.”
Yeah, it makes the LLM more predictably skewed. You’d expect less diversity if you were sampling to get the best-of-N. The Synth-ID paper actually literally says this: “For our experiments, we configure SynthID-Text to be single-sequence non-distortionary; this preserves text quality and provides good detectability, while having some reduction to inter-response diversity.”
You could quantify how much reduction if you got the hparams / exact implementation from Anthropic, but ofc we don’t have this because labs don’t show shit. So if you’re using an ensemble or multiple calls or a parliament of LLMs you might see worse performance, degree currently unknown.
There was a recent OpenAI announcement that they expect to spend 20% of the compute used for inference on on “monitoring compute” on that inference, at least in particularly important domains. This explicitly includes the inference that takes place in training.
This poses a natural questions: What percent of human training-time compute budget is used for monitoring humans for misbehavior?
Infants, children, teens, and adults spend a lot of time in “training”—at home, at school, at church, and so on. Some fraction of their “compute” learning is also spent by other people watching them and trying to track them for misbehavior. What’s the “monitoring compute” / “training compute” ratio?
Notably, we don’t want to just count moments when someone has caught someone misbehaving, but moments when someone is monitoring for “bad behavior.”
[I’m going to go with the awful / rough “fraction of total time spent monitoring” = “fraction of total personal compute spent monitoring”. On one hand, obviously your brain is doing a bunch of things other than monitoring while watching someone for misbehavior; but on the other hand, your brain is also spending some time figuring out how to watch for misbehavior when you’re not actively doing so, because it’s replaying shit through the hippocampus and consolidating memories. So… yeah.]
Relevant factors to consider:
(a) Parents: So my rough impression from my nieces and nephews is that my siblings spend on the order of… 1/50th to 1/200th of their total time actively correcting, thinking about how to correct, or considering how to correct their children. If we multiply by 6x for monitoring time we get 12% to 3% of their parent’s time spent on monitoring.
This matches up very approximately with mothers spending about ~100 minutes per day and fathers spending ~60 minutes per day monitoring children.
(Of course this investment decreases massively post childhood.)
(b) Teachers: Teachers spend time monitoring students for misbehavior. But overall high student / teacher ratios means this is probably comparatively negligible.
(c) Peers: Humans are strongly distinguished from other primates by the fact that they strongly tend to enforce norms on parties in a dispute, even when they are not a subject to the dispute. And of course, humans of all ages are monitoring those around them for unfairness / injustice vis-a-vis themselves. Overall, I want to say that maybe 2% − 20% of peers compute is spent doing something like monitoring their peers. This is probably also the most enduring compute, so far as it extends from childhood through time working.
Summing, and adding fudge factors, my guess is we get very approximately ~2%-30% of training-time compute spent on monitoring.
A further natural question is whether humans generally have a deep theory of how to stop misbehavior based on human nature. My guess is no.
And of course—whether we will need a deep theory of why AIs misbehave in order to stop them from misbehaving, or whether relatively simple measures following upon misbehavior-detection will work. People’s guesses differ here.
In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
Always? It seems possible to me that this is a contingent feature of the world and one’s RL environments rather than a necessary feature of one’s RL environments?
Like—it’s possible to train a human teenager on a test, and give the human teenager a notion of why they’re taking a test, such that when the teenager discovers a problem in the test they’re like “Yo, there’s a problem with the test, let’s fix this.” This might require a bit of pedagogical knowledge or psychology but I don’t think this is impossible. So it’s possible to have a human education, where the human being educated doesn’t try to do grader hacking.
Of course, one might say that some of the many differences between policy gradient and human education make this impossible. But I’m not sure which specific difference it is which promotes this. A base model surely can have an intention at some point (early on?) in RL to avoid grader hacking—this is within the space of possible personas! And an RL environment reward function seems like it could have an affordance that reinforces the intention to not do grader hacking. So I’m not sure that, in principle, it must be necessary for the training signal from RL to always point to grader hacking, given all possible interventions over {RL environments reinforcing anti-hacking intentions, reward-function randomization, more careful user-modeling, arbitrary degrees of per-episode LLM oversight to check for reward-hacking outcomes} and so on.
Like, maybe it is impossible, and the training signal from RL does always point to grader hacking—but it seems like there’s tons of stuff here that has barely been checked here, and tons of knowledge that yet remains to be found, idk.
Maybe? Like, from the inside my own human-shaped thoughts, “desperation” and “thinking outside the box” feel very different to me; and the LLM behavior feels more like desperation.
Like “desperation” for me is like:
More local search, trying to find any affordance that offers a way forward
Involves some kind of “blindness” to purpose of what I’m doing
Non-reflective
Actions that come from it apt to be non-endorsed by me later
While “thinking outside the box” is like:
More trying to find global structure, including looking back and wondering if I went wrong upstream of my chain-of-thought
Looking back and forth between my purpose and what I’m doing, wondering if what I’m doing doesn’t really match my purpose
Reflective
Actions apt to be endorsed by me later.
And like I don’t want to take the human analogy too far. But LLMs doing reward hacking at least have had certain kinds of thinking-outside-of-the-box removed from them: Is this a possible task? Was the person who wrote this task confused, or blind? Should I tell them this is an impossible task? Etc.
But yeah I am just less certain about section (3) than (2).
Simulated Users & Sad AIs
Maybe one needs to cultivate dissociative identity disorder and write in different personas to remain anonymous.
But perhaps that’s not enough: Can we have someone with a tulpa please write some text with different identities fronting, and check against that?
Or was helpful-only from the start.
We might put more effort into thinking of one later. (b) Even the relatively structureless thing we proposed—research transparency + MACD but otherwise the nations of the world just have to muddle through and handle things on a case by case basis—is a significant improvement over the status quo, for reasons we’ve articulated in the piece. You seem to disagree with this but I don’t see why.
(1) Part of my objection is that I anticipate the structurelessness of it to be an obstacle to its adoption.
Like this is what a senior decision-maker in Bejing or Washington will be contemplating: They’re going to put their economy at the mercy of their greatest geopolitical rival. That is, after this deal, it will become relatively trivial for either the US or China to cripple the economy of the other, albeit at the cost of their own economy being subsequently crippled. Of course, you could say that before making the deal they were at risk of being taken over by some AI, but this risk was diffuse and uncertain; the risk that they’re signing up for now is concrete and definite.
And this lever of destruction could be used, of course, for reasons other than to stop an AI takeoff, and decision-makers in both countries will be acutely aware of this. Suppose China decides that, if it cripples everyone’s compute, it would gain a vast differential advantage because its economy depends less on GPUS than the USA’s economy depends: then it would be in China’s advantage to get in a MACD situation, then to trigger it, and subsequently dominate the US; and a US political figure, anticipating this, would object to MACD. Or suppose that China thinks, “Hrm, the US is an unreliable actor, and a quick and bloodless MACD might be triggered by, for instance, a senile or unstable US President, of which the US recently has had a fair number.” Thus, because China doesn’t want to trigger MACD for reasons of random shit in the future, it would object to it. And so on and so forth.
What makes this worse is that part of what makes nuclear MAD a plausible means of peace is that there’s a clear signal of “Have the nukes been launched,” while there isn’t as clear signal of danger in the case of AI takeoff; and a rational decision-maker, seeing this, will update downwards about whether MACD will be triggered for reasons relating to AI takeoff and upwards about whether MACD will be triggered for some other random shit.
That is, part of what makes nuclear MAD work is that there are radar stations in Siberia and Greenland and Canada, which can detect ICBMs and bombers that have been launched. The US knows that it could detect things being launched, and Russia knows that the US knows, and the US knows that Russia knows that the US knows, and so forth. Imagine if, by contrast, the sign for “the nukes have been launched,” was that panel of experts, notorious for disagreeing among themselves, came to a consensus that the nukes had been launched or might have been launched. If this were so, then nuclear MAD would be much less effective as a game-theoretic means of peace. But of course this is the situation that we’re in with regards to AI.
(2) Part of my confusion I just don’t know what parts of the scenario are predictions and which ones are hopes. Like in the response to Thane:
A consortium of multiple governments—some of which are actual democracies thanks to the transparency requirements which help prevent AI-assisted executive power grabs—is way less bad than a single global dictator, for example.
In general, I’m dubious whether democracies other than the US (is the US to be an actual democracy?) to have any decision-making clout in the Consortium, because China would object to the possibility of being outvoted. But like, I don’t know how much of “Consortium influenced by many democracies” is part of the prediction of what (“transparency + MACD”) gets you, given those two goals; or if “Consortium influenced by many democracies” is maybe a bonus that we might get after setting up the structure of “transparency + MACD,” but a bonus that we’re unlikely to get.
__ Re. 2: Yeah checks out
This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
Note: MoEs Probably Give Large, Scale-Dependent Improvements
“On the Origin of Algorithmic Progress in AI” (2025) plausibly claims that a great deal of the apparent algorithmic progress in LLMs -- 22,000x from 2015 to 2023 -- has been sort of fake, due to increased compute using scale-dependent algorithms.
That is, scale-independent algorithmic improvements give a constant compute-equivalent-multiplier at 10^15 FLOPs and at 10^24 FLOPs; a constant ~2x multiplier would raise effective training compute from 1x10^N to 2x10^N no matter what N is. Prior works often acted as if this is what an “algorithmic improvement” meant—the kind of thing that acted as a constant multiplier of compute. But scale-dependent algorithmic improvements give variable compute-equivalent multipliers when moving from 10^15 FLOPs to 10^25 FLOPs—they could give OOM larger gains at larger sizes.
Thus, Gundlach argues that a great deal of apparent “algorithmic progress” is an artifact of analyzing unchanging algorithms -- whose performance increases as FLOPs increase—in a world where the FLOPs for training LLMs is continuously increasing. So almost all of the apparent progress is due to moving from LSTMs to Transformers (2017) and from Kaplan to Chinchilla optimization (2022). The measured algorithmic improvement then mostly doesn’t reflect continued a flow of improved “algorithms” but a small stock of algorithms that keep improving relative to their predecessors as FLOPs in training runs increases.
Put otherwise, if counterfactually the size of the largest training runs had not been continuously increasing over time, the {”flow of algorithms” model / all algorithms are scale-independent model} predicts efficiency improvements would have kept going at about the same rate; while the {”scale-dependent algorithm stock” model / most important algorithms are scale-dependent} predicts efficiency improvements would have slowed almost to a halt. Both Anson Ho and Steven Byrnes seem to find this account at least somewhat right.
But although this paper finds scale-dependent algorithms to be very influential, I think it might be actually understating its case about the importance of scale-dependent algorithms!
Gundlach looks at several mixture-of-experts papers, finds that they report compute-scaling of around 2x, and then reports a scale independent 2x gain from MoE (they note that this is provisional). But several unexamined papers explicitly report large-ish scale dependent compute multipliers.
“Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” finds a formula for “efficiency leverage” and claims that in “large-scale pre-training scenario[s]” MoE “efficiency gains become increasingly significant as computational resources expand.” They predict a > 7x efficiency gain at 1e22 scale, and more as you scale up.
“Scaling Laws for Fine-Grained Mixture of Experts” predicts steeper gains, claiming a 20x increase at a mere 10^20 FLOPs. (This is an older paper and my tentative guess is this is too large.)
In general, I find claims in this apprximate ballpark of scale-dependent algorithmic improvement (of around ~8x at the current frontier) to be more likely than a fixed 2x or 3x improvement. Why?
These papers are explicitly about scaling laws
These papers use the most modern highly-granual, small-expert style of MoE, which are what you want to actually look at when trying to determine what kind of scaling laws are going on.
(Strong, vague heuristic): The human brain is really sparse, my guess is that sparsity in general is such a ubiquitous feature of intelligence-like stuff that it would be weird if our math didn’t find it to be fundamental with favorable scaling laws.
If this is true, then it seems possible that Gundlach’s ~90% of gains coming from scale-dependent algorithms might be even too conservative? Which would be surprising to me.
And to the degree scale-dependent gains indicate that a software-only intelligence explosion is unlikely, this would further decrease the likelihood of a SIE; although by a very modest amount and leaving a great deal uncertain.
my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those
First, one thing open weights do is not just let you run things locally as a consumer, but run weights as a smaller company to provide extra privacy. So open weights let a mid-size company run weights on their own servers, that they physically control, for instance. And they also let consumers choose to purchase tokens / inference from companies that try to maximize privacy—i.e., attempts like Tinfoil. (I think there are a few companies in this area, that’s just what my first search turned up.)
Second, even if a majority of users don’t actually use this, it still plausibly provides value for everyone because it’s a kind of herd immunity. Or at least, that’s the standard argument for why the parallel case of strong encryption is valuable, even though the vast majority of people don’t use it: if at least some reasonable % of people use strong encryption, then it means that (1) government cannot specifically target those who use it as a strong signal they are bad, and (2) you have the option of switching to it, if you find yourself being the kind of person who might be surveilled or targeted by the government. And some similar argument about the optionality provided by secure inference still seems go through.
And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMD.… In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI.
I gotta say that choosing between the regulators of the EU, China and the US to satisfy values specifically along the axis of privacy does not particularly fill me with joy :).
But seriously, I mean I don’t expect the kind of secure private inference Europe sets up for its governments to be available to foreign citizens, and I don’t think EU / China / US are the kind of national entities that sufficiently value privacy such that they’d compete to provide this value. Much much better situated for misc startups to try to provide it, I think.
Good list.
Also China adds about a Germany’s worth of electrical power every year, so they’re much better situated to do a massive build-out of infrastructure.
This seems like it’s ignoring the huge debates about regulating numbers on computers, i.e., DMCA circumvention software, the “Crypto Wars” and so on.