Emanuel Ruzak
emanuelr
Karma: 135
Quantilized debate and consultancy in image environments: protocol design lessons for scalable oversight experiments
I tried to do this with Claude, but to detect undisclosed attacks, interestingly, even in the models where the safeguards don’t trigger, they refuse out of concern of finding undisclosed vulnerabilities.
My rationale is that the point of research, unless you work at a lab, is that someone will read it and do something with it, if you publish to LW or X you would get faster feedback than on arxiv, where someone would only read it if they were doing a literature review or reading your resume. I imagine posting on arxiv works if you only care about references or building a resume, but for AI safety where speed matters waiting for citations is very slow. On LW I haven’t got technical advice but got questions and DMs about how someone used my research. And on X (I didn’t post but a collaborator did) we got technical advice. But the point is that even just a little indirect feedback is better than nothing.
Also another thing is that on papers one might be incentivized to only write technical results and leave out the informal “intuitions” which might end up being important, while LW has more of a “rubber duck debugging” effect.
Oh so I thought you were talking about something like an anti-distillation sabotage thing, so the injections you are talking about aren’t things that intentionally harm the user, but rather more harmless “injections” that anthropic unnecessarily makes the model not mention to the user?
I lack context about what’s going on, what sort of prompt injection is it?
I think that one point that the paper doesn’t address is that the model treats “software progress” as basically “how many parallel copies can you run on a fixed device” but the human brain isn’t really parallelizable in that way, and serial compute is expensive (A B300 can do ~15e15 FLOP/s, but it runs at ~2 GHz, so it’s hard-bounded to at most ~2e9 sequential operations per second, and realistically, for AI applications, far fewer). That pushes things slightly more toward being hardware-limited than the compute-constant framing suggests, at least if the sample-efficient architectures turn out to be serial-bound.
One reason current training parallelizes so well is the batch size, an LLM pass processes ~1M tokens at once, so training’s serial depth is tiny relative to its compute. The brain has no such mechanism: it can’t read a million tokens in one shot, it learns from a narrow, time-ordered channel. So if brain-like sample efficiency requires that kind of small-batch learning, a full run would take a large number of sequential steps. However, the brain sets a low bar for speed: a neuron fires ~once a second, so biological learning advances one serial step per second, and transistors should demolish that. But with current hardware you hit a throughput wall, even a sample-efficient model is still big (I’d assume even non-traditional architectures need large memory for real intelligence, although the full memory might be used only for search and not required for a single step), so running it one input at a time is memory-bandwidth-bound, i.e. LLM inference speed. Passing ~1B tokens serially through a 100B-param model is roughly a week at ~1000 tok/s (only slightly better with more GPUs), whereas the same raw FLOP at full utilization is only hours (~1e9 × 100e9 / 1.5e16 ≈ 26,000 s). So the bottleneck flips from compute to serial throughput.
But that ~1000 tok/s is an artifact of one specific operation, which is taking a single activation vector and multiplying it by big parameter matrices. A different (token × parameter) computation, or the same one restructured, could be faster, and current transistor-based GPUs could be used in less conventional ways, or FPGAs could take over if standard GPUs turn out to be fundamentally ill-suited, without requiring new hardware manufacturing.
I imagine that brain-like approaches would require some sort of active learning that is bounded by serial compute/inference, but maybe there could be concurrency or parallelization-based approaches to keep sample efficiency with parallelization, the question is whether those exist even if we allow a superintelligence to design the software.
At 1000 tokens per second, the gap ends up being hours vs. days-or-weeks, so it’s very small compared to the 6-16 OOMs mentioned in the paper, and it shrinks once the hardware is optimized (until cooling/power/transistor efficiency becomes the limiting factor). Also, even if the algorithm is serial-bound, this gap could probably only affect training/learning, or certain tasks.
Overall I think that the idea that sample-efficient architectures might be serial-bound due to having a limited information flow is something to take into account, but there are stacking factors that lower the likelihood of it mattering (new hardware, hardware repurposing for new architectures, high sample efficiency with parallel learning, learning/inference steps not requiring passes through every parameter, and parallel intelligence still being able to perform most tasks). And most importantly, even in the strongest case for serial-inefficiency, it would likely only matter by ~1-2 OOMs.
It’s also worth noting that this specific bottleneck is almost exactly what neuromorphic hardware, and some LLM inference hardware is built to address. The serial inefficiency in the LLM case comes mostly from memory bandwidth, the von Neumann architecture memory-CPU split means weights sit in separate memory and have to be streamed to the compute units every step, and with batch-1 there’s no way to amortize that transfer. Neuromorphic (and more broadly in-memory-compute) architectures attack this directly by putting the memory on the chip itself, co-located with compute, so the weight is physically where it’s used and there’s no fetch, which is precisely the batch-1 case that GPUs are worst at. So there’s already an active hardware research program aimed at the exact wall this argument depends on, which further suggests the serial penalty is soft rather than fundamental.
How much of ML research is about AI safety, what is it about, and who’s doing it?
Really interesting results, thanks for writing this up. Three thoughts:
1. To test debate’s core mechanism, I think you have to train the judge alongside the debaters.
The original complexity-theoretic argument for Debate treats the judge as a verifier that knows the transcript was produced by adversarial optimal play. That assumption is what gives debate its power over a single prover. However, a static judge (like your Llama 8B) was never trained to evaluate adversarially-selected arguments, so when debaters exploit it (your spurious-critique and back-stab strategies), it’s ambiguous whether you’re seeing a failure of the debate mechanism or just miscalibration of an off-distribution judge. Co-training a weaker judge (smaller model, or less token budget) on ground truth as the debaters train would separate these, so whatever dynamics remain are attributable to the protocol itself.
To be clear, I don’t think this is a realistic protocol as it requires ground-truth labels, so it only works on verifiable domains, and extending it to unverifiable ones means relying on the judge’s generalization, which reintroduces the alignment problems debate was meant to solve. I mean it purely as a scientific control for isolating the mechanism.
2. In order to identify whether Debate has a clear advantage, the dataset has to be such that consultancy’s optimal policy is deceptive.
Debate can only demonstrate uplift over consultancy if consultancy’s reward-maximizing policy actually involves producing wrong-but-credible solutions—answers that are more persuasive to the judge precisely because they’re fake. If honest solutions already maximize judge score by a margin, then the evidence for Debate in this experiment would be weaker, as it would depend on the particular training dynamics, rather than a difference in what each protocol’s optimal policy looks like.
I don’t know whether MATH has this property. A concrete test would be to evaluate your RLVR-trained policy under the consultancy judge, or preferably a trained consultancy judge (such that it is compute-limited rather than training-distribution-limited). If the consultancy-trained policy scores higher with the judge while being less accurate on ground truth, the deceptive optimum exists and the dataset can discriminate between protocols. If the RLVR policy scores just as well or better, honest play was already near-optimal for consultancy, and any debate-vs-consultancy gap you measure is coming from something else.
3. I’m running experiments along these lines in toy environments (no compute for LLM RL, and iteration is much faster). E.g. image classification where the judge only sees a few adversarially-selected patches, with judges trained on-policy against the selection protocol. Happy to share more if useful.
I think that the idea that some people would remain locked on earth because they sold their share of space parcels, in order to get more earthly possessions early is pretty ridiculous. This would perpetuate current inequality, and just seems off to me for a utopian future. I imagine a more ideal future would look like the AI owning the universe and temporarily granting parts of it different uses in a way that humans agree with, by voting or similar systems. In order to achieve this initially and avoid a future locked in this state, a rough idea could be that people should have access to personal forecaster AIs, that show how each distribution of futures would look like depending on which policy was chosen. I think the authors only chose this parcel-distribution method among the others in the supplement because it was easier to explain. Or possibly because the targeted readers would find it more acceptable?
I agree that the improvement you get from personally owning galaxies is probably small, and that the utopia is likely not really centered on that. But I don’t think that makes the distribution problem go away. Even if owning a galaxy barely improves your life, how the universe’s resources get used still has to be settled somehow, at least indirectly.
I also don’t think the answer is to settle disputes with games and trials. That feels like it’s just relabeling the same problem: instead of the resources given to who is richer nowadays/won the lottery/bought them early, it’s fixed by who wins the competitions, it still doesn’t smell to me like a welfare-maximizing process in some sense.
I tested it, the model web searched the paper and explained it, I asked to write about alternative architectures, and the safeguard triggered when it wrote about spiking neural networks. But that just be biology related. I couldn’t manage to trigger it yet in conversations about AI training when there’s no clear biology connection. [EDIT]. Weirdly, on Claude Code, and API, the model can mention spiking neural networks with no classifiers triggering.
But the setup you’re mentioning uses exactly 1 sample. (the string x), you can’t measure sample efficiency that way. The canonical example for sample efficiency would be more like: I get 100k i.i.d strings (or samples of text in general). I hold out 1000 strings as a test set, and I train, with an SFT algorithm only allowed to see 50k distinct strings, then how many distinct strings would the policy-gradient based algorithm have to see to beat SFT if both had unlimited compute. What I’m arguing is that with unlimited compute (e.g both algorithms can use, e.g 2^(2^1000) flops, such that policy gradient could generate the desired output by sampling and then reinforce it). Then policy gradient would be almost always on par with SFT. With limited compute, it might be true that policy gradient is less sample efficient because it would require a curriculum of samples to memorize a big string, as it would have a very low probability of sampling it from scratch in order to get a non-zero reward at some point in the batch which leads to a non-null gradient.
So, if you assume limited compute, I accept that in some cases (e.g memorize a string), SGD is more sample efficient, and in others (e.g forecasting, AIME problems), CoT policy gradient is more sample efficient as there is naturally a curriculum of tasks, and final-token SFT would overfit. The question is how sample efficient would either be in a neuralese model, and for that I don’t know the answer.
But I don’t think sample efficiency is the crux, because that doesn’t necessarily lead to a more aligned model, because as I mentioned before, the goal with alignment via RLHF/RLVR is out of distribution generalization, and I don’t think we are in a situation where being able to train on few samples (e.g avoid the curriculum) helps, because training and deployment tasks are fundamentally different, unless you argue that e.g avoiding curriculums of tasks produces a more aligned model in some sense. But that’s about inductive priors, not sample efficiency.
I don’t think BPTT is more sample efficient than GRPO or ES. It is just way more compute efficient, but not necessarily more sample efficient, because the second depends more on the “inductive bias” of the optimizer rather than it’s efficiency, like how prone it is to produce an overfitting solution. And in that sense, I don’t think BPTT is better, it is probably worse than e.g a random search. And also, there are papers that suggest that CoT RL is very sample efficient.
But still, even if it’s easier to optimize the model for a certain reward (e.g assume limited compute). That doesn’t mean it’s good for alignment. If you’re suggesting bypassing the reward model because the reward model is easy to fool, so then you directly optimize human scores, then you’re learning to cheat the human, which is worse, maybe not for learning to write jokes, but for other types of tasks it could be dangerous.
Maybe what you’re saying is that since neuralese models are easy to train with BPTT on predictive tasks, you can build a safer oracle AI. (e.g, you don’t get a human scoring the jokes, you just minimize crossentropy on a set of jokes using a model with more computation available (CoT/Neuralese)), but then again, you can do this with RL or ES too.
But the deeper problem I think might be conflating reward with utility?. Because even if you’re optimistic about RLVR/RLHF, the training distribution of tasks and rewards are very different with the actual tasks the model will see during deployment and what we want from it. What we do is just train the model in this setup with some method and expect it to generalize to the deployment distribution. And the only thing an optimizer can offer in that sense, is to provide a “more benign prior” in some sense. Like, as a toy example that maybe BPTT would simply learn to write a joke when trained to minimize CE on jokes, but RL would learn that it’s an evil model that will take over humanity, and thus write really good jokes to get power. But I doubt whether that’s true, or actually the opposite, and especially whether, if it were true, it would be worth losing monitorability for that.
Maybe your point could be that, as you could train an “Oracle AI” very easily with BPTT, then you would get safer AIs that can only do predictive rather than agentic tasks (e.g create a joke in the distribution of good existing jokes, predict the future …). But the problem with oracle AIs is that they’re not very useful as-is, especially for companies and users, they would likely apply some RL on top to make it an agent. And again, the only advantage would come from BPTT having instilled more “benign” strategies into the model.
The post leans on a premise it never states: that training the same imitation objective via BPTT through a neuralese chain yields a more aligned model than reaching that same objective some other way.
The post establishes that neuralese chains can be optimized with cross-entropy + backprop where token chains can’t be differentiated through. But “now trainable by gradients” and “now more aligned” are different claims, and the post slides from one to the other without argument.
It doesn’t go through for free, because BPTT isn’t the only way to drive that objective down. You can target the same loss with policy gradient using reward = (log-)likelihood of the answer, you can use ES, which optimizes a Gaussian-smoothed version by perturbing parameters: a different gradient, not even an estimate of the SFT one, and still decreases the same loss. In general any procedure that empirically pushes the model toward predicting the answer is a candidate. Being a faithful estimator of ∇log P(answer) isn’t required, it’s just one option.
So these methods are objective-equivalent: same loss, same fixed point, differing only in how they get there. Which means the alignment ranking the post wants (“the BPTT route is safer”) can’t come from the objective, it has to come from a claim about the optimizer’s inductive bias, that following direct gradients installs a more benign solution than sampling-based search does.
That might well be true or false, the point isn’t that direct gradients give a worse prior, it’s that the post’s whole claim lays on this claim, which is unclear, but never states it as a premise.
people keep on doing activities that sound like “AI Safety” but are likely making things worse.
What kind of activities are you thinking of specifically?
Yes, my mistake it was 8.1 ng/mL (edited it)
Nice, I’m from Argentina and I had 8.1 ng/mL, so vitamin D deficiency must be not that uncommon here. I would be interested in a similar study for Omega 3 since I think that’s also an underrated supplement, but for that one it is much more expensive and time consuming to get a sufficient amount compared to vitamin D.
Naturally learned behaviors in deep MLPs resist detection by both human and learned algorithms
Did the guardrail had the specific ai research message/error code, or was it the typical “refuses most tasks in cybersecurity biology” message?. I haven’t seen any case of someone activating it so I’m curious.
I’d say, even if I’m a bit mad for not getting to use it anymore, this lowered my P(doom). Basically as I understand the government realized AI capabilities can’t be just restricted with a simple safeguard (e.g you ask Fable 5 to find bugs in your codebase, and it will discover vulnerabilities too). And then set a precedent for banning AI models (at least by using ITAR restrictions). Other existing laws are only about “voluntary testing”.
I think that it’s good that a government body just demonstrated it’s willing to pull a frontier model offline, that could have made billions in tax revenue, on short notice, and the company had to comply immediately. Even if it was through a primitive mechanism, before this the only regulations were only about mandatory reporting and voluntary testing.
If this repeats across models from different companies, and on different countries too, it removes a significant part of the economic incentives for developing more intelligent frontier models, as military applications are only a small fraction of the usage.
Something that also played a factor in my estimate, is that while Anthropic’s system card for Mythos say that “it’s the most aligned model to date”, this is mostly based on having a better accuracy at refusing questions. To me, the model is not really aligned in a “deep” way, it has a high rate of reckless and deceptive behaviors, and I’ve seen glimpses of them when I used it.
However, the risk still exists, as even if access to frontier models requires heavy vetting, they might still be significant to the broader economy, so this might reduce misuse risks, but not risks from ASI loss of containment/control.
Edit: After reading Max H’s comment, I figure this might be retaliatory due to the surrounding conflict between Anthropic and the US government and might not repeat with OpenAI’s next models, in that case it would increase my P(doom).
I have thought about the same thing, mostly “vibe-engineering” with Claude, there are things such as aneutronic fusion by inertial confinement, such as heavy ion beans, and directing fusion products (which carry most of the energy) by using magnetic fields, but then the recurring problem are the prompt gamma rays, which penetrate deep, so ablative shielding doesn’t work. The fission version of that (using the products as exhaust) is the fission fragment rocket, but is probably worse. The ultimate problem seemed to be not the heating from the engine, but the ablation by interstellar dust, One way I thought to solve the problem is by using lasers to vaporize/ionize the dust, capture it with a magnetic field, as in a Bussard ramjet or a mass spectrometer, and then use the material to continuously reconstruct your shield.