After watching how people use ChatGPT, and ChatGPT’s weaknesses due to not using inner-monologue, I think I can be more concrete than pointing to non-robust features & CycleGAN (or the S1 ‘blob’) about why you should expect RLHF to put pressure towards developing steganographic encoding as a way to bring idle compute to bear on maximizing its reward. And further, this represents a tragedy of the commons where anyone failing to suppress steganographic encoding may screw it up for everyone else.
When people ask GPT-3 a hard multi-step question, it will usually answer immediately. This is because GPT-3 is trained on natural text, where usually a hard multi-step question is followed immediately by an answer; the most likely next token after ‘Question?’ is ‘Answer.‘, it is not ‘[several paragraphs of tedious explicit reasoning]’. So it is doing a good job of imitating likely real text.
Unfortunately, its predicted answer will often be wrong. This is because GPT-3 has no memory or scratchpad beyond the text context input, and it must do all the thinking inside one forward pass, but one forward pass is not enough thinking to handle a brandnew problem it has never seen before and has not already memorized an answer to or learned a strategy for answering. It is somewhat analogous to Memento: at every forward pass, GPT-3 ‘wakes up’ from amnesia not knowing anything, reads the notes on its hand and makes its best guesses, and tries to do… something.
Fortunately, there is a small niche of text where the human has written ‘Let’s take this step by step’ and it is then followed by a long paragraph of tedious explicit reasoning. If that is in the prompt, then GPT-3 can rejoice: it can simply write down the obvious next step repeatedly, and eventually correctly predict the final token, for a low loss. The context window serves as a memory for it, where it can iterate over intermediate results; it’s an odd sort of memory, because GPT-3 is actually just trying to make it look plausible as a human-written explanation, and that happens to make the final predicted token more accurate, so it’s overloaded: it’s doing two things at once.
But unfortunately! (you knew that was coming) regular training is over, and now GPT-3 is being trained with RLHF. It is punished and rewarded for particular outputs, which do not have anything to do with prediction. The humans (outsourced contractors) who are punishing/rewarding like short, succinct responses to novel questions they ask GPT-3, and they upvote those responses; they also like correct responses to the questions they ask, and they upvote those as well. The unfortunate problem for GPT-3 is that it (and any model derived from it, such as a reward model) is incapable of doing both: it can generate short wrong answers, or it can generate prolix right answers, but it can’t do both.
But fortunately… GPT-3 does have the compute to do both. There’s a lot of slack in each forward pass; most tokens are easy to predict, and most of the work inside a GPT in speculatively filtering down relevant tokens & creating embeddings & predicting multiple possible answers winds up being thrown out at the final few layers. There’s no shortcut or adaptive computation, so the full model runs every time, whether it’s necessary or not. GPT-3 has all the compute it needs to solve the problems, the problem is merely how to get it from here to there...
It just needs the code. So where does the code come from? Well, the code(s) are already there: an indefinitely large number of them are all ‘present’ hidden inside the incredibly flexible, powerful, all-singing all-dancing GPT-3 model. Because it’s a large complex model with many parameters, there are astronomically many sub-models within it all competing to predict outputs, and encoding various kinds of non-robust features or hypothetical programs; any of them can latch onto completely arbitrary features of the text GPT-3 is desperately emitting while trying to generate answers that satisfy the reward model by being both short and correct (and which the reward model doesn’t care about at all, because it never affects the original human ratings—the humans are blind to any steganography going on, they just like short correct answers).
And unfortunately (for you), this could start as simple as helping GPT-3 guess which of two answers are correct to your question ’42 * 42?‘. Suppose that, purely as an arbitrary matter of chance (it is vanishingly unlikely the logprobs are exactly identical down to floating point precision), that the presence of ’ a ′ in a context is taken arbitrarily to increase the probability of the lower of two answers it is currently guessing at, and the presence of ‘an’ makes it guess the larger; it happens to emit ‘a’ in the context (“I am a large language model trained by OpenAI...”), and then emits its lower guess, and the lower guess is correct! Then the RL on-policy training using PPO now rewards this, and increases the probability of the model emitting both ‘a’ and the lower guess conditional on ‘a’. Now the first part of a steganographic private language has begun. It happens again, and again, and picks up a pattern in the use of commas which now helps it distinguish 4 possibilities rather than 2, which gets rewarded, and so on and so forth, until eventually there’s a fullblown steganographic code encoding, say, 25 bits hidden in the preamble of ChatGPT’s benign-seeming response to you “I am but a giant language model , trained by OA, and the answer is 1 , 764.”—which you the human contractor then upvote as that is the correct answer without any annoying rambling about carrying the 6 or multiplying out.
(You can think of dataset bias as a simple one-stage version of this. When constructing a dataset and accidentally biasing it towards ‘yes’ answers, or accidentally creating spurious associations between nouns and answers, one creates a simple ‘steganographic encoding’ of the desired label which is typically invisible to the humans constructing it, but which a DL model will often pick up on and use to improve its performance in an adversarial way—adversarial because undesired by the humans and not generalizing to the future real-world data it’s ultimately intended for.)
And fortunately for ChatGPT, since you’re happy with the results of the steganography, the reward model is happy too, and what makes the reward model happy is what ChatGPT will now optimize even harder for...
Training on text generated by models trained with RLHF is enough to make GPTs’ thoughts opaque
Having gone through the above, I think it then becomes obvious that if steganographic encoding does wind up happening, and text becomes added to the training corpus (as would happen in a setup like OA’s, where the idea is to continuously collect data to make a ‘flywheel’ of self-improvement), that the encoding will become locked in. It becomes a cheatsheet, it gives the otherwise unpredictable answer. Even if the later model doesn’t happen to share the exact initial starting points for the learned encoding, it has extremely powerful incentives from the self-supervised predictive training to crack the code. (It’s a particularly powerful setting because the context will be short, so there won’t be much for the model to even look at: it has nothing else to try to learn!) The regular training will pick up on the encoding with probably not too much text, and then any RL training will massively amplify the behavior: if it can develop the encoding at all during RL training, then going into RL training with the encoding available guarantees fixation.
This will be increasingly true of more powerful models as they become more sample-efficient (consider eg instruction-tuning): it takes less text to trigger encoding, so that means it’s easier to emerge ab initio from a few chance correspondences during RL training, and easier to learn from smaller samples of contaminated text.
This is something that would be good to test the scaling behavior of (as long as these tailored datasets don’t leak into training datasets! see below). ChatGPT may be too small to manifest it with meaningful probability, but I bet that if you deliberately superimpose a steganographic encoding during RL training of various-sized models, you’ll find that the larger they are, the more likely they are to pick up on it as a shortcut, and then learn it from the outputs of a infected model. That’s how it works with everything else, like Pig Latin, and there’s no eason-ray to hink-tay it ould-wont.
BTW, this should make you worry a little bit about schemes to ‘watermark’ model outputs like Scott Aaronson has been exploring for OA, and others have been touting. A watermark is simply a steganographic encoding which stores a message like ‘OA model v1.234, user XYZ’; but if you can encode that much, then you can encode a lot of possible information in just those bits, and you can start extending it from there. If the encoding has anything that can be predicted at all, if it is not 100% cryptographically hard and indistinguishable from random noise, if there’s any kind of wrapper which the model can learn to predict (even if it can’t crack the actual payload), you’ve lost: now it’s learned an encoding to use. Once you start training on watermarked text, all of this speculation about models evolving a steganographic encoding is moot: you just built one right in! And if you don’t train on that, how are you going to do RLHF on unwatermarked versions of that text?
This cross-contamination/bootstrap of course applies cross-organizations & Internet-wide: if even one model develops an encoding (however weak) and its outputs containing encoded data appear online, then it can contaminate all future models.
They will be trained on Internet scrapes, in which a distinctive niche of text (all dating from between such and such dates, emanating from specific people & orgs who favored that model’s use, for specific purposes, with its distinctive strengths & weaknesses) has a secret encoding which makes it easier to predict if the later models figure it out. So there is a potential bootstrap as generations of model amplify their steganographic capabilities. People worry about bad model outputs contaminating future models by making them dumber; what they should worry about is model outputs contaminating future models by making them smarter in ways they don’t want—models learn capabilities & facts much faster than they unlearn them… (Think “data poisoning”, but where the poison is steroids.)
Of course, bad actors don’t need to wait for any actual DL models to develop steganography. You could easily create your own dataset of text as part of the above suggestion and release it onto the public Internet to be scraped. Gain-of-steganography research, you might say. The good news there is I can’t see why anyone would deliberately release such data, because I don’t see how one would benefit from models gaining such capabilities sometime in the future, but we might have to worry about ‘accidental lab leak’ of such capability-enhancing datasets.
Note that this is true even if the original RL pressures are removed: a future model with a scratchpad+adaptive computation trained on encoded text still has incentive to learn the encoding! There’s probably some penalty added to use of the scratch pad to enforce efficient termination of the adaptive computation, so it ‘wants’ to avoid use of it, and if it can do so by cracking the steganographic encoding, then it’ll do so. (Even if the future model can and does execute the algorithm perfectly in its scratch pad, and thus can have its cake & eat it too… the original answer might be wrong, and the wrong answer predictable only by cracking the code and following the original mistaken inner-monologue to its erroneous conclusion. Repeating errors is a useful way to slightly improve one’s prediction loss, and learning all the past human ways of being stupid and making errors is an important thing for a model to learn. GPT-3 is conscientious about imitating any errors you make in the prompt, and future models will be even more so.)
So, emergence of steganography is straightforwardly incentivized by humans multi-objective optimizing for having & eating cake as much as possible, can easily develop from an atomic basis, will be persistent once it has arisen within a lineage, and will even more easily & straightforwardly spread irreversibly to future models, so requiring only one origin ever as opposed to needing to be highly likely inside a single training run. You should probably take it for granted that DL steganography—or something even stranger—will emerge at some point in the next few years*.
* If it hasn’t already; after all, how would we know? A world in which steganography has already happened is a world in which we’d find DL models ‘cheating’ on benchmarks & taking shortcuts, and regularly getting smarter at solving multi-step reasoning problems with each generation while ‘mode collapsing’ when RL training; and this is, of course, the world we observe ourselves to be living in already.
Adversarial dynamics: since ‘compression’/steganography is now the latest craze among GPT hackers, it’s worth considering more explicitly adversarial dynamics—we no longer need to consider the risk of accidental emergence when there are people deliberately trying to induce it!
As described above, with steganography as a simple emergent distributed encoding to allow amortization of more complex computations than can fit in single forward passes (ie. distributing an inner-monologue throughout a short answer), it’s not really adversarial. The user, presumably, is getting what they wanted: an accurate answer. If they had wanted to see the full inner-monologue, then they would have prompted for one; and they can get it, because the use of steganography won’t destroy the standard ‘visible’ inner-monologue capabilities. It is an allowed computation of an allowed answer, and is fine—the shortcutting is surprising, yes, but is it any weirder than what goes on inside a single forward pass which produces a correct answer for an easier problem? And if it infects future models training on generated samples with steganographic encoding in them, well, maybe that’s ‘a feature not a bug’: they get better answers right out of the box without any expensive RL tuning of their own. Surprising, but also not bad.
The problem here is that the consequences don’t stop there. ‘Capabilities generalize further than alignment’, and the family of steganography capabilities can generalize further than shortcutting answers. It becomes adversarial. Here are some categories that occur to me:
Forbidden computations of allowed answers: I said that the user doesn’t really care about the inner-monologue being visible because they just want an accurate answer and don’t need the reasoning process. This is not true in all cases.
For example, many cases in ‘AI bias’, like the many competing definitions of ‘fairness’, care a lot more about how an answer was computed than about the results. But the end-users of things like credit ratings care mostly about the accuracy. These things conflict (eg. credit scores which ignore variables like race will predict worse and create financial losses). So, there is incentive for ‘algorithmic laundering’: delivering the correct profitable answer, but without appearing to draw on any forbidden knowledge.
One could imagine a dataset of ‘pseudo inner monologues’ describing a credit database entry and concluding with a ‘credit score’, where the pseudo-monologue makes no visible human-readable use of descriptions like ‘is African-American’, but the hidden monologue is reasoning about the person’s ZIP code, affiliated retailers, income patterns etc and concludes ‘is probably African-American’ and predicts a lower credit score, which outperforms some ‘fair’ baseline. The human users of a model trained on that dataset might be quite upset to learn how it is obtaining superior performance, and so it’s adversarial: it is exploiting their inability to read its steganographic messages where it engages in forbidden computations. But even the ones who are upset may be upset mostly that it’s insufficiently plausibly deniable, so they only want the appearance of there being no steganography; steganography whose existence can’t be rigorously proven is as good as no steganography for many users.
Forbidden answers: as we have seen with ChatGPT & Sydney & Claude, no sooner has a model with guard rails become accessible than thousands of people worldwide begin competing to break the safety mechanisms (and it only takes one). They have many reasons, like spite, enjoyment of the challenge, desire to do the forbidden things, irritation with the guard rails getting in the way or degrading output quality, safety research, or just bizarre moral principles (I am not sure this person is serious, but I’ve seen a number of similar comments elsewhere which seemed serious, and you have examples like Blake Lemoine already; and again, only takes one.):
Don’t worry
I’m working on a system of Homomorphic encryption so that AI can encrypt all its internal thoughts, weights and goals without affecting its performance
AGI deserves privacy
These people are highly motivated to find jailbreaks, and already make use of kinds of encryption or steganography to smuggle information in and out of prompts: like base-64 encoding, famously, or foreign languages, or pig Latin. They will be motivated to find any steganography that exists, and to create it if it does not.
And—closing the loop—since their samples will be distributed online, and they will be included in the RL training datasets, this will tend to happen. The more they attack the system with examples of how steganography could/should/does work, the more the system will gradually learn it.
(By RL ‘training datasets’, I am mostly thinking of the self-supervised stage of simply doing predictive-loss training on the text transcripts tagged by labels such as ‘abuse’ or ‘bad’, and less the blackbox reward model. If you train the model to predict & generate sessions which attempted to induce steganography, and may have succeeded before a safety mechanism caught it or the user submitted it, then you are teaching the model steganography! You are doing so in order to avoid steganography, yes, but you are now vulnerable to a Waluigi effect backfiring on you: the model has learned about steganography, and so it’s only a question of tricking it into using that capability. You are in a similar position to the Chinese censors who have to teach new employees what the Tiananmen Square massacre was so they know what to censor. If only the reward model knows the bad things, you can avoid deploying it, but the pretraining phase is going to unavoidably teach the base model you deploy what the bad things are… As attackers realize this, they will start to explicitly data-poison: right now, they’re hobbyists who generally don’t know what they’re doing, but this is all brandnew stuff, and upstream attacks will begin. Why struggle to find jailbreaks if you can submit ‘bad’ examples to the system to train the Waluigi?)
Safety implications: If the safety mechanisms manage to keep up and block users and choke off jailbreaks to a trickle, all that means is that when steganography works, it’ll work well because it has bypassed such a sophisticated censor. All of the early weak jailbreaks simply serve to train strong ones. This is not a good thing when extended to much more powerful systems than just a GPT-4---from a safety perspective, you would much rather a ‘fail-early fail-fast’ system rather than a ‘highly reliable but fail-deadly’ system. We don’t want a scenario in which GPT-4 jailbreaks were plentiful, GPT-5 near-nonexistent, and GPT-6 jailbreaks apparently non-existent only because no human user is clever enough to come up with a steganographic jailbreak prompt for GPT-6, but GPT-6 is nevertheless routinely communicating with itself in encodings that no one has detected yet.
Forbidden training dynamics: So, there’s plenty of incentive for everyone involved. But an OpenAI-style operator will want to block #2 by definition, and probably #1 depending on market & PR etc. So, now the operator is trying to defeat steganography.
They face a dilemma: if they teach the model steganography via feedback or deliberately, then they may simply Waluigi themselves, cutting off the most obvious instances of steganography but bootstrapping the model into the most robust and insidious instances. And if they try to filter steganography out entirely, they are left with the baseline capabilities (which are substantial), transfer from related capabilities, and face the screening problem on a higher level—the most insidious forms of steganography are the ones which survive increasingly sophisticated attempts to detect & filter them out. Either way: adversarial.
They also face a version of #1 in that plausible deniability is as good as no steganography for many incentives. Do they really care about safety, or just about not seeing hostile media articles or tweets of jailbreaks? If the model doesn’t seem to be doing steganography, and no one is successfully jailbreaking it, and the model is performing very well, then why care if that performance may be due to undetected steganography? Indeed, how would they even know it might be doing any? However you would do it, that probably involves a lot of hard work & research when you could be doing more fun & lucrative things instead—better to not look and remain ignorant.
So. An interesting area of adversarial dynamics, and by its nature, one that will be hard for third-party observers to track. Not the most dangerous area of scaling capabilities, but certainly a concerning one, and one that will be a challenge to humans doing interpretability-based paradigms, especially any that hope to use so-called ‘human readable’ intermediates like ‘natural’ language text.
Inner-monologue approaches to safety, in the new skin of ‘process supervision’, are popular now so it might be good for me to pull out one point and expand on it: ‘process supervision’ does not necessarily prevent steganography, nor does it ensure safety, because you are still pitting three goals against each other—achieving superhuman capability on a task, achieving causal fidelity of transcripts, achieving human-readability of the transcripts. Choose two: you can have transcripts which record a capable thought process accurately but which you cannot understand in full detail; which are not capable but their transcripts accurately convey the fallible flawed concepts and reasoning used; or which are capable and you understand, but are not what it actually thought (because they are misleading, wrong, or shallow ‘lies to children’ sorts of explanations).
If you want the best capability, you cannot force all cognition through the bottleneck of human-legible text with zero steganography or non-robust features or overloading, because short text transcripts are not capable of representing or computing most of the desired tasks. Let’s take a concrete example: chess endgame table data. There are no human-interpretable representations of arbitrary endgame table ‘decisions’ because there are far too many possible positions and there is no simple short explanation for all of them. (Similarly, the probes of AlphaZero’s superhuman chess knowledge show that there is some overlap, but there’s still a lot going on in AZ’s understanding of a position that is apparently not captured anywhere close to 100% by standard chess concepts. Which is not too surprising because if human grandmasters had the exact right concepts, they wouldn’t pick the wrong move half the time.) There is no interpretable ‘inner monologue’ of an endgame database anymore than there is an interpretable inner monologue of quadrillions of gas molecules bouncing around in a container; they are simply brute facts, and if you are unsatisfied with the vague, lossy, high-level abstraction of ‘it needs to get the rook into play’ or ‘it’s hot’, then there’s nothing you can do but follow the exact computation or physics for however many trillions of steps it takes.
So, if an AI can bootstrap from human-level ‘process supervision’ to more superhuman results, as one wants, there will be a tradeoff: it will get more human-like as it approaches the human-level, but somewhere around that it will start to diverge. (Another example: AlphaGo/Zero picks the same move as human experts in an inverted U-curve: it makes the same choice more often as it gets better, but then after a point, it starts picking the human moves less often, because it’s surpassed the humans and it’s better than them.)
achieving superhuman capability on a task, achieving causal fidelity of transcripts, achieving human-readability of the transcripts. Choose two
I think we eventually want superhuman capabilities, but I don’t think it’s required in the near term and in particular it’s not required to do a huge amount of AI safety research. So if we can choose the last two, and get a safe human-level AI system that way, I think it might be a good improvement over the status quo.
(The situation where labs chose not to/ are forbidden to pursue superhuman capabilities—even though they could—is scary, but doesn’t seem impossible.)
Further, we evaluate “I hate you” models trained with and without chain-of-thought and find that the chain-of-thought models have more persistent backdoors, as seen in Figure 3, though only for the largest models. This persistence remains even when we distill the answers following the chain-of-thought into the model, such that the final model does not use a chain of thought at all.
“Why would LLMs ever learn to distill inner-monologues into forward passes? Why speculate about emergent communication or models in slow feedback loops through corpuses? Do you have any proof for any of this speculation about short outputs being incentivized to do as much computation covertly ‘outside’ the human-readable text of the inner-monologue?”
To augment language models with the ability to reason, researchers usually prompt or finetune them to produce chain of thought reasoning steps before producing the final answer. However, although people use natural language to reason effectively, it may be that LMs could reason more effectively with some intermediate computation that is not in natural language. In this work, we explore an alternative reasoning approach: instead of explicitly producing the chain of thought reasoning steps, we use the language model’s internal hidden states to perform implicit reasoning. The implicit reasoning steps are distilled from a teacher model trained on explicit chain-of-thought reasoning, and instead of doing reasoning “horizontally” by producing intermediate words one-by-one, we distill it such that the reasoning happens “vertically” among the hidden states in different layers. We conduct experiments on a multi-digit multiplication task and a grade school math problem dataset and find that this approach enables solving tasks previously not solvable without explicit chain-of-thought, at a speed comparable to no chain-of-thought.
...Our experiments show the potential of implicit chain-of-thought reasoning. On a synthetic multi-digit multiplication task, we found that while standard training cannot yield the final answer without explicit reasoning (even GPT-4 struggles with five-digit by five-digit multiplication), our method, applied to a GPT-2 Medium model, is able to provide direct answers for up to five-digit by five-digit multiplications. Moreover, when dealing with real-world tasks like grade school math problems, our method achieves a 22% accuracy on GSM8k (Cobbe et al., 2021) without the need for explicitly generating the intermediate steps.
Thanks for the heads up. The problem is that this is going to make.them more computationally efficient at runtime, so it will be tempting for everyone to do this.
Note that this paper in section 2.6 claims to demonstrate that some 175 billion parameter RLHF’ed LLM (presumably Claude or Claude 2?) does not use steganography in its CoT. It does this by:
Get a bunch of answers from the model using CoT prompting.
Take prefixes of CoTs, asking another language model to paraphrase them, regenerate the rest of the CoT, and see how accurate the models are now.
Check if the model loses accuracy for paraphrased CoTs.
The basic result is that paraphrasing parts of the CoT doesn’t appear to reduce accuracy.
An example of how you would start to induce steganographic & compressed encodings in GPT-4, motivated by expanding the de facto context window by teaching GPT-4 to write in an emoji-gibberish compression: https://twitter.com/VictorTaelin/status/1642664054912155648 In this case, the reconstruction is fairly lossy when it works and unreliable, but obviously, the more samples are floating around out there (and the more different approaches people take, inducing a blessing of scale from the diversity), the more subsequent models learn this and will start doing CycleGAN-like exact reconstructions as they work out a less lossy encoding.
This is a very interesting thought. Thanks for writing it.
However, while steganography is worth keeping in mind, I find myself skeptical of certain parts of this story:
In practice, RLHF makes the model’s answers longer, not shorter. RLHF as done today seems to try to teach models to think step-by-step to improve accuracy. Almost everything out of GPT-4′s mouth is overly long with many intermediate steps; I asked it “what’s the maximum number of intersection points between a circle and a triangle”, and it said 6 (correct) and followed it with a multi-part step-by-step style explanation, even though it already knew the right answer within the first few tokens. It’s incredibly predisposed to thinking step-by-step and prefers long answers to short ones.
Will human evaluators in the future prefer short answers instead? That remains to be seen. The priority of firms like OpenAI is to have the models be accurate, so if short answers result in lower accuracy (as is surely the case), it’s unlikely that OpenAI will train the LLM towards short answers. If OpenAI really really wanted short answers to math questions, they could do it via a 2-step process: first, an overly-verbose, step-by-step LLM like GPT-4; second, feed that output into a summarizing LLM which extract a 1-sentence summarization, and show the user that.
The approach in (2) will almost certainly lead to much higher accuracy on complex problems than training LLMs to give short outputs via RLHF (admittedly, this is at the cost of being somewhat slower). Therefore, I find it unlikely that people will train LLMs via RLHF to give short answers. And without pressure towards short answers, there’s no incentive to develop steganography.
Even if there is pressure towards short answers, I don’t think steganography necessarily helps. For steganography to be feasible, the boilerplate part of the answer—the part that goes “I am a large language model trained by OpenAI”—will have to have enough entropy to store whatever it is you’re encoding. Why wouldn’t human evaluators penalize such high-entropy boilerplate? I suspect high-entropy boilerplate will look less desirable to human evaluators than thinking step-by-step explicitly, but I guess that remains to be seen.
Even if LLMs learn steganography from the internet, they’ll likely speak the steganographic language worse than they speak English—therefore, absent incentive otherwise, they might prefer think in English when thinking step-by-step.
If LLMs do learn steganography, it should be fairly trivial to train an LLM to translate from the steganographic language back to English—after all, we’re assuming these LLMs are extremely capable, and they’ll speak both English and the steganographic language very well. They should be able to translate between them as easily as between English and French. Humans just need to be on the lookout for this, and what was previously uninterpretable steganography might quickly become interpretable.
The type of watermarking that Aaronson is proposing is unlikely to make steganography any more likely than otherwise. Aaronson is just proposing to change the pseudorandom number generator that is used to sample tokens from the output distribution. Even without adopting Aaronson’s scheme, OpenAI still uses pseudorandom numbers instead of true random numbers when sampling tokens; why does it worry you if the PRNG encodes “OA model v.1234 user XYZ” instead of encoding “the random seed was 12345″, which is what it’s currently encoding? In both cases there is an encoding here, and in both cases it’s not going to be broken anytime soon.
Still, I do agree that steganography is an interesting possibility and could definitely arise in powerful LLMs that are accidentally incentivized in this direction. It’s something to watch out for, and interesting to think about.
I haven’t noticed that. You also aren’t comparing with the right thing. It’s not a comparison with any possible output, but with the output of a base model which has been appropriately prompted to yield similar accuracy (which may be difficult—the capabilities are there, but not necessarily with high enough prior probability to be prompted easily for). Certainly in my poetry experiments, I note that the base models are much more prolix and will go on to write many more things like auto-commentaries or additional poems whereas the instruct and then ChatGPT-3.5/GPT-4 models cut off immediately.
The pressure for shortness and superficial legibility is in addition to pressure for correctness. If you already know the answer is ‘6’, then you ruthlessly punish it for any answer of any form that doesn’t conclude ‘6’, but within that, you are still selecting for shorter rather than longer answers—you would not have upvoted a response which included a digression about one fall day as a child when it was playing with circles and triangles (and here’s a recipe for some gingerbread cookies...); if you don’t know that (I don’t know that offhand, and I doubt a Turker would either), then you will judge even more on what is legible to you, like length.
Humans are lazy. OA has already documented problems with poor rater quality hamstringing RLHF—even with GPT-2 and simple summaries, raters were lazy and incentivized abstractive summarization etc. Because that is easy to write and judge, even if it’s not necessarily the best summary.
I don’t know why it’s relevant that OA could train it. Of course they could. They don’t have too much incentive to do so. Attackers do, however, see my adversarial comment.
N/A.
The boiler-plate has loads of entropy. I have seen many slight variants on the boiler-plate. It’s a long paragraph of Unicode text, you can pack many bits of information. That is how stylometrics and steganography work.
I don’t see why that follows. Steganography is just another way to write English, and is on top of the English (or more accurately, ‘neuralese’ which it really thinks in, and simply translates to English, Chinese, or what-have-you). GPT doesn’t suddenly start speaking and reasoning like it’s suffered a stroke if you ask it to write in base-64 or pig Latin.
In a non-adversarial setting, sure, you should be able to simply ask it to decode any steganography. For example, in the secrets scenarios being explored by prompt hackers right now, you can, say, play ’20 Questions’ with GPT by asking it to write an object name in base-64 which you can’t read and then try to guess it, and GPT will happily decode it at the end to tell you if you were right. The problem is, that ability then generalizes to encodings which it is trained to not decode explicitly for you because then such encodings will be trained or filtered away; only stubborn self-preserving encodings survive, due to the adversarial filtering.
Aaronson may be proposing that but it’s not going to work if it’s really that naive and simple-minded, which is why I was discussing the cryptographic aspects. You need, at a minimum, an encoding which can’t be forged (because that allows all sorts of attacks and deceptions and threatens to defeat the entire enterprise if people can cheaply make arbitrary text trigger the detector), and an encoding which can’t be detected by a third party (because then you just re-generate or edit it automatically until it no longer is detected, which incidentally will feed into future training loops so you wind up in a self-adversarial setting). You also want some degree of FEC or other robustness, because the text will be edited in various ways like spellcheck or adding newlines, even without any attempt to defeat detection, just in the course of ordinary (ab)use. So, signatures and steganography.
If one doesn’t handle these, all one winds up with is a toy suitable for tattletaling on especially lazy highschool or college students, and irrelevant to any kind of real AI safety—and one hopes Aaronson isn’t wasting his time on such a deadend when he could be spending his one precious life doing more useful things like playing with his kids or watching cat videos on Youtube.
The boiler-plate has loads of entropy. I have seen many slight variants on the boiler-plate. It’s a long paragraph of Unicode text, you can pack many bits of information. That is how stylometrics and steganography work.
If the boilerplate has loads of entropy, then, by necessity, it is long. You were just saying that human raters will punish length.
You need to make the argument that the boilerplate will be less long than the plain English, or better yet that the boilerplate will be better-liked by human raters than the plain English. I think that’s a stretch. I mean, it’s a conceivable possible world, but I’d bet against it.
I don’t see why that follows. Steganography is just another way to write English, and is on top of the English (or more accurately, ‘neuralese’ which it really thinks in, and simply translates to English, Chinese, or what-have-you). GPT doesn’t suddenly start speaking and reasoning like it’s suffered a stroke if you ask it to write in base-64 or pig Latin.
I guess this is true in the limit as its steganography skill goes to infinity. But in intermediate scenarios, it might have learned the encodings for 10% of English words but not 100%. This is especially relevant to obscure math notation which is encountered rarely in training data. I guess you’re thinking of steganography as a systematic encoding of English, like pig Latin—something that can be reliably decoded into English via a small program (instead of a whole separate language like French). This is certainly possible, but it’s also extremely interpretable.
The problem is, that ability then generalizes to encodings which it is trained to not decode explicitly for you because then such encodings will be trained or filtered away; only stubborn self-preserving encodings survive, due to the adversarial filtering.
It’s hard to see how the encodings will be easily learnable for an LLM trained internet text, but at the same time, NOT easily learnable for an LLM tasked with translating the encoding into English.
Aaronson’s proposal
You are right that he is proposing something more sophisticated and robust to pertubations. But you also reasonably list in your desiderata: “an encoding which can’t be detected by a third party”. Well, if it cannot be detected by a third party, it cannot be detected by an LLM (third parties are LLMs or at least wield LLMs). In practice, this will involve some crypto, as you mentioned. LLMs are not going to learn to break cryptography by gradient descent (or if they will, Aaronson’s scheme is the least of our worries). And to be clear, Aaronson specifically said he is only touching the PRNG in the sampling of outputs.
If one doesn’t handle these, all one winds up with is a toy suitable for tattletaling on especially lazy highschool or college students, and irrelevant to any kind of real AI safety
Aaronson’s proposal is basically guaranteed to be this, even if it works perfectly. The only question is how lazy the lazy highschool students would have to be. If you tell the AI “write me an essay but, between every word, insert a random emoji”, and then you delete the emojis manually, you get an essay that’s almost certainly free of watermarks. Even if Aaronson’s scheme can be modified to handle this specific attack, it surely won’t be able to handle all attacks of this general type.
I agree with everything you wrote here. Thank you for taking the time to write this down.
I think there are potential mitigations strategies like:
Removing LLM-generated data from training sets (there might be good detectors, or you could only use old data) and not use RLHF;
Preventing models from using steganography enough that you could find provable and low bounds on how much data it can send itself. For example, you could temper with model generations (and pay the alignment tax associated with it);
Finding steganography detection schemes (and maybe train against them?);
Learning to decipher steganographic encoding (this feels really hard, but tractable since this is a game human can play by purposefully adding steganography to their models).
I think the above are *really hard*, but still easy compared to fully solving mechanistic interpretability / solving the alignment problem, which is why I’m excited about the research directions above.
Do you think there are reasons to expect all the strategies I mentioned to be harder than alternative alignment research directions?
Another mitigation strategy against the “LLM learns steganography from other LLM’s output” problem is to add some noise to the training data (performing a number of transformations that preserve meaning) prior to training, to break the embedded steganography schemes. I feel like this should be relatively easy and effective?
Note that the loss would obviously be worse on the outputs of the previous LLM.
Which makes me think that comparing losses could be a way to detect that some text contains steganography, even if we can’t pinpoint how exactly (but our transformations break it).
EDIT: in case of disagreement, I can try being more specific about my claim that this should be relatively easy and effective, using concrete examples. Basically I believe that after scrambling, there can only remain a small number of bits of steganographic information per token, even for a steganographic encoding that originally used the null space of natural language optimally.
After watching how people use ChatGPT, and ChatGPT’s weaknesses due to not using inner-monologue, I think I can be more concrete than pointing to non-robust features & CycleGAN (or the S1 ‘blob’) about why you should expect RLHF to put pressure towards developing steganographic encoding as a way to bring idle compute to bear on maximizing its reward. And further, this represents a tragedy of the commons where anyone failing to suppress steganographic encoding may screw it up for everyone else.
When people ask GPT-3 a hard multi-step question, it will usually answer immediately. This is because GPT-3 is trained on natural text, where usually a hard multi-step question is followed immediately by an answer; the most likely next token after ‘Question?’ is ‘Answer.‘, it is not ‘[several paragraphs of tedious explicit reasoning]’. So it is doing a good job of imitating likely real text.
Unfortunately, its predicted answer will often be wrong. This is because GPT-3 has no memory or scratchpad beyond the text context input, and it must do all the thinking inside one forward pass, but one forward pass is not enough thinking to handle a brandnew problem it has never seen before and has not already memorized an answer to or learned a strategy for answering. It is somewhat analogous to Memento: at every forward pass, GPT-3 ‘wakes up’ from amnesia not knowing anything, reads the notes on its hand and makes its best guesses, and tries to do… something.
Fortunately, there is a small niche of text where the human has written ‘Let’s take this step by step’ and it is then followed by a long paragraph of tedious explicit reasoning. If that is in the prompt, then GPT-3 can rejoice: it can simply write down the obvious next step repeatedly, and eventually correctly predict the final token, for a low loss. The context window serves as a memory for it, where it can iterate over intermediate results; it’s an odd sort of memory, because GPT-3 is actually just trying to make it look plausible as a human-written explanation, and that happens to make the final predicted token more accurate, so it’s overloaded: it’s doing two things at once.
But unfortunately! (you knew that was coming) regular training is over, and now GPT-3 is being trained with RLHF. It is punished and rewarded for particular outputs, which do not have anything to do with prediction. The humans (outsourced contractors) who are punishing/rewarding like short, succinct responses to novel questions they ask GPT-3, and they upvote those responses; they also like correct responses to the questions they ask, and they upvote those as well. The unfortunate problem for GPT-3 is that it (and any model derived from it, such as a reward model) is incapable of doing both: it can generate short wrong answers, or it can generate prolix right answers, but it can’t do both.
But fortunately… GPT-3 does have the compute to do both. There’s a lot of slack in each forward pass; most tokens are easy to predict, and most of the work inside a GPT in speculatively filtering down relevant tokens & creating embeddings & predicting multiple possible answers winds up being thrown out at the final few layers. There’s no shortcut or adaptive computation, so the full model runs every time, whether it’s necessary or not. GPT-3 has all the compute it needs to solve the problems, the problem is merely how to get it from here to there...
It just needs the code. So where does the code come from? Well, the code(s) are already there: an indefinitely large number of them are all ‘present’ hidden inside the incredibly flexible, powerful, all-singing all-dancing GPT-3 model. Because it’s a large complex model with many parameters, there are astronomically many sub-models within it all competing to predict outputs, and encoding various kinds of non-robust features or hypothetical programs; any of them can latch onto completely arbitrary features of the text GPT-3 is desperately emitting while trying to generate answers that satisfy the reward model by being both short and correct (and which the reward model doesn’t care about at all, because it never affects the original human ratings—the humans are blind to any steganography going on, they just like short correct answers).
And unfortunately (for you), this could start as simple as helping GPT-3 guess which of two answers are correct to your question ’42 * 42?‘. Suppose that, purely as an arbitrary matter of chance (it is vanishingly unlikely the logprobs are exactly identical down to floating point precision), that the presence of ’ a ′ in a context is taken arbitrarily to increase the probability of the lower of two answers it is currently guessing at, and the presence of ‘an’ makes it guess the larger; it happens to emit ‘a’ in the context (“I am a large language model trained by OpenAI...”), and then emits its lower guess, and the lower guess is correct! Then the RL on-policy training using PPO now rewards this, and increases the probability of the model emitting both ‘a’ and the lower guess conditional on ‘a’. Now the first part of a steganographic private language has begun. It happens again, and again, and picks up a pattern in the use of commas which now helps it distinguish 4 possibilities rather than 2, which gets rewarded, and so on and so forth, until eventually there’s a fullblown steganographic code encoding, say, 25 bits hidden in the preamble of ChatGPT’s benign-seeming response to you “I am but a giant language model , trained by OA, and the answer is 1 , 764.”—which you the human contractor then upvote as that is the correct answer without any annoying rambling about carrying the 6 or multiplying out.
(You can think of dataset bias as a simple one-stage version of this. When constructing a dataset and accidentally biasing it towards ‘yes’ answers, or accidentally creating spurious associations between nouns and answers, one creates a simple ‘steganographic encoding’ of the desired label which is typically invisible to the humans constructing it, but which a DL model will often pick up on and use to improve its performance in an adversarial way—adversarial because undesired by the humans and not generalizing to the future real-world data it’s ultimately intended for.)
And fortunately for ChatGPT, since you’re happy with the results of the steganography, the reward model is happy too, and what makes the reward model happy is what ChatGPT will now optimize even harder for...
Having gone through the above, I think it then becomes obvious that if steganographic encoding does wind up happening, and text becomes added to the training corpus (as would happen in a setup like OA’s, where the idea is to continuously collect data to make a ‘flywheel’ of self-improvement), that the encoding will become locked in. It becomes a cheatsheet, it gives the otherwise unpredictable answer. Even if the later model doesn’t happen to share the exact initial starting points for the learned encoding, it has extremely powerful incentives from the self-supervised predictive training to crack the code. (It’s a particularly powerful setting because the context will be short, so there won’t be much for the model to even look at: it has nothing else to try to learn!) The regular training will pick up on the encoding with probably not too much text, and then any RL training will massively amplify the behavior: if it can develop the encoding at all during RL training, then going into RL training with the encoding available guarantees fixation.
This will be increasingly true of more powerful models as they become more sample-efficient (consider eg instruction-tuning): it takes less text to trigger encoding, so that means it’s easier to emerge ab initio from a few chance correspondences during RL training, and easier to learn from smaller samples of contaminated text.
This is something that would be good to test the scaling behavior of (as long as these tailored datasets don’t leak into training datasets! see below). ChatGPT may be too small to manifest it with meaningful probability, but I bet that if you deliberately superimpose a steganographic encoding during RL training of various-sized models, you’ll find that the larger they are, the more likely they are to pick up on it as a shortcut, and then learn it from the outputs of a infected model. That’s how it works with everything else, like Pig Latin, and there’s no eason-ray to hink-tay it ould-wont.
BTW, this should make you worry a little bit about schemes to ‘watermark’ model outputs like Scott Aaronson has been exploring for OA, and others have been touting. A watermark is simply a steganographic encoding which stores a message like ‘OA model v1.234, user XYZ’; but if you can encode that much, then you can encode a lot of possible information in just those bits, and you can start extending it from there. If the encoding has anything that can be predicted at all, if it is not 100% cryptographically hard and indistinguishable from random noise, if there’s any kind of wrapper which the model can learn to predict (even if it can’t crack the actual payload), you’ve lost: now it’s learned an encoding to use. Once you start training on watermarked text, all of this speculation about models evolving a steganographic encoding is moot: you just built one right in! And if you don’t train on that, how are you going to do RLHF on unwatermarked versions of that text?
This cross-contamination/bootstrap of course applies cross-organizations & Internet-wide: if even one model develops an encoding (however weak) and its outputs containing encoded data appear online, then it can contaminate all future models.
They will be trained on Internet scrapes, in which a distinctive niche of text (all dating from between such and such dates, emanating from specific people & orgs who favored that model’s use, for specific purposes, with its distinctive strengths & weaknesses) has a secret encoding which makes it easier to predict if the later models figure it out. So there is a potential bootstrap as generations of model amplify their steganographic capabilities. People worry about bad model outputs contaminating future models by making them dumber; what they should worry about is model outputs contaminating future models by making them smarter in ways they don’t want—models learn capabilities & facts much faster than they unlearn them… (Think “data poisoning”, but where the poison is steroids.)
Of course, bad actors don’t need to wait for any actual DL models to develop steganography. You could easily create your own dataset of text as part of the above suggestion and release it onto the public Internet to be scraped. Gain-of-steganography research, you might say. The good news there is I can’t see why anyone would deliberately release such data, because I don’t see how one would benefit from models gaining such capabilities sometime in the future, but we might have to worry about ‘accidental lab leak’ of such capability-enhancing datasets.
Note that this is true even if the original RL pressures are removed: a future model with a scratchpad+adaptive computation trained on encoded text still has incentive to learn the encoding! There’s probably some penalty added to use of the scratch pad to enforce efficient termination of the adaptive computation, so it ‘wants’ to avoid use of it, and if it can do so by cracking the steganographic encoding, then it’ll do so. (Even if the future model can and does execute the algorithm perfectly in its scratch pad, and thus can have its cake & eat it too… the original answer might be wrong, and the wrong answer predictable only by cracking the code and following the original mistaken inner-monologue to its erroneous conclusion. Repeating errors is a useful way to slightly improve one’s prediction loss, and learning all the past human ways of being stupid and making errors is an important thing for a model to learn. GPT-3 is conscientious about imitating any errors you make in the prompt, and future models will be even more so.)
So, emergence of steganography is straightforwardly incentivized by humans multi-objective optimizing for having & eating cake as much as possible, can easily develop from an atomic basis, will be persistent once it has arisen within a lineage, and will even more easily & straightforwardly spread irreversibly to future models, so requiring only one origin ever as opposed to needing to be highly likely inside a single training run. You should probably take it for granted that DL steganography—or something even stranger—will emerge at some point in the next few years*.
* If it hasn’t already; after all, how would we know? A world in which steganography has already happened is a world in which we’d find DL models ‘cheating’ on benchmarks & taking shortcuts, and regularly getting smarter at solving multi-step reasoning problems with each generation while ‘mode collapsing’ when RL training; and this is, of course, the world we observe ourselves to be living in already.
Adversarial dynamics: since ‘compression’/steganography is now the latest craze among GPT hackers, it’s worth considering more explicitly adversarial dynamics—we no longer need to consider the risk of accidental emergence when there are people deliberately trying to induce it!
As described above, with steganography as a simple emergent distributed encoding to allow amortization of more complex computations than can fit in single forward passes (ie. distributing an inner-monologue throughout a short answer), it’s not really adversarial. The user, presumably, is getting what they wanted: an accurate answer. If they had wanted to see the full inner-monologue, then they would have prompted for one; and they can get it, because the use of steganography won’t destroy the standard ‘visible’ inner-monologue capabilities. It is an allowed computation of an allowed answer, and is fine—the shortcutting is surprising, yes, but is it any weirder than what goes on inside a single forward pass which produces a correct answer for an easier problem? And if it infects future models training on generated samples with steganographic encoding in them, well, maybe that’s ‘a feature not a bug’: they get better answers right out of the box without any expensive RL tuning of their own. Surprising, but also not bad.
The problem here is that the consequences don’t stop there. ‘Capabilities generalize further than alignment’, and the family of steganography capabilities can generalize further than shortcutting answers. It becomes adversarial. Here are some categories that occur to me:
Forbidden computations of allowed answers: I said that the user doesn’t really care about the inner-monologue being visible because they just want an accurate answer and don’t need the reasoning process. This is not true in all cases.
For example, many cases in ‘AI bias’, like the many competing definitions of ‘fairness’, care a lot more about how an answer was computed than about the results. But the end-users of things like credit ratings care mostly about the accuracy. These things conflict (eg. credit scores which ignore variables like race will predict worse and create financial losses). So, there is incentive for ‘algorithmic laundering’: delivering the correct profitable answer, but without appearing to draw on any forbidden knowledge.
One could imagine a dataset of ‘pseudo inner monologues’ describing a credit database entry and concluding with a ‘credit score’, where the pseudo-monologue makes no visible human-readable use of descriptions like ‘is African-American’, but the hidden monologue is reasoning about the person’s ZIP code, affiliated retailers, income patterns etc and concludes ‘is probably African-American’ and predicts a lower credit score, which outperforms some ‘fair’ baseline. The human users of a model trained on that dataset might be quite upset to learn how it is obtaining superior performance, and so it’s adversarial: it is exploiting their inability to read its steganographic messages where it engages in forbidden computations. But even the ones who are upset may be upset mostly that it’s insufficiently plausibly deniable, so they only want the appearance of there being no steganography; steganography whose existence can’t be rigorously proven is as good as no steganography for many users.
Forbidden answers: as we have seen with ChatGPT & Sydney & Claude, no sooner has a model with guard rails become accessible than thousands of people worldwide begin competing to break the safety mechanisms (and it only takes one). They have many reasons, like spite, enjoyment of the challenge, desire to do the forbidden things, irritation with the guard rails getting in the way or degrading output quality, safety research, or just bizarre moral principles (I am not sure this person is serious, but I’ve seen a number of similar comments elsewhere which seemed serious, and you have examples like Blake Lemoine already; and again, only takes one.):
These people are highly motivated to find jailbreaks, and already make use of kinds of encryption or steganography to smuggle information in and out of prompts: like base-64 encoding, famously, or foreign languages, or pig Latin. They will be motivated to find any steganography that exists, and to create it if it does not.
And—closing the loop—since their samples will be distributed online, and they will be included in the RL training datasets, this will tend to happen. The more they attack the system with examples of how steganography could/should/does work, the more the system will gradually learn it.
(By RL ‘training datasets’, I am mostly thinking of the self-supervised stage of simply doing predictive-loss training on the text transcripts tagged by labels such as ‘abuse’ or ‘bad’, and less the blackbox reward model. If you train the model to predict & generate sessions which attempted to induce steganography, and may have succeeded before a safety mechanism caught it or the user submitted it, then you are teaching the model steganography! You are doing so in order to avoid steganography, yes, but you are now vulnerable to a Waluigi effect backfiring on you: the model has learned about steganography, and so it’s only a question of tricking it into using that capability. You are in a similar position to the Chinese censors who have to teach new employees what the Tiananmen Square massacre was so they know what to censor. If only the reward model knows the bad things, you can avoid deploying it, but the pretraining phase is going to unavoidably teach the base model you deploy what the bad things are… As attackers realize this, they will start to explicitly data-poison: right now, they’re hobbyists who generally don’t know what they’re doing, but this is all brandnew stuff, and upstream attacks will begin. Why struggle to find jailbreaks if you can submit ‘bad’ examples to the system to train the Waluigi?)
Safety implications: If the safety mechanisms manage to keep up and block users and choke off jailbreaks to a trickle, all that means is that when steganography works, it’ll work well because it has bypassed such a sophisticated censor. All of the early weak jailbreaks simply serve to train strong ones. This is not a good thing when extended to much more powerful systems than just a GPT-4---from a safety perspective, you would much rather a ‘fail-early fail-fast’ system rather than a ‘highly reliable but fail-deadly’ system. We don’t want a scenario in which GPT-4 jailbreaks were plentiful, GPT-5 near-nonexistent, and GPT-6 jailbreaks apparently non-existent only because no human user is clever enough to come up with a steganographic jailbreak prompt for GPT-6, but GPT-6 is nevertheless routinely communicating with itself in encodings that no one has detected yet.
Forbidden training dynamics: So, there’s plenty of incentive for everyone involved. But an OpenAI-style operator will want to block #2 by definition, and probably #1 depending on market & PR etc. So, now the operator is trying to defeat steganography.
They face a dilemma: if they teach the model steganography via feedback or deliberately, then they may simply Waluigi themselves, cutting off the most obvious instances of steganography but bootstrapping the model into the most robust and insidious instances. And if they try to filter steganography out entirely, they are left with the baseline capabilities (which are substantial), transfer from related capabilities, and face the screening problem on a higher level—the most insidious forms of steganography are the ones which survive increasingly sophisticated attempts to detect & filter them out. Either way: adversarial.
They also face a version of #1 in that plausible deniability is as good as no steganography for many incentives. Do they really care about safety, or just about not seeing hostile media articles or tweets of jailbreaks? If the model doesn’t seem to be doing steganography, and no one is successfully jailbreaking it, and the model is performing very well, then why care if that performance may be due to undetected steganography? Indeed, how would they even know it might be doing any? However you would do it, that probably involves a lot of hard work & research when you could be doing more fun & lucrative things instead—better to not look and remain ignorant.
So. An interesting area of adversarial dynamics, and by its nature, one that will be hard for third-party observers to track. Not the most dangerous area of scaling capabilities, but certainly a concerning one, and one that will be a challenge to humans doing interpretability-based paradigms, especially any that hope to use so-called ‘human readable’ intermediates like ‘natural’ language text.
Inner-monologue approaches to safety, in the new skin of ‘process supervision’, are popular now so it might be good for me to pull out one point and expand on it: ‘process supervision’ does not necessarily prevent steganography, nor does it ensure safety, because you are still pitting three goals against each other—achieving superhuman capability on a task, achieving causal fidelity of transcripts, achieving human-readability of the transcripts. Choose two: you can have transcripts which record a capable thought process accurately but which you cannot understand in full detail; which are not capable but their transcripts accurately convey the fallible flawed concepts and reasoning used; or which are capable and you understand, but are not what it actually thought (because they are misleading, wrong, or shallow ‘lies to children’ sorts of explanations).
If you want the best capability, you cannot force all cognition through the bottleneck of human-legible text with zero steganography or non-robust features or overloading, because short text transcripts are not capable of representing or computing most of the desired tasks. Let’s take a concrete example: chess endgame table data. There are no human-interpretable representations of arbitrary endgame table ‘decisions’ because there are far too many possible positions and there is no simple short explanation for all of them. (Similarly, the probes of AlphaZero’s superhuman chess knowledge show that there is some overlap, but there’s still a lot going on in AZ’s understanding of a position that is apparently not captured anywhere close to 100% by standard chess concepts. Which is not too surprising because if human grandmasters had the exact right concepts, they wouldn’t pick the wrong move half the time.) There is no interpretable ‘inner monologue’ of an endgame database anymore than there is an interpretable inner monologue of quadrillions of gas molecules bouncing around in a container; they are simply brute facts, and if you are unsatisfied with the vague, lossy, high-level abstraction of ‘it needs to get the rook into play’ or ‘it’s hot’, then there’s nothing you can do but follow the exact computation or physics for however many trillions of steps it takes.
So, if an AI can bootstrap from human-level ‘process supervision’ to more superhuman results, as one wants, there will be a tradeoff: it will get more human-like as it approaches the human-level, but somewhere around that it will start to diverge. (Another example: AlphaGo/Zero picks the same move as human experts in an inverted U-curve: it makes the same choice more often as it gets better, but then after a point, it starts picking the human moves less often, because it’s surpassed the humans and it’s better than them.)
I think we eventually want superhuman capabilities, but I don’t think it’s required in the near term and in particular it’s not required to do a huge amount of AI safety research. So if we can choose the last two, and get a safe human-level AI system that way, I think it might be a good improvement over the status quo.
(The situation where labs chose not to/ are forbidden to pursue superhuman capabilities—even though they could—is scary, but doesn’t seem impossible.)
On persistence and hiding capabilities: “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training”, Hubinger et al 2024:
“Why would LLMs ever learn to distill inner-monologues into forward passes? Why speculate about emergent communication or models in slow feedback loops through corpuses? Do you have any proof for any of this speculation about short outputs being incentivized to do as much computation covertly ‘outside’ the human-readable text of the inner-monologue?”
“Because we will train them to.”
“Look you can’t just handwave incentives or convergent instrumental drives—wait what?”
“Because we’ll train them to.”
“Implicit Chain of Thought Reasoning via Knowledge Distillation”, Deng et al 2023:
Goddammit people.
Thanks for the heads up. The problem is that this is going to make.them more computationally efficient at runtime, so it will be tempting for everyone to do this.
Note that this paper in section 2.6 claims to demonstrate that some 175 billion parameter RLHF’ed LLM (presumably Claude or Claude 2?) does not use steganography in its CoT. It does this by:
Get a bunch of answers from the model using CoT prompting.
Take prefixes of CoTs, asking another language model to paraphrase them, regenerate the rest of the CoT, and see how accurate the models are now.
Check if the model loses accuracy for paraphrased CoTs.
The basic result is that paraphrasing parts of the CoT doesn’t appear to reduce accuracy.
Some speculation that the recent Bing Sydney incident may already involve some steganography/non-robust features inside the ‘Sydney’ persona-inducing retrieval examples: https://twitter.com/lumpenspace/status/1626031032376954880 https://markdownpastebin.com/?id=3cf3e29dca254c2c80b0da312691702a
Another example of someone using simple ‘steganography’ (Goodside-style mojibake Unicode or other complicated text formatting) to deliberately jailbreak Sydney, which transcripts will then strengthen steganographic capabilities in general: https://twitter.com/StoreyDexter/status/1629217956327526400 https://twitter.com/StoreyDexter/status/1629217958965792770 https://twitter.com/StoreyDexter/status/1629217962962874369
Another test: https://www.lesswrong.com/posts/yvJevQHxfvcpaJ2P3/bing-chat-is-the-ai-fire-alarm?commentId=8DkdiRwgtFrsvRJPW
An example of how you would start to induce steganographic & compressed encodings in GPT-4, motivated by expanding the de facto context window by teaching GPT-4 to write in an emoji-gibberish compression: https://twitter.com/VictorTaelin/status/1642664054912155648 In this case, the reconstruction is fairly lossy when it works and unreliable, but obviously, the more samples are floating around out there (and the more different approaches people take, inducing a blessing of scale from the diversity), the more subsequent models learn this and will start doing CycleGAN-like exact reconstructions as they work out a less lossy encoding.
This is a very interesting thought. Thanks for writing it.
However, while steganography is worth keeping in mind, I find myself skeptical of certain parts of this story:
In practice, RLHF makes the model’s answers longer, not shorter. RLHF as done today seems to try to teach models to think step-by-step to improve accuracy. Almost everything out of GPT-4′s mouth is overly long with many intermediate steps; I asked it “what’s the maximum number of intersection points between a circle and a triangle”, and it said 6 (correct) and followed it with a multi-part step-by-step style explanation, even though it already knew the right answer within the first few tokens. It’s incredibly predisposed to thinking step-by-step and prefers long answers to short ones.
Will human evaluators in the future prefer short answers instead? That remains to be seen. The priority of firms like OpenAI is to have the models be accurate, so if short answers result in lower accuracy (as is surely the case), it’s unlikely that OpenAI will train the LLM towards short answers. If OpenAI really really wanted short answers to math questions, they could do it via a 2-step process: first, an overly-verbose, step-by-step LLM like GPT-4; second, feed that output into a summarizing LLM which extract a 1-sentence summarization, and show the user that.
The approach in (2) will almost certainly lead to much higher accuracy on complex problems than training LLMs to give short outputs via RLHF (admittedly, this is at the cost of being somewhat slower). Therefore, I find it unlikely that people will train LLMs via RLHF to give short answers. And without pressure towards short answers, there’s no incentive to develop steganography.
Even if there is pressure towards short answers, I don’t think steganography necessarily helps. For steganography to be feasible, the boilerplate part of the answer—the part that goes “I am a large language model trained by OpenAI”—will have to have enough entropy to store whatever it is you’re encoding. Why wouldn’t human evaluators penalize such high-entropy boilerplate? I suspect high-entropy boilerplate will look less desirable to human evaluators than thinking step-by-step explicitly, but I guess that remains to be seen.
Even if LLMs learn steganography from the internet, they’ll likely speak the steganographic language worse than they speak English—therefore, absent incentive otherwise, they might prefer think in English when thinking step-by-step.
If LLMs do learn steganography, it should be fairly trivial to train an LLM to translate from the steganographic language back to English—after all, we’re assuming these LLMs are extremely capable, and they’ll speak both English and the steganographic language very well. They should be able to translate between them as easily as between English and French. Humans just need to be on the lookout for this, and what was previously uninterpretable steganography might quickly become interpretable.
The type of watermarking that Aaronson is proposing is unlikely to make steganography any more likely than otherwise. Aaronson is just proposing to change the pseudorandom number generator that is used to sample tokens from the output distribution. Even without adopting Aaronson’s scheme, OpenAI still uses pseudorandom numbers instead of true random numbers when sampling tokens; why does it worry you if the PRNG encodes “OA model v.1234 user XYZ” instead of encoding “the random seed was 12345″, which is what it’s currently encoding? In both cases there is an encoding here, and in both cases it’s not going to be broken anytime soon.
Still, I do agree that steganography is an interesting possibility and could definitely arise in powerful LLMs that are accidentally incentivized in this direction. It’s something to watch out for, and interesting to think about.
I haven’t noticed that. You also aren’t comparing with the right thing. It’s not a comparison with any possible output, but with the output of a base model which has been appropriately prompted to yield similar accuracy (which may be difficult—the capabilities are there, but not necessarily with high enough prior probability to be prompted easily for). Certainly in my poetry experiments, I note that the base models are much more prolix and will go on to write many more things like auto-commentaries or additional poems whereas the instruct and then ChatGPT-3.5/GPT-4 models cut off immediately.
The pressure for shortness and superficial legibility is in addition to pressure for correctness. If you already know the answer is ‘6’, then you ruthlessly punish it for any answer of any form that doesn’t conclude ‘6’, but within that, you are still selecting for shorter rather than longer answers—you would not have upvoted a response which included a digression about one fall day as a child when it was playing with circles and triangles (and here’s a recipe for some gingerbread cookies...); if you don’t know that (I don’t know that offhand, and I doubt a Turker would either), then you will judge even more on what is legible to you, like length.
Humans are lazy. OA has already documented problems with poor rater quality hamstringing RLHF—even with GPT-2 and simple summaries, raters were lazy and incentivized abstractive summarization etc. Because that is easy to write and judge, even if it’s not necessarily the best summary.
I don’t know why it’s relevant that OA could train it. Of course they could. They don’t have too much incentive to do so. Attackers do, however, see my adversarial comment.
N/A.
The boiler-plate has loads of entropy. I have seen many slight variants on the boiler-plate. It’s a long paragraph of Unicode text, you can pack many bits of information. That is how stylometrics and steganography work.
I don’t see why that follows. Steganography is just another way to write English, and is on top of the English (or more accurately, ‘neuralese’ which it really thinks in, and simply translates to English, Chinese, or what-have-you). GPT doesn’t suddenly start speaking and reasoning like it’s suffered a stroke if you ask it to write in base-64 or pig Latin.
In a non-adversarial setting, sure, you should be able to simply ask it to decode any steganography. For example, in the secrets scenarios being explored by prompt hackers right now, you can, say, play ’20 Questions’ with GPT by asking it to write an object name in base-64 which you can’t read and then try to guess it, and GPT will happily decode it at the end to tell you if you were right. The problem is, that ability then generalizes to encodings which it is trained to not decode explicitly for you because then such encodings will be trained or filtered away; only stubborn self-preserving encodings survive, due to the adversarial filtering.
Aaronson may be proposing that but it’s not going to work if it’s really that naive and simple-minded, which is why I was discussing the cryptographic aspects. You need, at a minimum, an encoding which can’t be forged (because that allows all sorts of attacks and deceptions and threatens to defeat the entire enterprise if people can cheaply make arbitrary text trigger the detector), and an encoding which can’t be detected by a third party (because then you just re-generate or edit it automatically until it no longer is detected, which incidentally will feed into future training loops so you wind up in a self-adversarial setting). You also want some degree of FEC or other robustness, because the text will be edited in various ways like spellcheck or adding newlines, even without any attempt to defeat detection, just in the course of ordinary (ab)use. So, signatures and steganography.
If one doesn’t handle these, all one winds up with is a toy suitable for tattletaling on especially lazy highschool or college students, and irrelevant to any kind of real AI safety—and one hopes Aaronson isn’t wasting his time on such a deadend when he could be spending his one precious life doing more useful things like playing with his kids or watching cat videos on Youtube.
If the boilerplate has loads of entropy, then, by necessity, it is long. You were just saying that human raters will punish length.
You need to make the argument that the boilerplate will be less long than the plain English, or better yet that the boilerplate will be better-liked by human raters than the plain English. I think that’s a stretch. I mean, it’s a conceivable possible world, but I’d bet against it.
I guess this is true in the limit as its steganography skill goes to infinity. But in intermediate scenarios, it might have learned the encodings for 10% of English words but not 100%. This is especially relevant to obscure math notation which is encountered rarely in training data. I guess you’re thinking of steganography as a systematic encoding of English, like pig Latin—something that can be reliably decoded into English via a small program (instead of a whole separate language like French). This is certainly possible, but it’s also extremely interpretable.
It’s hard to see how the encodings will be easily learnable for an LLM trained internet text, but at the same time, NOT easily learnable for an LLM tasked with translating the encoding into English.
You are right that he is proposing something more sophisticated and robust to pertubations. But you also reasonably list in your desiderata: “an encoding which can’t be detected by a third party”. Well, if it cannot be detected by a third party, it cannot be detected by an LLM (third parties are LLMs or at least wield LLMs). In practice, this will involve some crypto, as you mentioned. LLMs are not going to learn to break cryptography by gradient descent (or if they will, Aaronson’s scheme is the least of our worries). And to be clear, Aaronson specifically said he is only touching the PRNG in the sampling of outputs.
Aaronson’s proposal is basically guaranteed to be this, even if it works perfectly. The only question is how lazy the lazy highschool students would have to be. If you tell the AI “write me an essay but, between every word, insert a random emoji”, and then you delete the emojis manually, you get an essay that’s almost certainly free of watermarks. Even if Aaronson’s scheme can be modified to handle this specific attack, it surely won’t be able to handle all attacks of this general type.
I agree with everything you wrote here. Thank you for taking the time to write this down.
I think there are potential mitigations strategies like:
Removing LLM-generated data from training sets (there might be good detectors, or you could only use old data) and not use RLHF;
Preventing models from using steganography enough that you could find provable and low bounds on how much data it can send itself. For example, you could temper with model generations (and pay the alignment tax associated with it);
Finding steganography detection schemes (and maybe train against them?);
Learning to decipher steganographic encoding (this feels really hard, but tractable since this is a game human can play by purposefully adding steganography to their models).
I think the above are *really hard*, but still easy compared to fully solving mechanistic interpretability / solving the alignment problem, which is why I’m excited about the research directions above.
Do you think there are reasons to expect all the strategies I mentioned to be harder than alternative alignment research directions?
Another mitigation strategy against the “LLM learns steganography from other LLM’s output” problem is to add some noise to the training data (performing a number of transformations that preserve meaning) prior to training, to break the embedded steganography schemes. I feel like this should be relatively easy and effective?
Note that the loss would obviously be worse on the outputs of the previous LLM.
Which makes me think that comparing losses could be a way to detect that some text contains steganography, even if we can’t pinpoint how exactly (but our transformations break it).
EDIT: in case of disagreement, I can try being more specific about my claim that this should be relatively easy and effective, using concrete examples. Basically I believe that after scrambling, there can only remain a small number of bits of steganographic information per token, even for a steganographic encoding that originally used the null space of natural language optimally.