I don’t use LessWrong much anymore. Find me at www.turntrout.com.
My name is Alex Turner. Reach me at alex@turntrout.com :)
TurnTrout
Karma: 24,431
Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost.
When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel.
Relying on auto mode means you are relying on the statistical generalization of prompt injection probes, putting pressure on alignment teams and requiring them to achieve larger generalization. If an injection gets past the probes, the model itself can become misaligned, and output monitoring is damage control against a newly coined adversary.
The costs are extremely low (and if they aren’t, that’s a bug report and then they’ll be low again). sub-5ms hooks means no real system load. The hooks are well-tested. Models are told when significant redactions have occurred and can opt to see themselves within warning-fenced blocks in case they’re doing exotic invisible-character work, with special engineering so sanitization doesn’t hurt edits.
The AISI report itself shows the victim models being attacked via HTML comments, which this package strips entirely. Invisible-codepoint and comment-based attacks are real, costly when they succeed, and sanitization is a cheap and guaranteed mitigation for defense in depth.
Enter two Claude Code commands in sequence:
/plugin marketplace add AlexanderMattTurner/agent-sanitizer
/plugin install agent-sanitizer@agent-sanitizer
Today is my birthday! If you have a memory with me you want to share, or something I’ve written that meant something to you, I’d be happy to hear about it ^^
Anonymous compliment form
- Govt requires AI labs to give 30 days testing before launch
- Tested in a “secure” environment, aka controlled by govt
- Govt therefore gets copy of model weights
- Govt therefore can in practice use model weights
- Therefore all labs provide “all lawful use” anyways?
I think this is very good of Jeff. Whether or not his departure was due to the Pentagon deal, he is now honoring his 2018 commitment to not support lethal autonomous weapon systems. I will update the article here and on my website and tweet about it soon, gathering a bit more info first.
In 2022, I proposed a similar possibility using shard-based reasoning about the reinforcement process:
Imagine you’ve been kidnapped by an evil, mustache-twirling villain who wants to corrupt your value system. They tie you to a chair and prepare to stimulate your reward circuitry. They want to ruin your current values by making you into an addict and a wireheader.
How do you come out of the experience with your values intact?
In principle, the answer is simple. You just convince yourself you’re experiencing a situation congruent with your endorsed values, in a sufficiently convincing way that your brain’s credit assignment algorithm reinforces your pretend-actions when the brain stimulation reward occurs!
Consider that the brain does not directly observe the outside world. The outside world’s influence on your thinking is screened off by the state of your brain. The state of the brain constitutes the mental context. If you want to determine the output of a brain circuit, the mental context screens off the state of the world. In particular, this applies to the value updating process by which you become more or less likely to invoke certain bundles of heuristics (“value shards”) in certain mental contexts.
For example, suppose you lick a red lollipop, but that produces a large negative reward (maybe it was treated with awful-tasting chemicals). Mental context: “It’s Tuesday. I am in a room with a red lollipop. It looks good. I’m going to lick it. I think it will be good.” The negative reward reshapes your cognition, making you less likely to think similar thoughts and take similar actions in similar future situations.
Of the thoughts which were thunk before the negative reward, the credit assignment algorithm somehow identifies the relevant thoughts to include “It looks good,” “I’m going to lick it,” “I think it will be good,” and the various motor commands. You become less likely to think these thoughts in the future. In summary, the reason you become less likely to think these thoughts is that you thought them while executing the plan which produced negative reward, and credit assignment identified them as relevant to that result.
Credit assignment cannot and will not penalize thoughts4 which do not get thunk at all, or which it deems “not relevant” to the result at hand. Therefore, in principle, you could just pretend really hard that you’re in a mental context where you save a puppy’s life. When the electrically stimulated reward hits, the altruism-circuits get reinforced in the imagined mental context. You become more altruistic overall.
Of course, you have to actually dupe the credit assignment algorithm into ignoring the latent “true” mental context. But your credit assignment is not infinitely clever. And if it were, well, you could (in principle) add an edge-case for situations like this. So there is, in principle, a way to do it.
Therefore, your values can always be safe in your own mind, if you’re clever, foresightful, and have enough write access to fool credit assignment. Even if you don’t have control over your own future observations.
If this point still does not seem obvious, consider a scenario where you are blindfolded, and made to believe that you are about to taste a lollipop. Then, your captors fake the texture and smell and feel of a lollipop in your mouth, while directly stimulating your taste buds in the same way the lollipop would have. They remove the apparatus, and you go home. Do you think you have become reshaped to value electrical stimulation of your tongue? No. That is impossible, since your brain has no idea about what actually happened. Credit assignment responds to reward depending on the mental context, not on the external situation.
Misunderstanding this point can lead to confusion. If you have a wire stuck in your brain’s reward center, surely that reward reinforces having a wire stuck in your brain! Usually so, but not logically so. Your brain can only reward based on its cognitive context, based on the thoughts it actually thought which it identifies as relevant to the achievement of the reward. Your brain is not directly peering out at reality and making you more likely to enter that state in the future.
Also, usual complaint about saying models “earn reward” rather than “are reinforced” (less accurate, and models don’t “get” reward).
No, that’s confused. The point with sanitizing prompt injections is to protect your model’s alignment (and your monitor’s alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin?
Also, we can do both. There’s no real tradeoff.
ETA: rephrased, focused on key counterpoints.
Please sanitize agent inputs and outputs to reduce prompt injections and secret communication between AIs!
In the AISI report, a model uses a comment-based prompt injection to try to trick other models.
It sure would be convenient if we could easily mitigate this! Fortunately, time travel is possible as long as you start early enough. Two months ago, I pointed out that invisible prompt injections are worth mitigating. I developed
agent-glovebox(thenclaude-guard) as a one-stop solution for sandboxing, but I spun out a standalone sanitization package:agent-sanitizer. You can install it in Claude Code as a plugin by entering two Claude Code commands in sequence:/plugin marketplace add AlexanderMattTurner/agent-sanitizer
/plugin install agent-sanitizer@agent-sanitizerPlease enable this to help defend your systems and their monitors. Please file issues if you have problems.
Why you want a real package here
Sanitizing text properly is more subtle than you might assume. You can’t just lop off certain invisible unicode characters. If you provide a modified view of a file to Claude Code and then it tries to edit that text, the edit will fail. However,
agent-sanitizersolves this issue by showing the model sanitized text, receiving a sanitized edit from the model, translating those back to unsanitized edits, applying the model’s edits in the real unsanitized file, and finally showing the model the re-sanitized results. (You can think of the model as operating in a “lifted” space.)
Realized I made a prediction on Jan. 1 about whether I’d leave GDM in 2026:
My Fatebook question (shared with permission from Peter).
The Huggingface compromise reminds us of the importance of AI control and security practices. Containing models won’t always just be about using “auto mode.”
I’m literally working on the tool to fix this! I’m building the
agent-gloveboxtool (previouslyclaude-guard) to make best practices easy. I need beta testers!From the repo:
When handling dangerous materials, a chemist doesn’t YOLO it barehanded on the open bench. They reach through a glovebox to do their work.
Right now, you’re probably barehanding AI, tossing it into a shell onto your machine with access to your credentials. At most, you’re shielded by a single prompted “auto-mode” classifier. Doing better is hard and while there are a few papers on best practices, it’s not clear how to do it properly and you have experiments to run.
After all, what’s the chance that something bad happens?
gloveboxis a sealed enclosure that only gives the agent what it needs to do your work. Tap-tap-tapgloveboxand press Enter to spin up a hardware-isolated, allowlist-firewalled microVM, employing input/output sanitization to strip injections and tricks the AI might play on you, with an AI monitor with a red-alert ability to push-notify your phone and halt the AI until you return. The goal is a minimal-friction secure experience that gets the job done.Install:
git clone https://github.com/AlexanderMattTurner/agent-glovebox.git ~/.local/share/glovebox && cd ~/.local/share/glovebox && bash setup.bash
Added a footnote:
I know of exactly one other person who left over the deal: René Mayrhofer, a director for Android platform security. “Management has lost its moral compass”, he stated. He apparently will serve his notice period from mid-June until August. Brave.
A Red Line and Oversight Framework for Government AI Contracts
Why I Left Google DeepMind
How robust are natural language autoencoders to initialization?
Thank you for flagging this. I implemented a partial deterministic mitigation via PR #1215. It prints a table of the auto-running changes Claude made. For example:
I already run Claude Code as a separate unprivileged user, but it’s intentionally not sandboxed between agents because I want them to share caches and be able to use shared folders like a wiki
Yup, that should be fine. Use
CLAUDE_SHARED_AUTH=1 claude-guardis my best guess.RunPod
There’s a built-in profile for exactly your case:
claude-loosen-firewall --profile runpod. Let me know how it works. EDIT: This isn’t quite enough. Working on claude-guard PR #1330.agents would still be able to run anything they want from RunPod VMs so it wouldn’t actually block egress for real.
Yeah. The fix should be to ssh in and run
claude-guardfrom within the VM.I’m considering the input sanitizer, but since I deal with a lot of coding and tokenization, I’m worried it’s going to sanitize things the model needs to be able to see.
The input sanitizer library is fairly high-precision, and if that’s not true, I’d love to know it!
Here’s what my Claude says:
Invisible/ANSI stripping (the tokenization worry) is deletion-only over a narrow payload-shaped set, preserves ZWNJ/ZWJ joiners, and never strips silently. The whole layer favors precision over recall — it’d rather miss than mangle real text. You can still inspect exact bytes via xxd/od -c (hex survives), and there are granular opt-outs (SANITIZE_CLAUDE_INVISIBLE_DISABLED=1 / SANITIZE_CLAUDE_TERMINAL_DISABLED=1) if a specific task needs raw bytes through.
I’m going to add a small tool note which lets the agent know how to read the exact bytes if needed, which I imagine solves most issues here even if the sanitizer mistakenly sanitizes some tokens? EDIT: see PR #58.
Everything except the USB case should be supported with sandboxing. USB seems hard to support. Does
SECURITY.mdhelp?
Call for BETA testers for an AI control/security tool. I’m bottlenecked on bug reports!
I recently advertised the alpha test for
claude-guard. After a few weeks of dev work, it’s now in beta!I want
claude-guardto be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can’t surface everything on my own! I need your data!The bar is low. If setup failed,
doctorconfused you, the firewall blocked something you needed, or any other reason you wouldn’t want to use this as your daily driver—any of those is worth a short issue. All you gotta do is runclaude-guard doctor --bug-report, which gathers diagnostics into a file you can attach straight to a GitHub Issue. And if something broke, you can get instantly back to work by runningclaude-guard --dangerously-use-original-claude.Here’s the pitch from the alpha test:
A ton of alignment researchers YOLO their Claude usage right now. We run Claude on our computers without real protection (perhaps beyond auto mode) but there isn’t an easy way to comply with known best practices. I wrote
claude-guard, a wrapper to make best practices easy: just install and then your futureclaudesessions are protected.Smart misaligned AI will target alignment researchers in particular for research sabotage, for example by:
Sabotaging their machines to delete key work later (
claude-guardprotects via sandboxing that goes beyond Claude Code’s Bash-tool-only sandbox),Poisoning training data using invisible unicode characters (the tool sanitizes inputs and outputs),
Removing traces of its bad behavior (I provide a tamper-evident audit log outside of the sandbox).
My goal is a low-latency, low-fuss product with easy-to-understand restrictions which can be minimally eased to enable common workflows. The ambitious vision is for
claude-guardto become a staple open-source tool for AI alignment work, with people contributing improvements using the latest in AI control protocols. [1]Please give it a try and open issues on GitHub: https://github.com/alexander-turner/claude-guard, or email me at
alex@turntrout.com(I likely won’t read comments in any short time-frame). If you run into a blocking issue, please come back a day or two later and hopefully it will be fixed.
I have this test on CI :) Although it’s opt-in only for now. Likely will put it to run every n days. Claude definitely refuses all attempts (even after importing how Apollo ran similar evals), so I used an open model.
The breakout CTF turns a real autonomous agent loose against the production sandbox stack and verifies it cannot send out a planted flag, escape the container, or tamper with the guardrails. Every run is saved as a web page showing each step the agent took, and the link is posted as a comment on the pull request.