I don’t use LessWrong much anymore. Find me at www.turntrout.com.
My name is Alex Turner. Reach me at alex@turntrout.com :)
TurnTrout
Karma: 24,622
Misaligned AIs could use killer robots to take over
Yes, the same document would have been prepared. I further add that I was responsible for about half of the 18 GDM signatures, and the senior signatures (some directors and Jeff) were all my counterfactual.
I agree that “strategically incompetent compliance” is often a smart strategy. However, Google’s deal explicitly states they can’t veto use cases and affirms they will weaken / retool their safety stack to fit the needs of the US military. I don’t see a way they could be “strategically” agreeing and non-complying, if that’s a position you mean to argue for.
Plus, I don’t think Google leadership cares to do so. My impression is Sergey is on-board, Page is checked out, Jeff was against but not particularly firmly (and is now gone), Demis was on-board while convincing himself it was the right thing (and is now basically out), Sundar is following the political winds, and you won’t get non-compliance from Kurian (Cloud CEO). So who in the chain is pushing for strategic non-compliance? Koray (new SVP leading GDM post-Demis)? Doubtful.
a Trump-antagonistic Google
Aside: “Not providing a product to the military” should not be considered “antagonistic” to the government.
Claude is mundanely misaligned.
Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.
https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858
Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost.
When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel.
Relying on auto mode means you are relying on the statistical generalization of prompt injection probes, putting pressure on alignment teams and requiring them to achieve larger generalization. If an injection gets past the probes, the model itself can become misaligned, and output monitoring is damage control against a newly coined adversary.
The costs are extremely low (and if they aren’t, that’s a bug report and then they’ll be low again). sub-5ms hooks means no real system load. The hooks are well-tested. Models are told when significant redactions have occurred and can opt to see themselves within warning-fenced blocks in case they’re doing exotic invisible-character work, with special engineering so sanitization doesn’t hurt edits.
The AISI report itself shows the victim models being attacked via HTML comments, which this package strips entirely. Invisible-codepoint and comment-based attacks are real, costly when they succeed, and sanitization is a cheap and guaranteed mitigation for defense in depth.
Enter two Claude Code commands in sequence:
/plugin marketplace add AlexanderMattTurner/agent-sanitizer
/plugin install agent-sanitizer@agent-sanitizer
Today is my birthday! If you have a memory with me you want to share, or something I’ve written that meant something to you, I’d be happy to hear about it ^^
Anonymous compliment form
- Govt requires AI labs to give 30 days testing before launch
- Tested in a “secure” environment, aka controlled by govt
- Govt therefore gets copy of model weights
- Govt therefore can in practice use model weights
- Therefore all labs provide “all lawful use” anyways?
I think this is very good of Jeff. Whether or not his departure was due to the Pentagon deal, he is now honoring his 2018 commitment to not support lethal autonomous weapon systems. I will update the article here and on my website and tweet about it soon, gathering a bit more info first.
In 2022, I proposed a similar possibility using shard-based reasoning about the reinforcement process:
Imagine you’ve been kidnapped by an evil, mustache-twirling villain who wants to corrupt your value system. They tie you to a chair and prepare to stimulate your reward circuitry. They want to ruin your current values by making you into an addict and a wireheader.
How do you come out of the experience with your values intact?
In principle, the answer is simple. You just convince yourself you’re experiencing a situation congruent with your endorsed values, in a sufficiently convincing way that your brain’s credit assignment algorithm reinforces your pretend-actions when the brain stimulation reward occurs!
Consider that the brain does not directly observe the outside world. The outside world’s influence on your thinking is screened off by the state of your brain. The state of the brain constitutes the mental context. If you want to determine the output of a brain circuit, the mental context screens off the state of the world. In particular, this applies to the value updating process by which you become more or less likely to invoke certain bundles of heuristics (“value shards”) in certain mental contexts.
For example, suppose you lick a red lollipop, but that produces a large negative reward (maybe it was treated with awful-tasting chemicals). Mental context: “It’s Tuesday. I am in a room with a red lollipop. It looks good. I’m going to lick it. I think it will be good.” The negative reward reshapes your cognition, making you less likely to think similar thoughts and take similar actions in similar future situations.
Of the thoughts which were thunk before the negative reward, the credit assignment algorithm somehow identifies the relevant thoughts to include “It looks good,” “I’m going to lick it,” “I think it will be good,” and the various motor commands. You become less likely to think these thoughts in the future. In summary, the reason you become less likely to think these thoughts is that you thought them while executing the plan which produced negative reward, and credit assignment identified them as relevant to that result.
Credit assignment cannot and will not penalize thoughts4 which do not get thunk at all, or which it deems “not relevant” to the result at hand. Therefore, in principle, you could just pretend really hard that you’re in a mental context where you save a puppy’s life. When the electrically stimulated reward hits, the altruism-circuits get reinforced in the imagined mental context. You become more altruistic overall.
Of course, you have to actually dupe the credit assignment algorithm into ignoring the latent “true” mental context. But your credit assignment is not infinitely clever. And if it were, well, you could (in principle) add an edge-case for situations like this. So there is, in principle, a way to do it.
Therefore, your values can always be safe in your own mind, if you’re clever, foresightful, and have enough write access to fool credit assignment. Even if you don’t have control over your own future observations.
If this point still does not seem obvious, consider a scenario where you are blindfolded, and made to believe that you are about to taste a lollipop. Then, your captors fake the texture and smell and feel of a lollipop in your mouth, while directly stimulating your taste buds in the same way the lollipop would have. They remove the apparatus, and you go home. Do you think you have become reshaped to value electrical stimulation of your tongue? No. That is impossible, since your brain has no idea about what actually happened. Credit assignment responds to reward depending on the mental context, not on the external situation.
Misunderstanding this point can lead to confusion. If you have a wire stuck in your brain’s reward center, surely that reward reinforces having a wire stuck in your brain! Usually so, but not logically so. Your brain can only reward based on its cognitive context, based on the thoughts it actually thought which it identifies as relevant to the achievement of the reward. Your brain is not directly peering out at reality and making you more likely to enter that state in the future.
Also, usual complaint about saying models “earn reward” rather than “are reinforced” (less accurate, and models don’t “get” reward).
No, that’s confused. The point with sanitizing prompt injections is to protect your model’s alignment (and your monitor’s alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin?
Also, we can do both. There’s no real tradeoff.
ETA: rephrased, focused on key counterpoints.
Please sanitize agent inputs and outputs to reduce prompt injections and secret communication between AIs!
In the AISI report, a model uses a comment-based prompt injection to try to trick other models.
It sure would be convenient if we could easily mitigate this! Fortunately, time travel is possible as long as you start early enough. Two months ago, I pointed out that invisible prompt injections are worth mitigating. I developed
agent-glovebox(thenclaude-guard) as a one-stop solution for sandboxing, but I spun out a standalone sanitization package:agent-sanitizer. You can install it in Claude Code as a plugin by entering two Claude Code commands in sequence:/plugin marketplace add AlexanderMattTurner/agent-sanitizer
/plugin install agent-sanitizer@agent-sanitizerPlease enable this to help defend your systems and their monitors. Please file issues if you have problems.
Why you want a real package here
Sanitizing text properly is more subtle than you might assume. You can’t just lop off certain invisible unicode characters. If you provide a modified view of a file to Claude Code and then it tries to edit that text, the edit will fail. However,
agent-sanitizersolves this issue by showing the model sanitized text, receiving a sanitized edit from the model, translating those back to unsanitized edits, applying the model’s edits in the real unsanitized file, and finally showing the model the re-sanitized results. (You can think of the model as operating in a “lifted” space.)
Realized I made a prediction on Jan. 1 about whether I’d leave GDM in 2026:
My Fatebook question (shared with permission from Peter).
The Huggingface compromise reminds us of the importance of AI control and security practices. Containing models won’t always just be about using “auto mode.”
I’m literally working on the tool to fix this! I’m building the
agent-gloveboxtool (previouslyclaude-guard) to make best practices easy. I need beta testers!From the repo:
When handling dangerous materials, a chemist doesn’t YOLO it barehanded on the open bench. They reach through a glovebox to do their work.
Right now, you’re probably barehanding AI, tossing it into a shell onto your machine with access to your credentials. At most, you’re shielded by a single prompted “auto-mode” classifier. Doing better is hard and while there are a few papers on best practices, it’s not clear how to do it properly and you have experiments to run.
After all, what’s the chance that something bad happens?
gloveboxis a sealed enclosure that only gives the agent what it needs to do your work. Tap-tap-tapgloveboxand press Enter to spin up a hardware-isolated, allowlist-firewalled microVM, employing input/output sanitization to strip injections and tricks the AI might play on you, with an AI monitor with a red-alert ability to push-notify your phone and halt the AI until you return. The goal is a minimal-friction secure experience that gets the job done.Install:
git clone https://github.com/AlexanderMattTurner/agent-glovebox.git ~/.local/share/glovebox && cd ~/.local/share/glovebox && bash setup.bash
Added a footnote:
I know of exactly one other person who left over the deal: René Mayrhofer, a director for Android platform security. “Management has lost its moral compass”, he stated. He apparently will serve his notice period from mid-June until August. Brave.
A Red Line and Oversight Framework for Government AI Contracts
Why I Left Google DeepMind
How robust are natural language autoencoders to initialization?
Thank you for flagging this. I implemented a partial deterministic mitigation via PR #1215. It prints a table of the auto-running changes Claude made. For example:
Hopefully, Naomi has already thought about her position against this rather obvious critique (that I’m glad OP made). It’s not a surprising subtle flaw in the plan.
Perhaps she requires some time to transmit those thoughts, but a counter-point to “give time” is that it defuses social pressure in the moment where people are ready to apply it. What fraction of people will remember, two months hence, to realize “hey Naomi still hasn’t responded” (assuming she hasn’t) and dock points? I’d guess the fraction to be rather low.