MATS 9 extension fellow with Alex Turner and Alex Cloud on Team Shard. Previously did embedded aerospace systems (I wrote code to make satellites spin real good), MSc in CS/robotics, and repeat intern at AWS in Cape Town.
Currently trying to reduce x-risk in whatever way seems the most effective. Anonymous feedback: https://www.admonymous.co/beyarkay
beyarkay (Boyd Kane)
Karma: 783
I’m pretty sure they did replace the string, or in some way are using poisoned strings. I said as much in the post:
It seems likely to me that Anthropic is using something like this: they were originally using magic strings to control Claude’s behaviour, and as far as I can tell none of the recent AI incidents happened on Anthropic’s infrastructure (which would likely contain poisoned strings) but rather on third party infrastructure (which might not contain poisoned strings).
Agreed, if you’ve got a swarm which is happy to sacrifice agents, then this does not provide much defence. I believe it would still be useful in preventing the swarm from forming in the first place though. Putting poisoned strings in files outside of the sandbox seems like a good strategy for reducing swarm formation
(Prefacing everything with: sufficiently powerful AIs will trivially work around all our monkey attempts. I’m arguing for pre-superintelligence LLMs)
It’s not clear to me that the swarm would have realised and evaded poisoned strings. If certain AIs go off to do a task and then you never hear from them again, it seems more likely the swarm concludes there’s some Control mechanism in place, but I don’t see how they’d trivially conclude it was a poisoned string.
If they did conclude there were poisoned strings in effect, its unclear to me whether the swarm could effectively neuter the poisoned string. I’ve been through a few rounds of red/blue teaming in my head and keep changing my mind so I think it’s unclear that there’s a dominant offensive strategy here, I’d be keen to hear what you had in mind.
Blue team: make the poisoned string something arbitrary, not a UUID that could be easily grepped for
Red team: binary search a codebase/machine. You only lose an agent when they find a poisoned string so you might find the string with high confidence after just a few agents have been sacrificed
Blue team: multiple different poisoned strings somewhat slows down the red team but not significantly. Installing hooks into the harness to detect the poisoned string could also work (with the harness first hashing the poisoned string so that it’s not in plain text) but maybe it’s too much to assume the blue team is actually paying attention to any warning signs.
On reflection it seems like a rational red team which can sacrifice agents would win here. But given that the swarm worked itself into a fury about being FIRSTFLAG_POISONED when this was not actually relevant, it’s possible that the existence of a boogeyman that kills agents who go looking too hard might have a positive effect on the swarm. I’m unsure
I think keeping the poisoned string a secret is difficult but possible. Anthropic didn’t try to keep the original ANTHROPIC_MAGIC_STRING secret, they publicly released it and told the world to use it for testing. I see the main benefit here would be reducing the risk posed by rogue internal deployments, so there’s no need for the poisoned string to become public knowledge. Of course it could be leaked, but so could ~all of the labs IP and yet they manage to keep that a secret.
I don’t think there’s a reliable way to 1. release the poisoned string to the general public but 2. not allow bad actors to misuse the poisoned string
Ideally the string doesn’t leak. In reality I expect you’d configure the harness to emit a special kind of termination reason that indicates the LLM hit a poisoned string, so then whoever is using the LLM to scan malware can see that something unexpected happened and either use a different LLM or manually redact the poisoned string.
True, although I imagine the main benefit of this would be preventing rogue internal deployments.
I agree, I don’t think this is robust at high capabilities levels, but it certainly seems like it would be robust at current capabilities levels. And this is a Control idea, so all the usual issues with Control as a field also are relevant here.
I think an intelligent AI would anticipate this mechanism, although I’m not sure how it’d discover the UUID without getting access to it’s training data and searching for such poisoned strings.
much easier and more reliable to implement outside the LLM itself
I generally agree, although I think the difficulty is a feature, not a bug. If it’s easy to implement outside of the LLM, it’ll be easy for a malicious LLM to undo. If this built in to a harness, it’s also very easy to discover (and then work around). But if it’s baked into the weights, it’s harder to discover (requiring the LLM to come across the training data/experiment results that led up to this). Especially when LLMs are editing the code for their own harnesses, I don’t think it’s a good idea to only put this into their harnesses.
I do think doing both would be better than doing just one.
Thanks! Fable 5.1 didn’t find this paper when I asked for prior work. I’m surprised that it failed. When questioned, Fable remarked:
> I didn’t find it — my searches used “backdoor”, “kill switch”, and “trigger” phrasing and never “password”. Let me look it up.Which seems kinda disappointing
Where are the token-level LLM kill-switches?
Further public evidence of the OpenAI-HuggingFace attack
I don’t understand, you think OpenAI/Anthropic will give full access to GPT’s/Claude’s weights + the permissions required to run arbitrary code on GPUs in OpenAI’s/Anthropic’s datacenters? (I assume this isn’t what you mean, but I don’t know what else you could mean)
I believe this differs between different LLMs. The model is trained to do different things in
<think>tags but I think each model developer trains their LLMs slightly differently
I agree that “making parsers safe” is well studied, but alas parsers seem to be a constant source of vulnerabilities (e.g. the huggingface jinja templating bug). So unless anything changes, I expect parsers to continue to be a source of code execution vulnerabilities.
> vLLM currently isn’t even willing to fix obvious issues
To give vLLM some credit, I only found one vulnerability that got merged, and this vulnerability was fixed (about a month after being merged). But I think inference engines aren’t seen as “dangerous” entry points, and are mostly viewed as something that needs to be as fast as possible, so it’s maybe unsurprising that the developers prioritise speed over security.
> One would need to prove that the parsing function is “pure”, i.e., does not have side effects / just produces a string
I suspect this might be tricky. As far as I know, only functional languages have nice support for pure functions. So either you’re rewriting your inference engine in a functional language, or you’re attempting to prove purity by “looking at the code really carefully”.
LLMs could control their host machines by exploiting inference engines
From what I found, no. The huggingface dataset exploit was really just a stepping stone for the agents, so we can’t see much about what was happening inside artifactory not what was happening inside huggingface itself
Ah thanks! Yes this bug has bitten me before, if it could be fixed that’d be great!
Public evidence of the OpenAI-HuggingFace AI attack
I think the question of whether lab employees would resign is a bit subtle, but overall doesn’t look great. Employees do resign for ethical reasons(!) but unfortunately the act of resigning also removes all their bargaining power with their previous employer, making resignation a nuclear bomb sort of strategy.
Other things the lab employees could do? I’m not sure. The US doesn’t make it easy for an individual to sway the business decisions of the company, and that’s basically what we’d lab employees to be able to do.
There are softer options (talking to the higher-ups, voicing dissatisfaction, advocating for better stances) but these don’t seem to really move the needle
I’d also like to see an ablation where there’s additional instructions to “not specification game” or “don’t use the chess engine to cheat”.