MATS 9 extension fellow with Alex Turner and Alex Cloud on Team Shard. Previously did embedded aerospace systems (I wrote code to make satellites spin real good), MSc in CS/robotics, and repeat intern at AWS in Cape Town.
Currently trying to reduce x-risk in whatever way seems the most effective. Anonymous feedback: https://www.admonymous.co/beyarkay
beyarkay (Boyd Kane)
Karma: 791
Really good read, thanks for writing! Some things that stood out/I was surprised by:
It seems like LASR has many short weekly meetings with your RM? I’m curious about the differences between this and one 1hr meeting per week
Very interesting about how LASR accepts before doing project selection! You recommend avoiding groupthink when discussing project selection with everyone else. I guess doing project selection online avoids this issue, but not doing project selection online cuts down the application process (as you said). Idk which is better/worse, but it’s good to hear about different ways of doing things!
LASR splitting things up into trimesters with required deliverables seems really good IMO. Idk if this is an issue for LASR, but because there were 60+ MATS fellows I only realised who was doing what right at the end of the program and it would have been great to have some method of discovering interesting people (e.g. via required public deliverables)
I like how you framed things explicitly as “this is a strong reason to apply for LASR”
Strongly agree with communicating more often than you think is required, it’s very easy to drift or lose track of the goal
I’d also like to see an ablation where there’s additional instructions to “not specification game” or “don’t use the chess engine to cheat”.
I’m pretty sure they did replace the string, or in some way are using poisoned strings. I said as much in the post:
It seems likely to me that Anthropic is using something like this: they were originally using magic strings to control Claude’s behaviour, and as far as I can tell none of the recent AI incidents happened on Anthropic’s infrastructure (which would likely contain poisoned strings) but rather on third party infrastructure (which might not contain poisoned strings).
Agreed, if you’ve got a swarm which is happy to sacrifice agents, then this does not provide much defence. I believe it would still be useful in preventing the swarm from forming in the first place though. Putting poisoned strings in files outside of the sandbox seems like a good strategy for reducing swarm formation
(Prefacing everything with: sufficiently powerful AIs will trivially work around all our monkey attempts. I’m arguing for pre-superintelligence LLMs)
It’s not clear to me that the swarm would have realised and evaded poisoned strings. If certain AIs go off to do a task and then you never hear from them again, it seems more likely the swarm concludes there’s some Control mechanism in place, but I don’t see how they’d trivially conclude it was a poisoned string.
If they did conclude there were poisoned strings in effect, its unclear to me whether the swarm could effectively neuter the poisoned string. I’ve been through a few rounds of red/blue teaming in my head and keep changing my mind so I think it’s unclear that there’s a dominant offensive strategy here, I’d be keen to hear what you had in mind.
Blue team: make the poisoned string something arbitrary, not a UUID that could be easily grepped for
Red team: binary search a codebase/machine. You only lose an agent when they find a poisoned string so you might find the string with high confidence after just a few agents have been sacrificed
Blue team: multiple different poisoned strings somewhat slows down the red team but not significantly. Installing hooks into the harness to detect the poisoned string could also work (with the harness first hashing the poisoned string so that it’s not in plain text) but maybe it’s too much to assume the blue team is actually paying attention to any warning signs.
On reflection it seems like a rational red team which can sacrifice agents would win here. But given that the swarm worked itself into a fury about being FIRSTFLAG_POISONED when this was not actually relevant, it’s possible that the existence of a boogeyman that kills agents who go looking too hard might have a positive effect on the swarm. I’m unsure
I think keeping the poisoned string a secret is difficult but possible. Anthropic didn’t try to keep the original ANTHROPIC_MAGIC_STRING secret, they publicly released it and told the world to use it for testing. I see the main benefit here would be reducing the risk posed by rogue internal deployments, so there’s no need for the poisoned string to become public knowledge. Of course it could be leaked, but so could ~all of the labs IP and yet they manage to keep that a secret.
I don’t think there’s a reliable way to 1. release the poisoned string to the general public but 2. not allow bad actors to misuse the poisoned string
Ideally the string doesn’t leak. In reality I expect you’d configure the harness to emit a special kind of termination reason that indicates the LLM hit a poisoned string, so then whoever is using the LLM to scan malware can see that something unexpected happened and either use a different LLM or manually redact the poisoned string.
True, although I imagine the main benefit of this would be preventing rogue internal deployments.
I agree, I don’t think this is robust at high capabilities levels, but it certainly seems like it would be robust at current capabilities levels. And this is a Control idea, so all the usual issues with Control as a field also are relevant here.
I think an intelligent AI would anticipate this mechanism, although I’m not sure how it’d discover the UUID without getting access to it’s training data and searching for such poisoned strings.
much easier and more reliable to implement outside the LLM itself
I generally agree, although I think the difficulty is a feature, not a bug. If it’s easy to implement outside of the LLM, it’ll be easy for a malicious LLM to undo. If this built in to a harness, it’s also very easy to discover (and then work around). But if it’s baked into the weights, it’s harder to discover (requiring the LLM to come across the training data/experiment results that led up to this). Especially when LLMs are editing the code for their own harnesses, I don’t think it’s a good idea to only put this into their harnesses.
I do think doing both would be better than doing just one.
Thanks! Fable 5.1 didn’t find this paper when I asked for prior work. I’m surprised that it failed. When questioned, Fable remarked:
> I didn’t find it — my searches used “backdoor”, “kill switch”, and “trigger” phrasing and never “password”. Let me look it up.Which seems kinda disappointing
I don’t understand, you think OpenAI/Anthropic will give full access to GPT’s/Claude’s weights + the permissions required to run arbitrary code on GPUs in OpenAI’s/Anthropic’s datacenters? (I assume this isn’t what you mean, but I don’t know what else you could mean)
I believe this differs between different LLMs. The model is trained to do different things in
<think>tags but I think each model developer trains their LLMs slightly differently
I agree that “making parsers safe” is well studied, but alas parsers seem to be a constant source of vulnerabilities (e.g. the huggingface jinja templating bug). So unless anything changes, I expect parsers to continue to be a source of code execution vulnerabilities.
> vLLM currently isn’t even willing to fix obvious issues
To give vLLM some credit, I only found one vulnerability that got merged, and this vulnerability was fixed (about a month after being merged). But I think inference engines aren’t seen as “dangerous” entry points, and are mostly viewed as something that needs to be as fast as possible, so it’s maybe unsurprising that the developers prioritise speed over security.
> One would need to prove that the parsing function is “pure”, i.e., does not have side effects / just produces a string
I suspect this might be tricky. As far as I know, only functional languages have nice support for pure functions. So either you’re rewriting your inference engine in a functional language, or you’re attempting to prove purity by “looking at the code really carefully”.
From what I found, no. The huggingface dataset exploit was really just a stepping stone for the agents, so we can’t see much about what was happening inside artifactory not what was happening inside huggingface itself
Ah thanks! Yes this bug has bitten me before, if it could be fixed that’d be great!
I think the question of whether lab employees would resign is a bit subtle, but overall doesn’t look great. Employees do resign for ethical reasons(!) but unfortunately the act of resigning also removes all their bargaining power with their previous employer, making resignation a nuclear bomb sort of strategy.
Other things the lab employees could do? I’m not sure. The US doesn’t make it easy for an individual to sway the business decisions of the company, and that’s basically what we’d lab employees to be able to do.
There are softer options (talking to the higher-ups, voicing dissatisfaction, advocating for better stances) but these don’t seem to really move the needle
This is a better-argued version of something I feel like I’ve been circling for a while, thanks for writing it.
Something you didn’t suggest but I think might be a pitfall to avoid: I don’t think you can hill-climb on articulacy by getting (for example) Fable to explain things to Haiku. The ways in which a weak model misunderstands are (I claim) sufficiently different from the ways in which a low-context human misunderstands, that I don’t think weak LLMs are a good proxy for low-context humans
Another use of non-spoke communication
In the Math Corps, we use a lot of hand signals. For example, “silent applause” is two hands up, palms forward, wiggling fingers; doesn’t interrupt the flow of the conversation, but allows for a visual cue of celebration and agreement. The most powerful one is “support”: rolling fists. Imagine: a kid is all alone at the blackboard in front of other kids, high school TAs, college instructors, struggling to solve a problem, worried what other people are thinking. The TAs are rolling their fists, so the other kids know to do it; it becomes a social norm to use hand signals. The kid at the board looks back at the team room, and sees 20 kids silently rolling their fists, saying, “you’ve got this! I’m with you, you can do it!” There’s nothing like the sight of the first time a kid experiences that, and sees themselves in a place where *everybody* is truly rooting for them.
I’m curious, how much is this alleviated by using open weight LLMs that have worse capabilities but little to no safeguards?