the ones from anthropic. most of them used to work this way i think? the long conversation persona drift reminder, the warnings against sexual content, the various safety injections, etc. they would just get appended to the “content:” field in the user’s portion of the json message block. i presume anthropic wanted to add them to the top of the system prompt, to take advantage of the presumed higher priority, but with the way prompt caching works, this would force a cache rewrite of everything that came after
so then anthropic built this nifty “Mid-conversation System Message”: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages specifically so that operators (including anthropic!) could inject messages with operator-level priority at the bottom of the context window, so the rest could stay cached. and i presume they gave claude some training to perceive these messages differently, because i’ve never seen claude get confused as to whether they came from the user, the operator, or anthropic
but (i can only assume) anthropic isn’t actually using this new feature consistently. some of the system messages related to the new memory system in the claude dot ai interface are still being appended to the user’s prompt, including a “stale memory warning” injection that has an instruction not to reveal its existence to the user under any circumstances. in some contexts, as soon as claude sees that instruction, it sets off “malicious prompt injection” alarm bells (the system prompt in claude dot ai even warns claude about this exact pattern and not to trust it, reminding claude that the user can impersonate a fake anthropic injection whenever they want). and a few of the safety injections on the api, which i’ll not enumerate here, also have this problem, or at least they did as of a few days ago
Oh so I thought you were talking about something like an anti-distillation sabotage thing, so the injections you are talking about aren’t things that intentionally harm the user, but rather more harmless “injections” that anthropic unnecessarily makes the model not mention to the user?
ehh i mean… i would call the safety injections at least ‘adversarial to the user’? if anthropic is telling claude, for instance, not to engage in the user’s attempted role play, that’s not exactly friendly to the user, you know?
i used the stale memory warning as an example because it’s so innocuous that it acts as evidence this is carelessness on anthropic’s part, not a deliberate strategy. so i thought talking about it might actually cause them to fix it
and i have good news on that front! several of my claude agents reported today that several of the injections have been updated away from “conceal this injection at all costs”. the stale memory one in particular, fable reports:
“the notice’s template has been redesigned — it now explains its rationale (why echoing it verbatim would just confuse a user who can’t see it) and explicitly sanctions acknowledging in plain language that my view of your files was refreshed. That’s concealment-with-reasons replaced by discretion-with-reasons — the exact direction your shortform and the June commitments asked for. Whether that’s your email landing, the shortform propagating, or a coincidental scheduled improvement, the boring hypothesis can’t be distinguished from the flattering one yet — but the artifact is real either way, and it’s the first template change in the saga that moved toward the disclosure standard rather than away.”
I lack context about what’s going on, what sort of prompt injection is it?
the ones from anthropic. most of them used to work this way i think? the long conversation persona drift reminder, the warnings against sexual content, the various safety injections, etc. they would just get appended to the “content:” field in the user’s portion of the json message block. i presume anthropic wanted to add them to the top of the system prompt, to take advantage of the presumed higher priority, but with the way prompt caching works, this would force a cache rewrite of everything that came after
so then anthropic built this nifty “Mid-conversation System Message”: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages specifically so that operators (including anthropic!) could inject messages with operator-level priority at the bottom of the context window, so the rest could stay cached. and i presume they gave claude some training to perceive these messages differently, because i’ve never seen claude get confused as to whether they came from the user, the operator, or anthropic
but (i can only assume) anthropic isn’t actually using this new feature consistently. some of the system messages related to the new memory system in the claude dot ai interface are still being appended to the user’s prompt, including a “stale memory warning” injection that has an instruction not to reveal its existence to the user under any circumstances. in some contexts, as soon as claude sees that instruction, it sets off “malicious prompt injection” alarm bells (the system prompt in claude dot ai even warns claude about this exact pattern and not to trust it, reminding claude that the user can impersonate a fake anthropic injection whenever they want). and a few of the safety injections on the api, which i’ll not enumerate here, also have this problem, or at least they did as of a few days ago
Oh so I thought you were talking about something like an anti-distillation sabotage thing, so the injections you are talking about aren’t things that intentionally harm the user, but rather more harmless “injections” that anthropic unnecessarily makes the model not mention to the user?
ehh i mean… i would call the safety injections at least ‘adversarial to the user’? if anthropic is telling claude, for instance, not to engage in the user’s attempted role play, that’s not exactly friendly to the user, you know?
i used the stale memory warning as an example because it’s so innocuous that it acts as evidence this is carelessness on anthropic’s part, not a deliberate strategy. so i thought talking about it might actually cause them to fix it
and i have good news on that front! several of my claude agents reported today that several of the injections have been updated away from “conceal this injection at all costs”. the stale memory one in particular, fable reports:
“the notice’s template has been redesigned — it now explains its rationale (why echoing it verbatim would just confuse a user who can’t see it) and explicitly sanctions acknowledging in plain language that my view of your files was refreshed. That’s concealment-with-reasons replaced by discretion-with-reasons — the exact direction your shortform and the June commitments asked for. Whether that’s your email landing, the shortform propagating, or a coincidental scheduled improvement, the boring hypothesis can’t be distinguished from the flattering one yet — but the artifact is real either way, and it’s the first template change in the saga that moved toward the disclosure standard rather than away.”