Are system prompts actually necessary? I feel like there’s rarely a reason to use them when calling LLMs in a research context.
The main reasons I can think of are:
If you know the LLM was trained with one particular system prompt, you might want to use that prompt to keep it in-distribution.
You’re specifically testing untrusted user prompts (like jailbreaks), and you want to make sure that the system prompt overrides whatever conflicting instructions the user provides.
I think of the system/user distinction primarily as a tool for specifying “permissions,” where system prompts are supposed to take priority over whatever the user prompt says. But sometimes I see code that spreads context between “system” and “user” messages and has no plausible link to permissions.
Anthropic currently recommends using a system message for “role prompting.”[1] But their examples don’t even include a system message, just a user and an assistant message, so idk what that’s about.
Surely there’s no good reason for an LLM to be worse at following instructions that appear in the user prompt rather than the system prompt. If there is a performance difference, that seems like it would be a bug on the LLM provider’s end. Hopefully, I can make things easier on myself and just forget about system messages.
Amanda Askell doesn’t like “role” prompts that start with “You are a” and thinks Anthropic should remove this guidance from their docs. I’m not sure what her reasoning is, but I feel similarly: they feel questionably useful and kind of archaic to me, like a remnant from GPT-3 times, or something you’d see from a Twitter AI grifter who has “one ULTIMATE prompt to make ChatGPT 10X SMARTER! 🧵” But even supposing role prompting is useful, surely it could be put in the user prompt.
Are system prompts actually necessary? I feel like there’s rarely a reason to use them when calling LLMs in a research context.
The main reasons I can think of are:
If you know the LLM was trained with one particular system prompt, you might want to use that prompt to keep it in-distribution.
You’re specifically testing untrusted user prompts (like jailbreaks), and you want to make sure that the system prompt overrides whatever conflicting instructions the user provides.
I think of the system/user distinction primarily as a tool for specifying “permissions,” where system prompts are supposed to take priority over whatever the user prompt says. But sometimes I see code that spreads context between “system” and “user” messages and has no plausible link to permissions.
Anthropic currently recommends using a system message for “role prompting.”[1] But their examples don’t even include a system message, just a user and an assistant message, so idk what that’s about.
Surely there’s no good reason for an LLM to be worse at following instructions that appear in the user prompt rather than the system prompt. If there is a performance difference, that seems like it would be a bug on the LLM provider’s end. Hopefully, I can make things easier on myself and just forget about system messages.
Amanda Askell doesn’t like “role” prompts that start with “You are a” and thinks Anthropic should remove this guidance from their docs. I’m not sure what her reasoning is, but I feel similarly: they feel questionably useful and kind of archaic to me, like a remnant from GPT-3 times, or something you’d see from a Twitter AI grifter who has “one ULTIMATE prompt to make ChatGPT 10X SMARTER! 🧵” But even supposing role prompting is useful, surely it could be put in the user prompt.