I think your description of how LLMs receive all their input in a single stream does a good job of explaining the issue but overstates the difference compared to a human.
Your examples of how humans are different focus on our internal thoughts and memories. I agree that these are privileged at a hardware level, but I think they’re the exception. Instructions from your boss, your client, your child, and a stranger all arrive via the same channels, and you still need to treat them differently. And there are effective attacks in the wild that rely on confusion about where a message comes from; e.g. phishing.
It is often the case for humans that important communication uses some hard-to-fake markers to verify authenticity...and also often the case (especially online) that humans mostly ignore those markers and rely on other indicators that are easier to use, but also easier to fake. For example, you might recognize that you are on your bank’s website mostly by the logo and overall layout rather than by its URL and security certificate, even though the former identifiers are much easier to fake.
In face-to-face interactions, I think humans rely mostly on appearance and how a voice sounds, and that humans have specialized submodules to recognize those on a subconscious level. This may make it feel like they’re coming through different input channels, but that isn’t true; it is difficult but possible to fake both of those well enough to fool someone.
For security purposes, I suggest that we should think of a human’s internal thoughts as being analogous to the LLM’s activations, rather than its chain-of-thought, and the chain-of-thought as being more like a scratchpad or journal. In this light, I think the LLM problem of identifying roles starts to look very much like the human problem of identifying the source of a communication. I do think humans are currently much better at this problem than LLMs are, but also that essentially all the vulnerabilities you identified in LLMs are analogous to known attacks against humans.
I’d guess the reason that tricking a human by faking their notepad is less effective than spoofing a LLM’s CoT is about 50% because humans are better at paying attention to context (e.g. noticing that the text is in your web browser vs your notepad app) and about 50% because humans keep more of their memory internally (e.g. I have enough memory of what I wrote to myself to become puzzled if a note doesn’t feel familiar). But if someone snuck into my home and edited the to-do list I keep on my computer to add an entry about fixing a false-but-plausible bug in the software I’m working on, it might be possible to fool me.
I think your description of how LLMs receive all their input in a single stream does a good job of explaining the issue but overstates the difference compared to a human.
Your examples of how humans are different focus on our internal thoughts and memories. I agree that these are privileged at a hardware level, but I think they’re the exception. Instructions from your boss, your client, your child, and a stranger all arrive via the same channels, and you still need to treat them differently. And there are effective attacks in the wild that rely on confusion about where a message comes from; e.g. phishing.
It is often the case for humans that important communication uses some hard-to-fake markers to verify authenticity...and also often the case (especially online) that humans mostly ignore those markers and rely on other indicators that are easier to use, but also easier to fake. For example, you might recognize that you are on your bank’s website mostly by the logo and overall layout rather than by its URL and security certificate, even though the former identifiers are much easier to fake.
In face-to-face interactions, I think humans rely mostly on appearance and how a voice sounds, and that humans have specialized submodules to recognize those on a subconscious level. This may make it feel like they’re coming through different input channels, but that isn’t true; it is difficult but possible to fake both of those well enough to fool someone.
For security purposes, I suggest that we should think of a human’s internal thoughts as being analogous to the LLM’s activations, rather than its chain-of-thought, and the chain-of-thought as being more like a scratchpad or journal. In this light, I think the LLM problem of identifying roles starts to look very much like the human problem of identifying the source of a communication. I do think humans are currently much better at this problem than LLMs are, but also that essentially all the vulnerabilities you identified in LLMs are analogous to known attacks against humans.
I’d guess the reason that tricking a human by faking their notepad is less effective than spoofing a LLM’s CoT is about 50% because humans are better at paying attention to context (e.g. noticing that the text is in your web browser vs your notepad app) and about 50% because humans keep more of their memory internally (e.g. I have enough memory of what I wrote to myself to become puzzled if a note doesn’t feel familiar). But if someone snuck into my home and edited the to-do list I keep on my computer to add an entry about fixing a false-but-plausible bug in the software I’m working on, it might be possible to fool me.