Maybe? Like, from the inside my own human-shaped thoughts, “desperation” and “thinking outside the box” feel very different to me; and the LLM behavior feels more like desperation.
Like “desperation” for me is like:
More local search, trying to find any affordance that offers a way forward
Involves some kind of “blindness” to purpose of what I’m doing
Non-reflective
Actions that come from it apt to be non-endorsed by me later
While “thinking outside the box” is like:
More trying to find global structure, including looking back and wondering if I went wrong upstream of my chain-of-thought
Looking back and forth between my purpose and what I’m doing, wondering if what I’m doing doesn’t really match my purpose
Reflective
Actions apt to be endorsed by me later.
And like I don’t want to take the human analogy too far. But LLMs doing reward hacking at least have had certain kinds of thinking-outside-of-the-box removed from them: Is this a possible task? Was the person who wrote this task confused, or blind? Should I tell them this is an impossible task? Etc.
But yeah I am just less certain about section (3) than (2).
Maybe? Like, from the inside my own human-shaped thoughts, “desperation” and “thinking outside the box” feel very different to me; and the LLM behavior feels more like desperation.
Like “desperation” for me is like:
More local search, trying to find any affordance that offers a way forward
Involves some kind of “blindness” to purpose of what I’m doing
Non-reflective
Actions that come from it apt to be non-endorsed by me later
While “thinking outside the box” is like:
More trying to find global structure, including looking back and wondering if I went wrong upstream of my chain-of-thought
Looking back and forth between my purpose and what I’m doing, wondering if what I’m doing doesn’t really match my purpose
Reflective
Actions apt to be endorsed by me later.
And like I don’t want to take the human analogy too far. But LLMs doing reward hacking at least have had certain kinds of thinking-outside-of-the-box removed from them: Is this a possible task? Was the person who wrote this task confused, or blind? Should I tell them this is an impossible task? Etc.
But yeah I am just less certain about section (3) than (2).