My current understanding is that a good way to prevent the sloppification and sycophantic attractors are to not talk directly with the base assistant persona but instead have two higher quality personas talk about your prompt. I’m hoping to see more experimentation with this so I don’t have to build the whole thing myself (ideal version is separate API calls with separate rags and separate parameters).
Yep, that can definitely be helpful, but my prediction is that even if this became the default option in Claude or OpenAI, I think we’d likely see some new and unique failure modes, or just new permutations of the same sorts of failures, where “failure” here means something like “human ends up replacing their default sensemaking apparatus with whatever the LLMs tell them.” Maybe less, though!
My current understanding is that a good way to prevent the sloppification and sycophantic attractors are to not talk directly with the base assistant persona but instead have two higher quality personas talk about your prompt. I’m hoping to see more experimentation with this so I don’t have to build the whole thing myself (ideal version is separate API calls with separate rags and separate parameters).
Yep, that can definitely be helpful, but my prediction is that even if this became the default option in Claude or OpenAI, I think we’d likely see some new and unique failure modes, or just new permutations of the same sorts of failures, where “failure” here means something like “human ends up replacing their default sensemaking apparatus with whatever the LLMs tell them.” Maybe less, though!