When you’re a person interacting with a chat model directly, sycophancy and sophistry are a minor nuisance, or maybe even adaptive. When you’re a team trying to compose these models into larger systems (something necessary because of the aforementioned memory issue), wanting-to-look-good cascades into breaking problems.
If you replace “models” with “people”, this is true of human organizations too.
If you replace “models” with “people”, this is true of human organizations too.