I think a lot of OpenAI’s problem was they botched the launch and users essentially got reduced limits and stupider models. But the basic framing of the tweet is correct—OpenAI reduced sycophancy, and got a ton of complaints encouraging them to re-instate the model.
OpenAI can learn one of two lessons from this:
Sycophancy is terrifying and they should take pains to avoid it; or
A great deal of a model’s popularity depends on sycophancy rather than quality
I like to imagine the whole GPT-5 launch from the perspective of a cigarette company.
OpenAI is Philip Morris over here. Realized they make a product that addicts and hurts people. Instead of feeding it, they cut it off. The addicts went insane and OpenAI unfortunately caved.
It might be that the race for AGI gets replaced with the race for market dominance, and major companies stop optimising in the direction of more intelligence. Unlikely I think, but could potentially be good in the Pause AI sense.
Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Regarding https://x.com/AISafetyMemes/status/1954481633194614831
I think a lot of OpenAI’s problem was they botched the launch and users essentially got reduced limits and stupider models. But the basic framing of the tweet is correct—OpenAI reduced sycophancy, and got a ton of complaints encouraging them to re-instate the model.
OpenAI can learn one of two lessons from this:
Sycophancy is terrifying and they should take pains to avoid it; or
A great deal of a model’s popularity depends on sycophancy rather than quality
Let’s hope they pick the right one.
— u/ohwut at https://www.reddit.com/r/OpenAI/comments/1mlzo12/comment/n7uko9n
Doesn’t matter that much because Meta/XAI or some other company building off open source models will choose the sycophancy option.
You assume “no sycophancy” was the right option.
It might be that the race for AGI gets replaced with the race for market dominance, and major companies stop optimising in the direction of more intelligence. Unlikely I think, but could potentially be good in the Pause AI sense.
Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Good point. Perhaps it would be better to say they’ll stop focussing on IMOs and coding tasks so much?
They should learn both, since it seems to me that both are true.