Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Good point. Perhaps it would be better to say they’ll stop focussing on IMOs and coding tasks so much?