It seems to me that “vibe checks” for how smart a model feels are easily gameable by making it have a better personality.
My guess is that it’s most of the reason Sonnet 3.5.1 was so beloved. Its personality was made much more appealing, compared to e. g. OpenAI’s corporate drones.
The recent upgrade to GPT-4o seems to confirm this. They seem to have merely given it a better personality, and people were reporting that it “feels much smarter”.
Deep Research was this for me, at first. Some of its summaries were just pleasant to read, they felt so information-dense and intelligent! Not like typical AI slop at all! But then it turned out most of it was just AI slop underneath anyway, and now my slop-recognition function has adjusted and the effect is gone.
I agree with this section I think