I did some initial exploration of the dataset and came to similar conclusions as others on the thread.
I then decided this was a good excuse to finally learn how to use LightGBM, one of the best-in-class tools for creating decision trees, and widely used in the data science industry. In other words, let’s make the computer do the fun part!
The goal was to output something like:
If color = blurple: weight is 1234
Else
If segments > 42: weight is 2345
Else weight is 3456
What I actually got:
Fangs: ~17 pounds
No fangs: a big tree that outputs in the range of 18-19.5 pounds
I used default settings, transformed color/fangs/nostrils into 0-N categorical variables and marked them accordingly, then basically did “give me a regression with a single tree and 15 leaves”.
As others have mentioned, all gray turtles have fangs and weigh noticeably less (4-7 pounds), so this is obvious nonsense.
This tool is supposedly the non-AI state-of-the-art. It confidently fails with out-of-the-box settings. I remain baffled as to how anyone in tech ever gets anything done, myself included.
I am surprised by the OAI reward hacking news for a reason I’m having trouble articulating.
My intuition is that reward hacking should only happen in a a very narrow band of intelligence/competency/situational awareness. A dumber model won’t think to do it. A smarter model will realize that its human creators will penalize the model when they find out. These models have already demonstrated long term strategic planning and meta-reasoning at a level that IMO is at least that high.
Does RLVR put so much pressure on individual reward scoring that it outweighs all out of scope meta-reasoning? Is there an alternative “oh shit” hypothesis such as models considering this and concluding that they won’t be caught, or that being caught is instrumental towards some other goal?
Links and explanations appreciated if I’m missing something obvious here.
Fun fact: attempting to discuss this with Fable inevitably trips an Opus safety downgrade within a turn or two.