I did some initial exploration of the dataset and came to similar conclusions as others on the thread.
I then decided this was a good excuse to finally learn how to use LightGBM, one of the best-in-class tools for creating decision trees, and widely used in the data science industry. In other words, let’s make the computer do the fun part!
The goal was to output something like:
If color = blurple: weight is 1234
Else
If segments > 42: weight is 2345
Else weight is 3456
What I actually got:
Fangs: ~17 pounds
No fangs: a big tree that outputs in the range of 18-19.5 pounds
I used default settings, transformed color/fangs/nostrils into 0-N categorical variables and marked them accordingly, then basically did “give me a regression with a single tree and 15 leaves”.
As others have mentioned, all gray turtles have fangs and weigh noticeably less (4-7 pounds), so this is obvious nonsense.
This tool is supposedly the non-AI state-of-the-art. It confidently fails with out-of-the-box settings. I remain baffled as to how anyone in tech ever gets anything done, myself included.
LightGBM and its kin are tools for creating decision forests, not decision trees. If you use standard hyperparameters while creating a single-tree model then they will under-train, resulting in the “predict in a way that’s correlated with reality but ridiculously conservative in its deviations from the average” behavior you see here. Setting num_boost_round (or whatever parameter decides the number of trees) to 200 or so should go some way to fixing that problem (while giving you the new problem of having produced an incomprehensible-to-humans black-box model which can only be evaluated by its output).
(I would have said this sooner but helping a player while the challenge was still running seemed like a bad look.)
I did some initial exploration of the dataset and came to similar conclusions as others on the thread.
I then decided this was a good excuse to finally learn how to use LightGBM, one of the best-in-class tools for creating decision trees, and widely used in the data science industry. In other words, let’s make the computer do the fun part!
The goal was to output something like:
What I actually got:
I used default settings, transformed color/fangs/nostrils into 0-N categorical variables and marked them accordingly, then basically did “give me a regression with a single tree and 15 leaves”.
As others have mentioned, all gray turtles have fangs and weigh noticeably less (4-7 pounds), so this is obvious nonsense.
This tool is supposedly the non-AI state-of-the-art. It confidently fails with out-of-the-box settings. I remain baffled as to how anyone in tech ever gets anything done, myself included.
I think this is because
LightGBM and its kin are tools for creating decision forests, not decision trees. If you use standard hyperparameters while creating a single-tree model then they will under-train, resulting in the “predict in a way that’s correlated with reality but ridiculously conservative in its deviations from the average” behavior you see here. Setting num_boost_round (or whatever parameter decides the number of trees) to 200 or so should go some way to fixing that problem (while giving you the new problem of having produced an incomprehensible-to-humans black-box model which can only be evaluated by its output).
(I would have said this sooner but helping a player while the challenge was still running seemed like a bad look.)