Scoring is budget-adjusted final-layer MSE: adjusted_final_layer_score = final_layer_mse × max(0.1, effective_compute / flop_budget). If the budget is exceeded (analytical FLOPs or effective_compute = F_m + λ·R_m), predictions are zeroed and the multiplier is forced to 1.0 (no compute discount).
But in scoring-model.md you say
Your score is pure MSE — the closer to zero, the better.
...
primary_score is what the leaderboard ranks on.
These are quite different—e.g. the former means I should care about generally reducing FLOP count while the latter implies I should only care about staying below the 6.8e10 FLOP limit
Your score is the budget-adjusted MSE. You can see how it looks on the leaderboard, where the rank is by “adjusted score” which is 0.1-1x the “final layer MSE.”
scoring-model.md is wrong or at least confusing, I’ll pass on the feedback. (I think maybe that’s because that document is just describing the “scoring model,” which is the input into the leaderboard score, but regardless it’s a misleading way to describe what’s going on.)
Regarding the leaderboard, also not quite clear on what the effective difference between Warm-up and Phase 1 is at this point. June 18th is slated as “leaderboard opens” but unless it’s materially different from the existing leaderboard I’m not sure what Phase 1 actually signifies.
I think the only difference between the phases is how locked in the rules are. During the warm-up round we may make big changes while we uncover bugs. In phase 1 we may make smaller changes. In phase 2 we won’t make changes so that people know how they are being graded.
(AICrowd has run a lot of competitions and ARC has run none so we just handed it off to them and letting them make these kinds of decisions.)
FYI I found the grading criteria confusing:
In
problem-setup.mdyou say:But in
scoring-model.mdyou sayThese are quite different—e.g. the former means I should care about generally reducing FLOP count while the latter implies I should only care about staying below the 6.8e10 FLOP limit
Your score is the budget-adjusted MSE. You can see how it looks on the leaderboard, where the rank is by “adjusted score” which is 0.1-1x the “final layer MSE.”
scoring-model.md is wrong or at least confusing, I’ll pass on the feedback. (I think maybe that’s because that document is just describing the “scoring model,” which is the input into the leaderboard score, but regardless it’s a misleading way to describe what’s going on.)
Regarding the leaderboard, also not quite clear on what the effective difference between Warm-up and Phase 1 is at this point. June 18th is slated as “leaderboard opens” but unless it’s materially different from the existing leaderboard I’m not sure what Phase 1 actually signifies.
I think the only difference between the phases is how locked in the rules are. During the warm-up round we may make big changes while we uncover bugs. In phase 1 we may make smaller changes. In phase 2 we won’t make changes so that people know how they are being graded.
(AICrowd has run a lot of competitions and ARC has run none so we just handed it off to them and letting them make these kinds of decisions.)