Yes, we were excited when we learned about ARC Evals. Some kind of evaluation was one of our possible paths to impact, though real-world data is much more messy than the carefully constructed evaluations I’ve seen ARC use. This has both advantages and disadvantages.
So this project was something along the line of ARC Evals?
Yes, we were excited when we learned about ARC Evals. Some kind of evaluation was one of our possible paths to impact, though real-world data is much more messy than the carefully constructed evaluations I’ve seen ARC use. This has both advantages and disadvantages.