I have also thought that it was scored really unfairly, and I think it’s very impressive that there is really remarkable progress despite that.
Yet we should not mix the internal track (no internet connection, so no large LLMs can be used directly) and the external track (where Astra has officially scored 99.95% in a good harness).
The external track is a benchmark for LLM abilities, but the internal track, where serious LLMs are not available directly during test runs, is a benchmark for the ability to do AI research.
I have also thought that it was scored really unfairly, and I think it’s very impressive that there is really remarkable progress despite that.
Yet we should not mix the internal track (no internet connection, so no large LLMs can be used directly) and the external track (where Astra has officially scored 99.95% in a good harness).
The external track is a benchmark for LLM abilities, but the internal track, where serious LLMs are not available directly during test runs, is a benchmark for the ability to do AI research.