Currently looking at some of the example problems used to assess AI capabilities in this paper. This… obviously doesn’t work as intended? Based only on recall (no search), Deepseek answers these questions somewhat well, getting 2⁄3 attempts for the first, 0⁄3 for the second, [assumes Star Wars refers to Episode I and gives the answer for that movie, changes to correct answer after clarification, so either 0⁄3 or 3⁄3]. It identifies Satan, but misidentifies the squished animal as a cat for #4 in all 3 tests. So a model with zero visual ability could score quite well just from the context given by the question and maybe the metadata of the upload file. The first question doesn’t even require the name of the movie for a correct answer!
This seems to indicate fundemental problems with their test construction—it gives “written by someone from 1990 who makes an AGI test without ever interacting with an LLM” vibes. Maybe that was the intent?
They also cited actual College Board example AP tests from ~2014 as their sources for like 30% of their problems. I would be shocked they weren’t in the training data multiple times.
A Definition of AGI
Currently looking at some of the example problems used to assess AI capabilities in this paper. This… obviously doesn’t work as intended? Based only on recall (no search), Deepseek answers these questions somewhat well, getting 2⁄3 attempts for the first, 0⁄3 for the second, [assumes Star Wars refers to Episode I and gives the answer for that movie, changes to correct answer after clarification, so either 0⁄3 or 3⁄3]. It identifies Satan, but misidentifies the squished animal as a cat for #4 in all 3 tests. So a model with zero visual ability could score quite well just from the context given by the question and maybe the metadata of the upload file. The first question doesn’t even require the name of the movie for a correct answer!
This seems to indicate fundemental problems with their test construction—it gives “written by someone from 1990 who makes an AGI test without ever interacting with an LLM” vibes. Maybe that was the intent?
They also cited actual College Board example AP tests from ~2014 as their sources for like 30% of their problems. I would be shocked they weren’t in the training data multiple times.
This whole paper just feels wrong.