I mean, I think so. In those papers it’s often not clear how “elicited” that key step was. The advantage of this example is that it very clearly claims the researchers made no contribution whatsoever, and the result still seems to settle a problem someone cares about! Only caveat is that it comes from OpenAI, who has a very strong incentive to drive the hype-cycle about their own models (but on the other hand, also has access to some of the best models which are not publicly available yet, which lends credibility).
If you follow maths, one can be reasonably confident that the models can now sometimes solve math problems that are “not too hard in retrospect”. I don’t know how substantial this particular problem was supposed to be, but it feels like it tracks?
By “not too hard in retrospect” I mean that the models are applying well-known techniques in settings where these are not obvious, but also not too surprising, e.g. where experts in that subfield will say things like “of course you’d do that” when examining the solution. (Of course, one should be careful with such self-reports, but I tend to find this believable)
Is this really that different from previous papers where a model provided a key step of the argument?
I mean, I think so. In those papers it’s often not clear how “elicited” that key step was. The advantage of this example is that it very clearly claims the researchers made no contribution whatsoever, and the result still seems to settle a problem someone cares about! Only caveat is that it comes from OpenAI, who has a very strong incentive to drive the hype-cycle about their own models (but on the other hand, also has access to some of the best models which are not publicly available yet, which lends credibility).
If you follow maths, one can be reasonably confident that the models can now sometimes solve math problems that are “not too hard in retrospect”. I don’t know how substantial this particular problem was supposed to be, but it feels like it tracks?
By “not too hard in retrospect” I mean that the models are applying well-known techniques in settings where these are not obvious, but also not too surprising, e.g. where experts in that subfield will say things like “of course you’d do that” when examining the solution. (Of course, one should be careful with such self-reports, but I tend to find this believable)