For what it’s worth, the most relevant difficult-to-fall-prey-to-Goodheartian-tricks measure is probably cross entropy validation loss, as shown in this figure from the GPT-3 paper:
Serious scaling efforts are much more likely to emphasize progress here over Parameter Count Number Bigger clickbait.
Further, while this number will keep going down, we’re going to crash into the entropy of human generated text at some point. Whether that’s within 3 OOM or ten is anybody’s guess, though.
For what it’s worth, the most relevant difficult-to-fall-prey-to-Goodheartian-tricks measure is probably cross entropy validation loss, as shown in this figure from the GPT-3 paper:
Serious scaling efforts are much more likely to emphasize progress here over Parameter Count Number Bigger clickbait.
Further, while this number will keep going down, we’re going to crash into the entropy of human generated text at some point. Whether that’s within 3 OOM or ten is anybody’s guess, though.