Many have asserted that LLM pre-training on human data can only produce human-level capabilities at most. Others, eg Ilya Sutskever and Eliezer Yudkowsky, point out that since prediction is harder than generation, there’s no reason to expect such a cap.
The latter position seems clearly correct to me, but I’m not aware of it having been tested. It seems like it shouldn’t be that hard to test, using some narrow synthetic domain.
The only superhuman capability of LLMs that’s been clearly shown as far as I know is their central one: next-token prediction. But I expect that to feel like a special case to most people. There’s also truesight and related abilities, but that’s just a corollary of next-token prediction, and also I’m not aware of a conclusive comparison between LLMs and humans on that capability.
Is anyone aware of some other superhuman capability that’s been clearly demonstrated? Bonus points if it’s something that at least some humans care about for unrelated reasons; bonus points if the capability is broad rather than narrow. But I’d love to hear about any!
Extending base64-encoded text? E.g. if I feed the text “Rm91ciBzY29yZSBhbmQgc” (the base64-encoded opening of the Gettysburg Address) into a base model, it’ll likely continue that base64-encoded speech much more accurately than any human would.
Many have asserted that LLM pre-training on human data can only produce human-level capabilities at most. Others, eg Ilya Sutskever and Eliezer Yudkowsky, point out that since prediction is harder than generation, there’s no reason to expect such a cap.
The latter position seems clearly correct to me, but I’m not aware of it having been tested. It seems like it shouldn’t be that hard to test, using some narrow synthetic domain.
The only superhuman capability of LLMs that’s been clearly shown as far as I know is their central one: next-token prediction. But I expect that to feel like a special case to most people. There’s also truesight and related abilities, but that’s just a corollary of next-token prediction, and also I’m not aware of a conclusive comparison between LLMs and humans on that capability.
Is anyone aware of some other superhuman capability that’s been clearly demonstrated? Bonus points if it’s something that at least some humans care about for unrelated reasons; bonus points if the capability is broad rather than narrow. But I’d love to hear about any!
Extending base64-encoded text? E.g. if I feed the text “Rm91ciBzY29yZSBhbmQgc” (the base64-encoded opening of the Gettysburg Address) into a base model, it’ll likely continue that base64-encoded speech much more accurately than any human would.
LLMs are of course also superhuman at knowing lots of facts, but that’s unlikely to impress anyone since it was true of databases by the early 1970s.