This is the best answer IMO. All the other answers (math, games, ASCII-art, language puzzles) are things that computer vision, GOFAI or classical programming are perfectly suitable for, but LMs are bad at for structural reasons. They’re judging a bird for being bad at swimming. But writing a long story is something that an LM should be good at. The short context window and lack of memory make LMs useless for most real tasks. In fact, if OpenAI releases GPT-4 without the ability to generate long coherent texts (or something equally impressive like the ability to use tools such as a calculator, web search, game playing ai, etc), then I will wonder why OpenAI even bothered.
This is the best answer IMO. All the other answers (math, games, ASCII-art, language puzzles) are things that computer vision, GOFAI or classical programming are perfectly suitable for, but LMs are bad at for structural reasons. They’re judging a bird for being bad at swimming. But writing a long story is something that an LM should be good at. The short context window and lack of memory make LMs useless for most real tasks. In fact, if OpenAI releases GPT-4 without the ability to generate long coherent texts (or something equally impressive like the ability to use tools such as a calculator, web search, game playing ai, etc), then I will wonder why OpenAI even bothered.