Wow, this beautifully illustrates the problem with current AI (they are very smart at short tasks and poor at long tasks) and the trend of improvement against this problem.
However I want to point out that the inability to do long tasks isn’t the only weakness AI have. There are plenty of 5 minute tasks which are common sense to humans but which AI fails at (and many benchmarks catch these weaknesses). It’s not just the length of the task but the type of the task.
I think AI are also bad at inventing new ideas and concepts if it’s too far from their training data.
Yes they used a 50% success rate and even then some sub 10min tasks are still troublesome for LLMs as seen in the graph. But I think this will improve aswell if we make the algorithms better
Wow, this beautifully illustrates the problem with current AI (they are very smart at short tasks and poor at long tasks) and the trend of improvement against this problem.
However I want to point out that the inability to do long tasks isn’t the only weakness AI have. There are plenty of 5 minute tasks which are common sense to humans but which AI fails at (and many benchmarks catch these weaknesses). It’s not just the length of the task but the type of the task.
I think AI are also bad at inventing new ideas and concepts if it’s too far from their training data.
Yes they used a 50% success rate and even then some sub 10min tasks are still troublesome for LLMs as seen in the graph. But I think this will improve aswell if we make the algorithms better