“coding, writing, research, bizdev advice, and general ideation”
Depending what you mean by general ideation, these all basically require memory recall and low-IQ recombination of existing memories and techniques. The actual intelligence required is extremely low (though would maybe be much higher for a human who doesn’t natively reason the way LLMs do, the same way calculators can beat us at multiplication problems).
But if you ask AI to do any task that requires real creativity or intelligence, it flops. Game dev design, novel programming problems that require judgement, writing jokes, writing stories anybody would want to read, writing scripts for social media content anyone would want to watch, coming up with app ideas, doing research, etc. These things all require Actual Intelligence, i.e. the ability to efficiently navigate concept space by using judgement and forming new judgements to refine the search, and LLMs don’t have much of it. Maybe their “true” IQ is 10, to the extent IQ makes sense for an AI model. But they fake it with memory the same way humans fake walking ability based on evolutionary memory.
Joke writing is my favorite personal benchmark, because the results are somewhat objective for a personal benchmark. You laugh at the AI output or you don’t. I have never had an AI model yet that could write funny jokes at all except by accident. Fable was the first model where I was often able to detect a faint hint of something. Some of the premises felt like they had once been in the same room as an actual joke. But the model is still unfunny. The first funny model will probably be the one that kills us, because it implies intelligence that also unlocks all that other stuff, so I’m paying attention to this.
“coding, writing, research, bizdev advice, and general ideation”
Depending what you mean by general ideation, these all basically require memory recall and low-IQ recombination of existing memories and techniques. The actual intelligence required is extremely low (though would maybe be much higher for a human who doesn’t natively reason the way LLMs do, the same way calculators can beat us at multiplication problems).
But if you ask AI to do any task that requires real creativity or intelligence, it flops. Game dev design, novel programming problems that require judgement, writing jokes, writing stories anybody would want to read, writing scripts for social media content anyone would want to watch, coming up with app ideas, doing research, etc. These things all require Actual Intelligence, i.e. the ability to efficiently navigate concept space by using judgement and forming new judgements to refine the search, and LLMs don’t have much of it. Maybe their “true” IQ is 10, to the extent IQ makes sense for an AI model. But they fake it with memory the same way humans fake walking ability based on evolutionary memory.
Joke writing is my favorite personal benchmark, because the results are somewhat objective for a personal benchmark. You laugh at the AI output or you don’t. I have never had an AI model yet that could write funny jokes at all except by accident. Fable was the first model where I was often able to detect a faint hint of something. Some of the premises felt like they had once been in the same room as an actual joke. But the model is still unfunny. The first funny model will probably be the one that kills us, because it implies intelligence that also unlocks all that other stuff, so I’m paying attention to this.