If you could match the performance of Gemma 4 E2B or maybe MiniCPM5 1B across a wide range of tasks involving natural language understanding, writing and very simple agentic tasks, that would be incredibly interesting. It would also shatter my understanding of what’s feasible.
Except that this largely been tried. The Cyc project pushed various GOFAI approaches about as hard as anyone could, with plenty of funding and a small army of PhDs. And it never handled natural language as well as even a very basic LLM.
I have a simple toy problem that I use to explain this, something any simple visual LLM can mostly do:
Your input is an image.
Something is happening in that image which tells a story or presents a situation.
The story or situation implies a moral choice.
The program must understand the image, determine which choice is morally correct, and output an explanation as text. Any plausible answer is sufficient.
To implement this using pre-LLM techniques, you would need to solve:
Turning a matrix of pixels into a description of the scene.
Extracting the story and the moral dilemma.
Applying common sense moral rules to the dilemma.
Explaining a decision as English-language text.
The fact that this problem starts with an image input is deliberate. Nobody has ever managed to implement natural language understanding with if and else statements, but many smart people people have wasted decades trying. Using an image as input forces the implementor to deal with an inherently matrix-shaped problem immediately. In practice, the best algorithms for all these subproblems turn out to be matrix-shaped: A large number of inputs, a many-to-many transformation, and a weighted list of possible interpretations as outputs. (In practice, you need multiple layers of matrices and various non-linearities for these problems.)
If you can solve this problem as well as a good phone-sized LLM using ordinary Python code, then you will dramatically change my opinions about a lot of things.
(If you can furthermore guarantee that your system returns the correct answer for every image with a moral dilemma, then I’ll additionally believe that solving alignment robustly might be possible. That was the original goal for this thought experiment.)
Thank you for sharing this entirely terrifying anecdote.