The recent improvements in practical utility of AI have decreased my probability of near-term recursive self-improvement. I’m updating towards a Chollet-like view. I’m increasingly convinced that LLMs really are a collection of fairly dumb heuristics (surface-level pattern matching) that are intelligent through breadth, and that there’s a qualitative difference from human intelligence. We use fewer heuristics but each one is smarter (more generally applicable, abstract, robust).
In the past it was hard to distinguish whether LLMs were developing smarter heuristics with each improvement or just deploying more and better-selected dumb heuristics. “Needs smart heuristics to solve” and “really hard to do” are highly correlated properties of tasks, so every capabilities gain looks like evidence for model general smartness. Now that we see really hard tasks being done we can differentiate the two properties, so I revised my prior capabilities-based timeline updates down. I worry people are now making the same conflation between “Needs smart heuristics to solve” and “unverifiable.”
Problems that are bottlenecked by deploying smart heuristics won’t be accelerated as much as people expect. AI research is one of them, as evidenced by labs paying extreme premiums for research taste rather than massively expanding headcount.
I think people anchor on the AI framing when thinking about post-AGI utopia. Most stories I’ve read have assistants where you request a thing and it happens, maybe with some constraints. But the better way I found to think about this is to try to forget the AI framing and the heuristics I developed by learning about the specific historical path by which humans improved our world with technology.
If you frame the question around what is the ideal human life (which is basically the same question as what is utopia, but without asking about the mechanisms that make it possible) then I believe most people who have ever lived would picture some configuration of humans having recognizable experiences, where the distribution of those experiences is lopsided towards happiness. If you then replied that, for you, the ideal life is having a constraint-bound genie that does nearly whatever you ask, they would find that very random given the prompt. It’s oddly specific in a way that wouldn’t come naturally just from introspecting on what is an ideal life.
In contrast, I think they would not find it as odd if you pictured a world where everyone is lucky, where there seems to be a law that everything turns out well by default, even though that is no more realizable to them than the genie.
My vague intuition is that SAI should optimize the distribution of human experience, maximizing happiness and minimizing extreme or persistent suffering, under a “KL penalty” against the experience distribution of the ancestral environment, where happiness is “in-distribution”.