Vagabond
CEEALAR alumni
Samuel Knoche
Karma: 403
I think you’re attributing an argument to me which I wasn’t making (in the context of that post that you copied the diagram from). I agree that comparing 30 hours of teen driving practice to umpteen gazillion hours of Waymo training data is apples-and-oranges because the teen also has life experience.
But I was making a different point, which (in my own words) was: “…we don’t have AGI (artificial general intelligence) yet—not as I use the term…”. (I’m not even sure you disagree with that??)
Yes, sorry, I can see how that might give people the wrong impression of your views. The diagram was just meant to illustrate the reading that the self-driving case points to a missing major algorithmic ingredient that makes humans very sample efficient learners. The later line (“Humans don’t learn to drive in thirty hours, they are fine-tuned on driving after a roughly two decade-long pretraining run”) was aimed at a more naive version of the argument someone might hold, not at you specifically.
I agree that we don’t have AGI yet, but given what current deep learning architectures/algorithms have already achieved, I don’t think that the ingredients will look too exotic or different from the types of algorithms we have now.
On the narrow self-driving car case, I find it plausible that a model that’s basically a significantly scaled up version of EZ-V2 (requiring >3 OOMs more memory than current Tesla FSD hardware at inference), and given maybe ~2 years of diverse real driving experience (and perhaps alongside a few non-exotic additional regularization and data augmentation techniques), could learn to drive as well as a human. Well, I’d put it at maybe 60% chance that this is true.
On why we can’t teach a model to drive the way a human would, I agree that we’re still missing a few algorithmic ideas, especially related to how to train a good world model for generalizable/transferable online learning and spatial navigation, but even if we did have the right algorithm, I disagree that it’d be “way way way easier” than what Waymo and Tesla have been doing, and I think “the resulting AI would be too big to fit in a car computer” is one major factor. Beyond the scale of the frozen policy itself though, I do expect that the algorithmic thing that is missing for human-like learning will require more compute of some form. Though I may just be lacking imagination about what alternative algorithms are possible.
Hmm, yes, that was poorly phrased. You both think we need a new paradigm, and that something fundamental is missing, but your conclusion as to what it means for how worried we should be is opposite.
Changed that last sentence to:
> They both agree that current algorithms are missing something fundamental, yet draw opposite conclusions about what that means for AGI risk and timelines.
Dissolving the Deep Learning Sample Efficiency Gap
Nice, thanks!
Love it, but sad that the FTX song didn’t make it.
Between the cosmic endowment, the doomsday argument, and simulation theory, my sense of cosmic importance has been yoyoing across ~60 orders of magnitude.
It’s crazy how realistic sci-fi sounds these days.
And newer approaches[8] seem to eliminate the problem entirely, so it is now a contingent engineering limitation rather than a fundamental critique of LLMs.
Only skimmed it, but I think the two main contributions are a way to generate useful synthetic text from a limited corpus so pretraining can keep improving even when new real data is scarce (paper), and an automated AI research loop design (paper), a bit more fancy and automated, but similar to what Karpathy has recently been posting about.
If you’re looking for examples of LLMs that do weight based continual learning/belief update, the Titans, and Nested Learning papers might be a better fit (I described them in a recent post), or, if you want something closer to a vanilla transformer architecture, E2E-TTT.
I recently wrote a post surveying some weight based continual learning research that you may find interesting.
I agree that because base models can have such broad pre-existing knowledge and skills, can be trained with RL to manage context and memories externally, and have a super-human context window/working memory, weight-based continual learning isn’t strictly necessary to get to some fairly powerful systems, probably able to automate much of white-collar work. But I also suspect that weight-based continual learning may not be that far off, and that it could offer some qualitative advantages leading to faster capability progress than we otherwise would have.
Are We in a Continual Learning Overhang?
WebGPT probably already can, because it can use a text based browser and look up the answer. https://openai.com/blog/webgpt/
Worth noting though that you should expect those who are most successful at getting more of what they want to have a utility function with a big overlap with the utility function of others, and to be able to credibly commit to not destroy value for other people. We live in a society.
This one should work: https://discord.gg/56cve5YZQH
There’s a lot of value to be had in understanding the wisdom of existing informal evaluation systems and scaling them into formal ones.
One consideration to keep in mind though is that there might also be a social function in the informality and vagueness of many evaluation systems.
From Social Capital in Silicon Valley:
The illegibility and opacity of intra-group status was doing something really important – it created space where everyone could belong. The light of day poisons the magic. It’s a delightful paradox: a group that exists to confer social status will fall apart the minute that relative status within the group is made explicit. There’s real social value in the ambiguity: the more there is, the more people can plausibly join, before it fractures into subgroups.
There is probably a lot to be improved with current evaluation systems, but one always has to be careful with those fences.
I maybe wasn’t clear about what I meant by ‘the game.’ I didn’t mean how to be a good public intellectual but rather the broader ‘game’ of coming up with new ideas and figuring things out.
One important metric I use to judge public intellectuals is whether they share my views, and start from similar assumptions. It’s obviously important to not filter too strongly on this or you’re never going to hear anything that challenges your beliefs, but it still makes sense to discount the views of people who hold beliefs you think are false. But you obviously can’t build an objective metric based on how much someone agrees with you.
The issue is that one of the most important metrics I use to quickly measure the merits of an intellectual is inherently subjective. You can’t have your system based on adjudicating the truth of disputed claims.
I like the thought. Though unlike sports, intellectual work seems fundamentally open-ended, and therefore doesn’t seem to allow for easy metrics. Intellectuals aren’t the ones playing the game, they’re the ones figuring out the rules of the game. I think that’s why it often is better to focus on the ideas rather than the people.
A similar question also applies within academia. There, citation counts already serve as a metric to measure intellectual accomplishment. The goodharting of that metric probably can tell you a lot about the challenges such a system would face. What metrics do you have in mind?
In a way, this problem is just scaling up the old reputation/prestige system. You find people you respect and trust, and then you see who they respect and trust, and so on. While regularly checking for local validity of course. Maybe some kind of social app inspired by liquid democracy/quadratic voting might work? You enter how much you trust and respect a few public intellectuals (who themselves entered how much they trust other intellectuals) and then it computes how much you can trust everyone else.
True. I still think that market solutions would arise. Income Sharing Agreements (ISA) in particular seem promising for this kind of situation.
I agree that it is less bad in Europe and the UK. To quote from a response I gave to a related question:
“I see the education in countries where it is free/mostly free as better than the US, but many of the same criticisms still apply. What is taught, and how it is taught is the same as in the Anglo-saxon World. See here.
Also, the higher education system in those other countries is still quite wasteful (people learning things they don’t need or want to learn, zero-sum competition for credentials...). It’s just that the cost is borne by the taxpayer and not the students themselves. This is probably better than getting young people deep into debt, but still not great.
I think that countries where higher education is free are more ‘locked-in’. So, I’m much more hopeful about significant improvements coming out of the Anglo-saxon world than out of continental Europe, since in Europe any innovative education startup has to compete with a free alternative.”
On the question of AI safety work being done in academia, it is my understanding that many people working in academia have to hide the true intent of their research, or frame it in some way that sounds more in line with more traditional lines of inquiry. This post has some interesting discussions on this topic: Intellectual Progress Inside and Outside Academia.
There’s an old question, “What does the Bible God need to do for the Christians to say he is not good?” What would academia need to do before you let it go? -Yudkowsky
By “if public education is all about helping employers in their hiring process” I am only referring to the ~80% signaling. I mean to say that the government shouldn’t help companies select candidates, and definitely not in such a wasteful manner.
I agree that increasing human capital is a good goal (alongside creating the public good of an educated citizenry). It’s just that the government does this very very inefficiently. I discuss this in more details in The Case for Education
Pagliardini et al. (2024) and Heddes et al. (2025) are even older. Would be surprised if closed frontier models haven’t been using such methods for a while.