It’s not a Schelling point if you communicate about it!
utilistrutil
Karma: 454
If you had a person who could only learn by imitation, but could learn very sample efficiently, and could effectively recall and synthesize past knowledge, would you count that as continual learning?
For example, this person learns metalworking in a day by working under a blacksmith. Then they learn violinmaking from a luthier the next day. Then all on their own they can make a violin made of metal.
But they don’t do any online learning / exploration, just imitation.
I would claim this is still “true” continual learning in the ways it’s typically discussed. Do you agree?
If so, I think the bar is lower than your post makes it out to be: it would seem feasible that improvements in our current actual imitation learning algorithms could achieve continual learning.
AGI Should Have Been a Dirty Word
Epistemic status: passing thought.
It is absolutely crazy that Mark Zuckerberg can say that smart glasses will unlock personal superintelligence or whatever incoherent nonsense and be taken seriously. That reflects poorly on AI safety’s comms capacities.
Bostrom’s book should have laid claim to superintelligence! It came out early enough that it should have been able to plant its flag and set the connotations of the term. It should have made it so Zuckerberg could not throw around the word so casually.
I would go further, and say that the early safety writing on AGI should have been enough that the labs were too scared to say in public in 2026 that they’re trying to build AGI. Instead, it’s commonly accepted in Silicon Valley that developing AGI is a shared goal.
I would still bet that the early safety writing has gone a long way toward establishing the credibility and influence of AI safety in the present. But there must have been something we could have done better.
Thanks! :)
Re going along with lies—Yeah, I think the coverage of data center water usage has been an example of that at its worst :/
Re journalists sitting on scoops—I’m curious if you’re able to share any examples? I don’t doubt that it happens.
I agree it’s possible and it’s worth thinking through considerations like this. But I still don’t think this is a good model of journalists’ incentives.
In practice, “probability of being seen as inaccurate” is the term that dominates, which means inaccuracies tend to show up at points in the news article that face the least scrutiny, eg the part of an AI article where the journalist rushes through what a transformer is. These are the parts that are often least important to readers, and least important to you as a source.
And then I would describe the motivation more as “career success” than “political benefit”. As in getting a big scoop or writing a successful story, more than pushing a particular agenda. I think what journalists’ consider a successful story is kind of correlated with importance to the reader, barely correlated with what’s impactful, and barely correlated with how frustrating it would be for you to be misquoted. Consider the ChatGPT suicide example: the journalist is focused on their big scoop, but probably cares much less about the paragraph I pulled out. Ditto for readers. But I think it’s very valuable it was included.
I’ll have more on this in the epistemics post.
To make the analogy stronger, what if it only inverts it 1⁄10 times? Then I think the answer is non-obvious and depends on your principles.
If you’re deontological about it, I think you could make a case that your hands are not dirty for making the best of a bad system.
If you’re consequentialist about it, I’m saying the 9⁄10 accuracies could outweigh the 1⁄10 inaccuracies. And as Zack said, the 1⁄10 errors are rarely true inversions. That’s why
even if you do get misquoted, it doesn’t mean talking to the journalist was net-negative, even for that particular piece and even ex-post. As annoying as it is, it might be outweighed by the value of steering the article in positive ways.
You are doing the lord’s work fr
MIT Tech Review doesn’t break much news. Try Techmeme.
Re “what people are talking about”
Sure, the news is biased toward topics people already think are important because you need readers to click etc etc. But you are people, so you might also think that at least some of those topics are important. Even if the overall news is mostly uncorrelated with your interests, you can filter aggressively.
Re “what they’re saying about it”
I think you have in mind articles that are mostly commentary, analysis, opinion. News in the sense I mean it here tells you about some event, action, deal, trend, etc that wasn’t previously public. News articles might also tell you what some experts are saying about it, but my recommendation is just to get the object-level scoop from the headline and move on.
Re is it worth the time of sifting through
Skimming headlines is fast. Maybe the news tends to be less action-relevant for your research, but I bet AI safety collectively wastes time and misses out on establishing expertise by being behind the news. Reading Zvi’s newsletter falls under what I’m advocating for (even though it’s mostly that what-people-are-saying commentary, the object-level news still comes through.)
Conditioning as a Crux Finding Device
Say you disagree with someone, e.g. they have low pdoom and you have high pdoom. You might be interested in finding cruxes with them.
You can keep imagining narrower and narrower scenarios in which your beliefs still diverge. Then you can back out properties of the final scenario to identify cruxes.
For example, you start by conditioning on AGI being achieved—both of your pdooms tick up a bit. Then you also condition on that AGI being misaligned, and again your pdooms increase a bit (if the beliefs move in opposite directions, that might be worth exploring!). Then you condition on the AGI self-exfiltrating, and your pdooms nudge up again.
Now you’ve found a very narrow scenario in which you still disagree! You think it’s obvious that a misaligned AGI proliferating around the world is an endgame, they don’t see what the big deal is. From there, you’re in a good position to find cruxes.
(Note that you’re not necessarily finding the condition of maximum disagreement, you’re just trying to get information about where you disagree.)
Got it thanks!
(eg. any o1 session which finally stumbles into the right answer can be refined to drop the dead ends and produce a clean transcript to train a more refined intuition)
Do we have evidence that this is what’s going on? My understanding is that distilling from CoT is very sensitive—reordering the reasoning, or even pulling out the successful reasoning, causes the student to be unable to learn from it.
I agree o1 creates training data, but that might just be high quality pre-training data for GPT-5.
Why does it make the CoT less faithful?
Favorite post of the year so far!
My favored version of this project would involve >50% of the work going into the econ literature and models on investor incentives, with attention to
Principal-agent problems
Information asymmetry
Risk preferences
Time discounting
And then a smaller fraction of the work would involve looking into AI labs, specifically. I’m curious if this matches your intentions for the project or whether you think there are important lessons about the labs that will not be found in the existing econ literature.
How does the fiduciary duty of companies to investors work?
OpenAI instructs investors to view their investments “in the spirit of a donation,” which might be relevant for this question.
I would really like to see a post from someone in AI policy on “Grading Possible Comprehensive AI Legislation.” The post would lay out what kind of safety stipulations would earn a bill an “A-” vs a “B+”, for example.
I’m imagining a situation where, in the next couple years, a big omnibus AI bill gets passed that contains some safety-relevant components. I don’t want to be left wondering “did the safety lobby get everything it asked for, or did it get shafted?” and trying to construct an answer ex-post.
I don’t know how I hadn’t seen this post before now! A couple weeks after you published this, I put out my own post arguing against most applications of analogies in explanations of AI risk. I’ve added a couple references to your post in mine.
Adult brains are capable of telekinesis, if you fully believe in your ability to move objects with your mind. Adults are generally too jaded to believe such things. Children have the necessary unreserved belief, but their minds are not developed enough to exercise the ability.
Seems like a general argument against all dual use technology?
Also the risk seems greater from military-specific robotics like drones and quadrupeds rather than general purpose robots.
Even supposing the argument holds in general, robotic capabilities are lagging far behind AI capabilities, so I would expect most people to argue that working on AI capabilities is much worse. Also because a misaligned AI controlling a robot army would be more dangerous adversary than a totalitarian human controlling a robot army.