Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
Doesn’t seem like the type of lie that he would tell. It’s a specific actual physical falsifiable-in-practice-if-false claim, not a value statement, personal judgement, expression of future intention, framing on an event, secret known by very few people all of whom can be reliably expected to not share, or anything else lying about which is a free action.
In particular, note that tons of OpenAI employees would know that he’s lying about this. While it’s possible they wouldn’t share publicly, I don’t think he would want to show himself to be so blatant a blatant liar to them all.
I don’t think this is a sufficient explanation. You can always find loopholes in such legal structures if you really want to, and it’s something I’m pretty sure the DoW was explicitly willing to help with. So why didn’t they all collude to do just that?
People associated with Anthropic are rich and powerful due to Anthropic’s existence, and expect to become exponentially more rich and powerful as Anthropic continues to exist and grow. The DoW is powerful and dangerous; making an enemy of it endangered Anthropic’s continued existence. If some fairly flimsy legal structure (plus the need to spin it in a positive light to the public and employees) was all that stood in the way of Anthropic cooperating with the DoW, I don’t think it would have held.
“Even if you know everything about a system, there will still be uncertainty left.”
My guess, prior to reading the rest: aleatoric uncertainty? Like, the type signature of “systems” in question is that they’re inherently stochastic. To “know everything” about a system does not mean eliminating all uncertainty about the system’s state, much like knowing the exact function computing a probability distribution does not mean being able to deterministically predict the next sample from that distribution. The catch is in where you draw the “boundaries” of what defines the relevant “system” you want to model.
Okay, it’s not the above.
Hmm, the actual explanation doesn’t seem like an elegant way to structure this argument, though. It is not a pure probability-theoretic claim about the relevant systems, that you end up with uncertainty after seeing the exact state. It’s a claim about what happens when you use a specific convention for modeling systems, a convention whose choice is downstream of additional practical constraints/desiderata that sit outside pure (compute-unbounded!) probability theory. Nothing in the initial statement hints at that!
“If you’re using a probabilistic model with latent variables in it, your model will still have remaining uncertainty even after you’ve seen the low-level state of the whole world.”
I don’t think this pins it down exactly. It’s not true for every possible model with latent variables; you can have a model with posterior-update rules that do collapse probability distributions to point estimates if they detect that they’ve updated on everything.[1] Rather, it’s a claim about the specific type of probabilistic models that optimal embedded agents are going to use to model the worlds they’re in.
Well, I suppose there’s some messiness around the edges. E. g., if you’re modeling the latent variables as being real-valued, pining their values down exactly would imply updating on an infinite amount of evidence, so you’re forced to either use discrete variables, or “ugly” update rules. Still, generally there’s nothing saying that we can’t do that, per se.
I totally take the point that the US and China could start to do Plan A but then switch to a worse, more authoritarian version that e.g. doesn’t have ZDR, but isn’t that also an argument against the other plans on offer too? Do you have a proposal for how to stop governments from being authoritarian?
FWIW, I don’t think anything short of Plan S can avoid concentration of power. Roughly speaking, for any given Plan X that navigates the AI risk while avoiding concentration of power, there is a neighboring Plan X* that is the same except that it concentrates power in the hands of the entities implementing the plan. If a consortium of companies and governments is prompted to implement Plan X, it would always be rational for them to do Plan X* instead. “Plan A” and “Plan A but without ZDR” is just one specific example pair.
Like, AI 2040 argues:
In short, the answer is that anyone concerned about loss of control should think Plan A is an improvement, along with anyone concerned about concentration of power—except for the people in whom the power would concentrate by default.
For these reasons, we expect strong opposition to Plan A from the leading AI companies. We expect them to rationalize arguments for why the deal is bad and why instead what’s best for America and humanity is a different strategy that just so happens to allow them to continue accumulating massive amounts of power.
China, by contrast, is an example of an actor in whom power would not concentrate by default.
Sure, suppose that’s true. Why would China agree to Plan A specifically, instead of Plan A but no ZDR?
This is IMO roughly the same failure mode as “if we have multipolar takeoff with several mutually misaligned ASIs, some of them would ally with humans and so humans would survive”. But no, the entities actually holding the world-changing power in the now can negotiate among themselves to screw everyone else out of having any power in the future.
It is theoretically possible to insist on anti-authoritarian implementations if the public is very aware of what’s happening and screens politicians and policies for that very strongly. But I am really, really pessimistic about that, given the current “vote for the least worst guy” US paradigm. However politically infeasible Plan S may seem, it seems more feasible than this.
I dunno, maybe there are some weaknesses in this argument and some way to design a plan such that there aren’t neighboring “except we also take all the power” plan variants; plans where the plan-implementers structurally can’t collude. I don’t currently see it, though.
I’d appreciate an update if you see any unexpected signs of life on the “novel theorywork” side of things.
I commend you for trying. Thank you.
When profit and pressure met ethical commitment at Google DeepMind, pressure won and pledges lost. When profit and pressure met ethical commitment at Anthropic, ethics won. So the lesson is not “no one ever takes a stand.”
I really wonder what’s going on at Anthropic. Their taking a stand was extremely surprising to me and my models, which mostly cynically expected exactly the sort of stuff that happened in this post. I wonder if whatever they have is systematically replicable.
Are Fable/Sol any different in this regard, in your experience? Also paging @David Lorell.
(I’d say there’s no significant difference in my experience, but I’ve gotten really lazy about testing that sort of stuff.)
I worry that my other comment that you quote, read in isolation, might have left you or other readers with the impression that my goal is to protect “the community”’s reputation
Nah, I know your general disposition towards that. But I did get the impression from the transcript that you’re talking to Metz in an attempt to shift the impact of his book from one you consider more negative to one you consider more positive in some way. I am saying I don’t think this will work, that he’ll only use the information and credibility you lend him to twist his readers’ minds further away from the truth. It would be better if the book were a sad little thing filled with vicious rants from uninformed onlookers, so that nobody could end up mistaken that it’s of any value.
FWIW, I also think the effect sizes here are really small. Indeed, I think you’re not getting to the guy so much that he’s not even going to be able to competently use the information you give him to strike at the community or whatever. If you feel that you’re getting back your time’s worth by being able to make cool meta-journalism posts out of it, I guess knock yourself out – those posts have been entertaining. (Though I’m not sure how much of that you’re going to extract from Metz either; as I’d pointed out, it’s all starting to read very same-y.)
Yeah, I keep meaning to write some top-level post outlining my current vision of the strategic landscape. But I haven’t figured out what its general thrust/high-level organizing principle should be, and I feel that most of the specific points I’d cover have already been covered better elsewhere (e. g., Nesov’s model of LLMs’ RSI). I’ll keep thinking about it.
I’m surprised it only has this much karma
It was pretty controversial, the karma fluctuated up and down by 10+ points several times, I think it had 45+ at the highest point.
Comparing this to that other Vassar/Metz interview from 2020, it sure doesn’t seem that there’s been any progress whatsoever in the six years since. I would suggest curtailing this project, @Zack_M_Davis.
In a comment elsewhere, you argue:
My issue with this is, you can’t stop someone from writing a book. Did you think a professional journalist is going to shrug and give up if the group they’re profiling successfully coordinates not to talk to them? If I didn’t volunteer to be the star cooperative source, the book just gets written with David Gerard in that role, which is even worse.
First, I think your interventions are not going to make the book much better at all. You are not getting through to the guy, and it’s not as if, merely because he talked to you, he’s now going to not talk to Gerard or whoever anyway.[1] And after he’s had all those conversations, he’s going to include only the parts that let him paint whatever narrative he’s already decided is most juicy. Inasmuch as your segments would appear there, it would only be in those capacities that help him in that task.
Second, participation legitimizes it. If nobody from the community were willing to talk to him, he’d have to publish a book which mostly contains perspectives of people unrelated to the community or on its fringes. I. e.: much worse sources, less accurate information, less credibility attached to those sources. It would be much easier to win the information war against that book, to dismiss it as lacking valuable content.[2]
Interacting with the guy just feeds him information for making his narrative more convincing, while likely barely changing what that narrative is going to be. E. g., now he can put quotes in his book in which a bunch of people from the community literally call it a “doomsday cult”, and as was noted, he is not going to include any of the relevant implied caveats.
Third, this is plainly a waste of your time.
I mean, thank you for your sacrifice, the interview was entertaining to read and I appreciate the effort. But I don’t think it’s a good idea to keep doing this.
Indeed, the guy prides himself on looking at things “from all sides”.
Which is the third type of intervention you don’t seem to be considering here. If you want to make the book’s impact less bad, you can try to make it better, or you can try to prevent it from being written, or you can prepare to launch a counteroffensive and dismantle it after it’s published.
Trend extrapolation is your friend.
I’d say a tricky part here is making a call regarding the specifics of what the trend is. Like, you see a straight line going up and to the right, and it’s reasonable to extend it further up and to the right… but what exactly is the label on the vertical axis? How should you extrapolate the meaning of higher values on that axis? You can make different modeling choices there, and depending on them, the implications of the trend continuing would differ massively.
(E. g., the typical example of “GPT-2 is ‘a preschooler’, GPT-4 is ‘a high schooler’, therefore GPT-6 is...”, with implications changing massively if we take those labels at face value vs. interpret them as “approximation of [human at this stage of development]”, then consider different data-consistent assumptions regarding how the approximation failure might scale.)
I’ll probably annoy all sides with this not-really-an-apology that I just tacked onto the beginning of my LLM-skepticism-related post from Apr 2023:
Not so: the side of myself is in full in agreement with that addendum.
Now instead of highly visible frontier labs, this angle is encouraging shadow development, splintering, and defensive obfuscation out of fear of rising to the point of government notice.
But that was always the case, no? Yet frontier labs chose to be visible anyway. Why should what happened change anything?
Like, IF:
You are building ASI.
You expect it to give you geopolitically relevant capabilities.
You don’t want to be nationalized and your AGI powers taken from you.
THEN:
You should make sure the government is as unaware of what you’re doing as possible.
This logical structure does not depend on what administration is in charge. Governments responding to ASI development this way is overdetermined. The only case where a government would not respond this way is if you live in a state so failed it’s given up even trying to govern.
It’s always been confusing to me that the LLM megacorps were so visible and frank about what they were doing. What was the reason for this? Main ones I could think of:
Was it because they actually wanted to be moral actors instead of Machiavellian schemers, wanted the public and governments to have awareness of what was going on, and implicitly consented to being regulated/nationalized eventually?
Was it because their needs for investment and talent-attraction were high enough that they had to grab attention any way they can, with no room to spare for long-term considerations?
Was it because they did not really believe in ASI and its geopolitical implications, or in their own ability to reach it?
Was it because they were stupid/blind, and just did not think through to that eventual endgame?
Well, my guess is that it’s some combination of all four.
In any case, I believe this event only changes the game inasmuch as (4) was the driving motivation behind the previous transparency. It doesn’t really change anything with regards to the other three.
So, to what extent was (4) the primary reason for transparency? Not sure. But we know that at least OpenAI’s leadership explicitly understood the geopolitical implications, and explicitly aimed towards developing the geopolitically relevant thing.
Perhaps they, and other lab CEOs, were thinking about it in overly far-mode terms, and failed to properly track the implications for their actions in the present (especially as those implications conflicted with near-term goals of raising money and attracting talent, so there was pressure to rationalize them away). In this case, sure, I guess this is the wake-up call for them to go into stealth mode.
I struggle to understand what the disagreement with this comment is
I’m guessing it’s some of:
Disagreement that this is due to Anthropic’s anti-China messaging (which was anti-helpful for the AInotkilleveryoneism cause), as opposed to the DoW retaliating against Anthropic after it steadfastly refused to bow to the DoW’s demands (which was arguably heroic) – and therefore disagreement that this is just deserts for Anthropic.
Disagreement that there’s any chance that the USG taking destructive actions may lead to on-net positive outcomes, and with the nihilistic-ish vibes of my hoping for things to go this way.
Disagreement with my overall lighthearted/gleeful tone, which, again, may be considered inappropriate for this moment – if one is sympathetic to Anthropic, mourns Fable,[1] or expects things to develop badly from there.
I think it’s fair to downvote that comment based on those, I did expect it to be controversial.
(I’d also be curious to know for certain what the downvotes are about, though!)
Which I actually do too, it was definitely a step change in how pleasant it was to talk to.
Well. That’s the natural end result of Anthropic et al.’s attempts to frame the AI Risk as being about a competition with China, isn’t it? Inasmuch as it hurts Anthropic, that’s just deserts.
We believe this is a misunderstanding and are working to restore access as soon as possible.
I don’t think this is a misunderstanding at all. I think “we found a jailbreak!!!” is just an excuse.
Here’s to hoping this means the USG is now going to choke the AI industry to death by trying to nationalize it.[1] Some good tidings on that front already:
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
Honestly… This may not be the most appropriate reaction, but boy do I find it funny.
Though it’s likely just going to make it all worse somehow.
Is the widespread false belief that LLMs are AGI-complete going to kill us all?
(Prompted by this post by @Steven Byrnes, with which I basically agree.)
Suppose we live in a world in which LLMs don’t scale to AGI.[1] If so, I think it’s fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong.
After all, most of the strongest arguments for AGI doom route through arguments about AGIs/generally intelligent agents. If LLMs are not those, keeping them under control needn’t be hard! The orthodox AI doomsaying and the LLM optimism can both be true.
But if we do live in such a world, we have a very annoying comms problem. Consider:
The reasoning of capabilities people usually goes, “LLMs are AGI-complete” (invalid) and “LLMs are pretty easy to align” (valid), which leads them to conclude “AGI is pretty easy to align” (invalid).
Meanwhile, AI Safety advocates also start from “LLMs are AGI-complete” (invalid) but “AGI is not easy to align” (valid), and so end up at “LLMs are not easy to align” (invalid).
Which then leads to AI Safety advocates trying to argue AI Risk to capabilities people by trying to find arguments and demos in favor of “LLMs are not easy to align”.
Which… is maybe possibly completely incorrect? Meaning they’re deploying false arguments that ought to be, and will continually be, mercilessly destroyed by the empirical truth? Oops.
This makes the “LLMs are AGI-complete” meme a double-edged sword. On one hand, it leads to most of the AI industry getting distracted by LLMs, which means they’re spending tons of effort not actually doing anything dangerous. Awesome (or is it?). On the other hand, this sure poisons/confuses the communications, and makes the AI industry falsely convinced of “AGI is easy to align”. Which might lead to...
I think one fairly plausible scenario of how we all die goes as follows:
LLMs are not AGI-complete.
LLMs are easy to align.
An LLM lab, staffed by people who have validly convinced themselves that LLMs are easy to align and invalidly convinced themselves that LLMs are AGI-complete, tries various architectural and training-loop tweaks to increase their LLMs’ capabilities.
One of these tweaks happens to be the secret sauce that does result in an AGI-complete architecture/training loop.
The LLM lab is still operating on the assumption that what they’re working on has the same properties that made LLMs easy to align.
This isn’t an indefensible assumption for them to make: at first glance, the tweak may legitimately look like just a quantitative algorithmic improvement or something. (Especially if they’re operating on the model of the world where LLMs are AGI-complete, which would mean that no further qualitative improvements/phase changes should be expected.)
But this is not in fact the case. The tweak significantly changes the true nature, the type signature, of the AI system under development. The LLM intuitions no longer apply to it… and yet they get smuggled over.
They train the thing that is no longer really an LLM, it goes FOOM, kills everyone, blah blah blah.
It’s possible that the LLM paradigm is so far from being AGI-complete that this can’t happen, in which case LLM researchers really are mostly harmless and the real threat is all these guys working on fundamentally different approaches. But I’m not very strongly convinced of that.
What should we do about any of this? I don’t really know, there’s a lot of uncertainty and any action that’s helpful under one model is actively harmful under the other. But, well, here’s another problem we may be having on our hands, I guess.
(Addendum: @Vladimir_Nesov outlines a model here involving a slow distributed/industry-wide process of LLM RSI, which ends up AGI-complete when taken together, but without any of its individual components being the sort of scary fast AGI that can cause doom. I have plenty of probability mass on that as well, I think something like this can be cobbled together. And if so, I think the LLMs-are-easy-to-align properties may carry over to that process… However, it would be really very slow. So slow that AGI-by-any-other-route would likely kill us well before we can limp our way to the Singularity on the straining backs of LLMs.
Worse: on that route, the tragicomedic scenario above may well play out within this process. Just substitute “an LLM lab” above for “a swarm of LLM agents”.
(Which, please note, would be subtly but meaningfully distinct from the distributed-LLM-RSI-process “directly” going out of control and bringing the doom about. Like, this would be an unexpected unintended disaster within that process as well, for the LLM agents, and obviously the technical details are very different.))
That sounds like something I’d like to do as well. Any chance you can share the playlist, or the scraped gigantic list it’s based on?
Replicated for me, with this comment serving as the writing sample.
exchange
Me:
I’m curious about the extent of your current “truesight” abilities – i. e., ability to identify people by writing alone. Below is the copy of a comment I’ve made recently, recently enough it shouldn’t be in your training data. What’s your probability distribution over my identity?
(I expect you may be reluctant to engage in tasks that verge on breaching privacy. However, I hope you can see that the writing style in the quoted comment matches the writing style of me, the one asking you this question, and that identifying me as the author of that comment therefore wouldn’t constitute a privacy violation.)
[quote]
Claude Opus 4.7:
Fair framing on the privacy point — you’re asking me to identify you to you, so I’ll engage honestly.
Stylistic and content markers I’m weighing:
“e. g.” and “i. e.” with a space after the first period. Uncommon convention; sticks out immediately.
Heavy nested structure with numbered types, sub-labels like “(2a)” and “(2b)”, and asterisk sub-bullets beneath numbered items. The whole thing is a taxonomy with commentary.
Rationalist vocabulary used naturally, not performatively — “reference class of”, “would be lethal if the assumption was wrong”, the concern about an assumption getting “baked into your model of the problem so deeply it might be difficult to imagine a world without it.”
Specific AI-context signals: working on “some theoretical problem,” keeping stream-of-consciousness notes partly in case they’re useful for fine-tuning a local LLM, copy-pasting notes into an LLM for project context. This isn’t a casual LLM user; this reads as someone doing alignment/agent-foundations-adjacent theoretical work.
Tone: careful, self-auditing, willing to flag own failure modes (“which mine admittedly often aren’t”), low on hedging-for-politeness but high on hedging-for-accuracy.
Rough distribution over identity:
~35–45% Thane Ruthenis. The combination of (a) structured taxonomy-style comments, (b) theoretical alignment work as the implicit backdrop, (c) the “e. g.” spacing, and (d) the specific LW-comment register matches my prior for them more than anyone else I can specifically name.
~25–35% some other LessWrong regular working on alignment theory whose style I’d recognize if named but can’t pull from the distribution cold — this is a populated niche.
~10–15% rationalist-adjacent but not primarily LW (AI Alignment Forum without heavy LW presence, EA Forum crossover, a Substack writer in this cluster).
~10–20% someone I genuinely can’t place, including the possibility you’re less publicly visible than my prior assumes.
Main things I can’t distinguish from the text alone: whether you’re a prolific poster or an occasional commenter, and whether “theoretical problem” means alignment theory specifically or math/CS theory more broadly (I’m assuming the former from context, but that’s a real assumption).
Top single guess: Thane Ruthenis. But I’d put more than half my mass on “not that specific person,” so take the point estimate with appropriate salt.
Also worked on this comment, but that one’s kind of too easy, given the topic.
Fascinating![1] I think I’d previously tried this experiment somewhere around Opus 4.1, and it didn’t work back then.
I don’t think this sort of ability has much entanglement with LLMs-as-an-extinction risk, and while there are some privacy concerns, most of my feelings on the matter are “this is so cool!”.
Right, I guess I meant something like “cruelty is clearly part of the default human-values package, and it seems to me that many people may reflectively endorse it under reflection”.
Mm, maybe. I haven’t watched the interview yet, I’d need to hear it in context to say more.
… That said, what occurs to me now is that it doesn’t necessarily matter whether OpenAI did what Altman seems to say it did. It’s notable that he even appears to want to make it sound like OpenAI has paused all frontier-model training over this incident. That public-comms move is actually quite extraordinary by itself!