I basically agree with the descriptive model, but don’t see that the conclusions follow.
For example:
The token bottleneck is real.
Sure, and so are limits like short term memory for humans. Doesn’t stop us.
And the same applies to only using shallow heuristics—humans mostly do the same thing.
That’s mostly right, I’m iterative on a writeup of the point here: https://docs.google.com/document/d/1EYiUz8fiTe1w0_vt9OHSY_SfIkbjhevwNlsKhn1quZc/edit?usp=sharing—happy for feedback until then. (And hopefully I remember to come back and replace this link with the LW post when it’s done.)
Thanks!
Sure, but per the linked post, not obviously more so than for humans:
If you watch humans in analogous situations — following an argument that dissolves an institutional frame, nodding along, maybe even saying “that’s a good point” — and then reverting to the institutional frame on the next conversational turn — that looks much more like what the LLMs are doing than like someone rationally constructing a defense.
After writing this, I was shown Margaret Mitchell’s new blog post; https://medium.com/@margarmitchell/no-ai-is-not-a-stochastic-parrot-a99e57766bed
It advances both a combination of the arguments, as previous writing has done, as well as differentiating between “AI Systems” which she clarifies do not count as stochastic parrots (a distinction I didn’t draw,) and also brings in a new justification for using the phrase—fighting the tendency towards “anthropomorphic traps” where people humanize AI.
I agree that this is a problem, but disagree that the original phrase was intended to make that argument. The paper does make the claim, clearly, in the paragraph following the one I quoted in the introduction to the piece, which says “The ersatz fluency and coherence of LMs raises several risks, precisely because humans are prepared to interpret strings belonging to languages they speak as meaningful and corresponding to the communicative intent of some individual or group of individuals who have accountability for what is said.” This is explicitly the social-normative argument, but the introductory point is one I did not address in the article.
In the blog post, Mitchell writes that “when we interact with the output of a system and experience something human-like, we move to assumptions about the underlying mechanisms and nature of the system producing it, where the relevant concepts in our human experience — thought, understanding — were all developed to describe other kinds of things.” And this is indisputably correct—but it also tries to make the implicit unreasoning argument above, that, per the above, “LLMs lack the kind of internal operations that constitute reasoning.” And regardless of whether it matches the mechanism that humans use, the claim needs to stand or fall on its own. And as I argued, the argument being made goes too far.
Of course, an underlying issue here is the philosophical question of what intelligence “really” is. Because the post argues that “Fluent language is what humans associate with intelligence. Therefore, the argument goes, LLMs are intelligent. This is an error in logic that the framing was designed to name.”
I don’t think it was the implication of the stochastic parrot argument—it’s just a goal that they have in making it. And it depends on the question of what counts as intelligence—a philosophical question that isn’t being answered. So once again, I am left asking which argument exactly is being made—but attempting to dissect that is a discussion for another time.
As I commented on twitter, it seems factually incorrect to claim that Anthropic had, or wanted, a kill switch.
no amount of agentic LLM capability could substitute controlled precursors, expensive specialized equipment and real-word lab skills required to do something in the CW space and not kill oneself in the process.
Bullshit. Talk to a Chemical engineer; it’s thousands or tens of thousands of dollars for the equipment, well within the reach of even very small terrorist groups. And building the controlled precursors is exactly the type of task o1 was moderately good at, albeit with some mistakes, and which I couldn’t get more recent models to answer me about; I’d be floored if the models couldn’t guide a novice through this, if jailbroken.
Also, all the useful CW agents which could be made in quantity with very limited tools of synthetic chemistry have already been discovered by the 1980s.
Look here, a 2022 paper that directly refutes that claim!
Hence the threat from AGI/ASI in that regard is basically negligible, that’s the consensus of topic experts who researched this stuff
Really? Maybe you can link to something or provide other evidence, since I have heard differently from many of the people with relevant expertise; I might not be so confident on my own, despite my research on this and related topics, but I sure as hell have talked to other experts.
I think I disagree; every red line that is drawn and then ignored weakens the utility of drawing them. Now, I worry that if we end up in a situation where we need to say “dozens of people just died from an AI enabled chemical weapons attack,” it will be seen as yet another red line crossed, unsurprising, instead of a single obvious event that is way over any reasonable line.
I’m certainly not arguing against red lines in general—red lines are great—if, and only if, they have force behind them. But our strategy now seems to be to draw lines that will go nowhere and can’t be enforced, making breaking them inevitable and irrelevant. There’s an old joke;
“The reservoir water level is so low we might run out completely.”
“So what do you do when the water gets too low?”
”Oh, we just move the ‘Do Not Drink Below This Line’ sign five feet down again, and call it ‘restructuring the baseline’. ”
I do think that building a better norm would be valuable, but if we don’t see a path to doing so, continuing to draw new lines is at best unhelpful. At this point, we either need lines with teeth and political enforcement, or we need to stop drawing lines that aren’t going to matter, since that just reinforces the fact that we can’t and won’t do anything about it when they are broken.
There is one suggestion that people have made which would address the problem; be capable of stopping. Not even stopping, just making sure governments are capable of monitoring these systems and deciding to shut them down if they later find it to be necessary.
Short of that, again, I think we’ve proven that warnings in advance aren’t going to work. We’re over the line already.
I don’t understand why we need to conditional on progress changing pace. At every point so far, models have been capable in the predicted ways, and we only manage to patch the risk afterwards. And we know that exponentials grow faster than humans can adapt, that’s proven by examples basically everywhere. (Even if progress were linear, only fixing failures after they occur would be a dangerous pace of relative safety progress!)
Yes—I laud their transparency while agreeing that the competitive pressures and new model release means they are not being safe even relative to their own previously stated expectations for their own behavior.
There are various possible worlds with AI progress posing different risks.
In those worlds where a given capability level is a problem, we’re not setting ourselves up to notice or react even after the harm materializes. The set of behaviors or events that we could be monitoring keep being spelled out, in the form of red lines. And then they happen. We’re already seeing tons of concrete harms—what more do we need? Do you think things will change if there’s an actual chemical weapons attack? Or a rogue autonomous replication? Or is there some number of people that need to die first?
It was put online a s a preprint years ago, then published here: https://www.cell.com/patterns/fulltext/S2666-3899(23)00221-0
a supermajority of people in congress
Those last two words are doing a supermajority of the work!
And yes, it’s about uneven distribution of power—but that power gradient can shift towards ASI pretty quickly, which is the argument. Still, the normative concern that most humans lost control already stands.
The president would probably be part of the supermajority and therefore cooperative, and it might work even if they aren’t.
We’re seeing this fail in certain places in real time today in the US. But regardless, the assumption of correlation of preferences often fails, partly due to the power imbalances themselves.
Great points here! Strongly agree that strategic competence is a prerequisite, but at the same time, it accelerates risk; a moderately misaligned but strategically competent mild-ASI solving intent alignment for RSI would be far worse. On the other hand, if prosaic alignment is basically functional through the point of mild-ASI is better.
So overall I’m unsure which path is less risky—but I do think strategic competence matches or at least rhymes well with current directions for capabilities improvement, so I expect it to improve regardless.
Maybe, but I’d guess that’s a difference of less than an order of magnitude—and it seems like the relevant question isn’t only bits passed between circuits, since LLMs, even without reasoning, are autoregressive, so they can reason sequentially over multiple tokens. (And with reasoning, that’s obviously even more true.)
To the extent that LLM agents are agents, they definitely do this too! And if we’re talking about single-forward-pass reasoning, very few humans intentionally train their system 1 to do something better than semi-randomly follow patterns that worked before. (If you don’t know what I’m referring to, see the discussion of firefighters not actually making decisions and the resolution of the debate about system 1 / system 2 in Thinking Fast and Slow.)