I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI.
“Overconfident” gets thrown around a lot by people who just mean “incorrect”. Rarely do they mean actual systematic overconfidence. If everyone involved in building AI shifted their confidence down across the board, I’d be surprised if this changed their safety-related decisions very much. The mistakes they are making are more complicated, e.g. some people seem “underconfident” about how to model future highly capable AGI, and are therefore adopting a wait-and-see strategy. This isn’t real systematic underconfidence, it’s just a mistake (from my perspective). And maybe some are “overconfident” that early AGI will be helpful for solving future problems, but again this is just a mistake, not systemic overconfidence.
At no point in this discussion do I reference “limits of intelligence”. I’m not taking any limits, or even making reference to any kind of perfect reasoning. My x-risk threat models in general don’t involve that kind of mental move. I’m talking about near-human-level intelligence, and the reasoning works for AI that operates similarly to how they work now.
Wtithout optimally learning from mistakes
You’re making a much stronger claim than that and retreating to a Motte. Of course it’s not optimal. Not noticing very easy-to-correct mistakes is extremely, surprisingly sub-optimal on a very specific axis. This shouldn’t be plausible when we condition on an otherwise low likelihood of making mistakes.
If you look at the most successful humans, they’re largely not the most-calibrated ones.
The most natural explanation for this is that it’s mostly selection effects, combined with humans being bad at prediction in general. And I expect most examples you could come up with are more like domain-specific overconfidence rather than across-the-board overconfidence.
but just because it’s not the only useful thing and so spending your “points” elsewhere can yield better results.
I agree calibration is less valuable than other measures of correctness. But there aren’t zero-sum “points” to be distributed here. Correcting for systematic overconfidence is basically free and doesn’t have tradeoffs. You just take whatever your confidence would be and adjust it down. It can be done on-the-fly, even easier if you have a scratchpad.
If you think there’s a strong first-mover advantage you should care a lot about what the minimum viable scary system looks like, rather than what scary systems at the limit look like.
No, not when it comes to planning mitigations. See the last paragraph of my response to Tim.
This assumes that [intelligent agents that can notice their own overconfidence] is the only/main source of x-risk
Yeah, main. I thought this was widely agreed on, I’m still confused by how your shortform got upvoted. So maybe I’m missing a type of x-risk, but I’d appreciate the mechanism being explained more.
My current reasoning: It takes a lot of capability to be a danger to the whole world. The only pathway to destroying the world that seems plausible while being human-level-dumb is by building ASI. But ASI building still presumably requires lots of updating on evidence and learning from mistakes, and a large number of prioritisation decisions.
I know it’s not impossible to be systematically overconfident while succeeding at difficult tasks. But it’s more and more surprising the more subtasks it succeeds on, and the more systematically overconfident it is. Being systematically overconfident is a very specific kind of incompetence (and therefore a priori unlikely), and easily noticeable (and therefore likely to be human-corrected or self-corrected), and extremely easy to correct for (and therefore unlikely that the standard online learning process or verbalised reasoning didn’t generalise to this).
I don’t think the first AI smart enough to cause catastrophe will need to be that smart.
I think focusing on the “first AI smart enough” leads to a lot of low-EV research. If you solve a problem with the first AI smart enough, this doesn’t help much because a) there are presumably other AIs of similar capability, or soon will be, with somewhat different capability profiles and b) it won’t be long before there are more capable AIs and c) it’s hard to predict future capability profiles.
Yes, but what’s your point? Are you saying that highly capable (ASI building, institution replacing) but extremely epistemically inefficient agents are plausible? Without the ability to learn from mistakes?
I see that you’re doing large edits and additions to your previous responses, after I had already responded.
This, and the way you’re playing with definitions, makes me think you might be arguing in bad faith. I’m going to stop responding. If you had good intentions, I’m sorry.
Not what I meant. See Fully Update Deference.
That’s corrigible behaviour, but the mechanism is not, because it stops being corrigible after some number of updates. The idea of corrigibility is that it is correctable, and remains so, in spite of designers making mistakes in goal alignment (or other design mistakes). (Of course mistakes in however we enforced the corrigibility property itself might not be stably correctable, but the hope is that this part is easier to get right on the first try than the rest of it).
No. The kind of intelligent agent that is scary is the kind that would notice its own overconfidence—after some small number of experiences being overconfident—and then work out how to correct for it.
There are more stable epistemic problems that are worth thinking about, but this definitely isn’t one of them.
Trying to address minor capability problems in hypothetical stupid AIs is irrelevant to x-risk.
It doesn’t seem to me very conceptually complex to build an agent (in the AI sense) that knows “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests, and part of the challenge of doing this is that I don’t yet fully know what those are”.
I’m not claiming it’s conceptually complex. The conceptual simplicity is a large part of the motivation for pursuing this. Some of the difficulty comes up when you try to design the implementation details and need it to also be reflectively stable and generalise appropriately. The other difficulty comes from different implementations being difficult to distinguish from a training perspective, but having different reflective stability and generalisation properties.
b) Bayesian updating when the principal offers corrections (behavior which I would use the term “corrigibility” for)
That was not ever the intended meaning of the word. Using it like that is a great way to confuse everyone.
Which is why I prefer the older term “Value Learning”, or the newer term “AI-Assisted Alignment”
Both of these terms have different meanings from what I intend. AI-assisted alignment is closely related, but it is a process made possible by stable correctability, not the same thing as stable correctablility. Having a term for stable correctability itself is useful.
A sufficiently-close to aligned agent already knows and believes “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests”, so it only has to notice the second half of the sentence — which is really rather obvious to anything intelligent that isn’t actually built like AIXI.
I’m aware that there are certain kinds of almost-aligned that will correct themselves. But I don’t see this as very relevant, that kinds of brokenness was never the important kind, the important kind of design-error are the ones that affect a reflectively consistent degree of freedom.
That’s also why I’m less concerned we’re going to “run out of fuel” as we get closer to ASI — that’s when the AI can do more and more of the work itself.
You misunderstand my intended thesis, it’s not related to quantity of work, it’s closer to how much “ability to course-correct” that humans have.
ofI think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren’t that bad, so you can just build it to double the strength it would need on the lake and you’re good to go.
Yeah this does seem to be how a lot of people are thinking about it. I think the way to resolve this is to have people meditate on the non-analogy distribution shifts, but yeah doing this well requires having at least one somewhat-detailed model of intelligence, which isn’t that common.
I don’t think there’s much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail.
I think there’s a lot of evidence that AI builders don’t know what they’re doing in the relevant ways, and this evidence will likely get stronger and more widely acknowledged over time (as deployment stakes and capabilities make the occasional OOD weirdness more obvious). I’m sure the game of training against each embarrassing behaviour as it comes up will continue, but I hope that some will notice the pattern and extrapolate. It’s not that I’m wanting arguments to convince people, I think reality can convince people and good clear arguments just smooth the process.
The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it
I think in the particular case of this post, it would be obvious that humans don’t have the corrigibility property and are equally susceptible to distribution shifts.
That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.
Eh, all arguments are equal if you don’t think them through. I think it’s better to think of this kind of argument as setting the stage for the future, rather than winning over large groups of people right now (who you’re assuming aren’t even evaluating the arguments). There are possible futures where world leaders are deciding on a course of action, where a background fact is that it has become extremely obvious that we couldn’t win against a rogue AI. And many other potential futures where different things have become obvious and widely known. Many should provide evidence about alignment competence, but even the ones that don’t directly provide evidence here will provide plenty of motivation to think really carefully. And that plays to the benefit of careful and correct arguments.
combined with “ummm I’d like to not be permanently jobless” might make a powerful political movement for slowing/stopping. But that’s an entirely separate project from arguing the case for alignment difficulty on its merits.
I’m hoping that this, while somewhat misguided, might give politicians more leeway to make the correct choices.
I wonder if some well-considered optimists are assuming we’ll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they’re deployed. If I was confident developers would at least try to do that, I’d be a good bit more optimistic.
I don’t see how “improve training” is an available option even in theory.
That feels like flag waving or rationalizing or something. Obviously it’s not that simple, and you know that. You’ve got to do the prioritisation calculation. I don’t think it works out in that direction, but even if it did you should argue that by comparing the difficulties and counterfactual value of each research agenda.
The nearest unblocked strategy problem combines with the adversarial nonrobustsness of learned systems, to make a much worse combined problem. So I expect constraints to often break given more thinking or learning than was present in training.
Separately, reflective stability of constraint-following feels like a pretty specific target that we don’t really have ways of selecting for, so we’d be relying on a lot of luck for that. Almost all my deontological-ish rules are backed up by consequentialist reasoning, and I find it pretty likely that you need some kind of valid and true supporting reasoning like this to make sure an agent fully endorses its constraints.
Fortunately, things will probably be more gradual than that; we can try to control/steer AI systems not that different from todays, and then get their help controlling AI systems smarter than them, and so forth in a long chain all the way to ASI.
Why do you believe this reasoning at 50%? Is this due to new evidence or arguments? I’ve never heard it supported by anything other than bad high-level analogies.
There are a couple of factors that make the chain weaker as it goes along.
The level of human understanding decreases with each iteration, otherwise we would just do it without AI help. So any failure of helpfulness becomes less and less noticeable.
With this kind of test-driven engineering loop, especially in the current ML paradigm, it’s easier to hide problems than it is to deeply fix them. Accidentally hiding problems is easy today, but it becomes even easier as the AI becomes more complex and intelligent.
The only way I can see it going well is if an early AI realizes how bad of an idea this chain is and convinces the humans to go develop a better paradigm.
(I wrote a post about similar stuff recently, trying to explain the generators behind my dislike of prosaic research)
It looks like you wrote up a correct but abstract&approximate description of the problems with that approach. I claim that as you make your model of these problems more detailed and careful, refine the vague concepts and reduce the model uncertainties, the way that this looks like it just might work will disappear.
I know this isn’t very helpful to hear, and isn’t convincing. But that’s what my thoughts are.
We had a conversation in private messages to clarify some of this. I thought I’d publicly post one of my responses:
AI-2027 strategy
Step 1: Train and deploy Safer-1, a misaligned but controlled autonomous researcher. It’s controlled because it’s transparent to human overseers: it uses English chains of thought (CoT) to think, and faithful CoT techniques have been employed to eliminate euphemisms, steganography, and subtle biases.
Step 2: Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
Step 3: Train and deploy Safer-2, an aligned and controlled autonomous researcher based on the same architecture but with a better training environment that incentivizes the right goals and principles this time.
<...>
Step 4: Design, train, and deploy Safer-3, a much smarter autonomous researcher which uses a more advanced architecture similar to the old Agent-4. It’s no longer transparent to human overseers, but it’s transparent to Safer-2. So it should be possible to figure out how to make it both aligned and controlled.
Step 5: Repeat Step 4 ad infinitum, creating a chain of ever-more-powerful, ever-more-aligned AIs that are overseen by the previous links in the chain (e.g. the analogues of Agent-5 from the other scenario branch).
My post is mainly about how this step breaks:
Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
This process leads to finding a set of training environments that lead an AI to have a corrible-looking CoT. It doesn’t lead to finding training environments that produce an AI that is truly corrigible.
So Safer-2 has been shaped by a process that makes one good at looking corrigible. It probably isn’t corrigible, because there’s lots of ways to be non-corrigible but corrigible-looking-to-the-design-process.
Step 4 involves putting a huge amount of trust in Safer-2. Since Safer-2 is untrustworthy, it betrays us in a way that isn’t visible to us (which is extremely straightforward in this scenario because we’ve assumed there are important things that Safer-2 is managing that humans don’t understand).
Yeah, I’d be excited to do so via conversations with people, after building a shared ontology about AGI internals. And maybe publishing that, if it feels shareable.
I am kinda disappointed by how relatively unpopular this post was though, it makes me suspect that I’m missing something (likely just writing quality, but maybe I’m incorrectly extrapolating my model of people like you and Max and therefore misunderstanding my audience). This makes me think that me writing a more detailed version of this post isn’t the right next step. I think it might be a good project for others though.
I think there may be some real help from roughly human-level AGI that still thinks it’s aligned, and/or is functionally aligned in those use cases (it hasn’t yet hit much of “the ocean” in your metaphor). That could be really useful.
Yeah I agree with this, although the only way I can concretely imagine it helping is via the scenario I wrote at the end of the post.
I’d like to see more work on modeling conditions out at sea. That’s how I view my work: trying to envision the most likely path from here to AGI, which I see as going through LLMs enhanced in very roughly brainlike directions.
Agreed, although three comments:
I think you’re shooting yourself in the foot here by choosing the building material in advance (LLMs).
Separately, it’s a terrible building material, it lacks transparency or hooks or modularity.
It’s difficult to make intellectual progress on topics like this unless it’s a bit more formal. To some extent for communication reasons, we need to be able to iterate and build on each others work. But also because human reasoning is just too fuzzy to be doing this kind of OOD guesswork.
So I consider tiling agents and logical induction to be good work modelling the distribution shifts of learning/reasoning and self-modification.
I think there’s a ton of model uncertainty still. Playing to our outs involves working toward alignment on the current path
You’ve gotta be careful with this particular chain of reasoning. It may look like model uncertainty implies hope, but that doesn’t mean it implies hope in any specific work.
involves working toward alignment on the current path AND trying to slow or stop progress.
These two things can sometimes be opposed to each other. E.g. I’d bet that alignment work at OpenAI has been overall negative via giving everyone there the impression that they might be doing the right thing.
Based on that uncertainty, I think it’s quite possible that relatively minor and realistic changes to alignment techniques might be enough to make the difference. So that’s what I’m working on; more soon.
Mining your model uncertainty for concretely useful actions just doesn’t seem like the kind of thing that could work. I look forward to reading about it anyway.
By “retraining or patching”, I’m not talking about Zvi’s most forbidden technique. I’m talking about any kind of update to the training procedure, including making a different training environment. When you don’t have a deep understanding of the problems and the distribution shifts, iteration on the training environment can just as easily hide problems as the most forbidden technique.
AI-2027 was wrong to imply that that that strategy could plausibly work, it’s alchemy.
You question marked this section
I mean this to include things like LLM experiments that show that they plot to kill people to avoid shutdown, or express preferences that are alarming in some way, or insert backdoors into code in some situations. At best these are weak analogies for real problems, but studying most ways to make them go away in LLMs won’t help make future AGI safer, even if future AGI is technologically descended from LLMs.
The real problems are caused by self-improvement and large-scale online learning and massively increased options. The problems people study in LLMs are only weakly related to these.
This seems like it’s building on or inspired by work you’ve done? Or was this team interested in embeddedness and reflective oracles for other reasons?
It’s ridiculously long (which is great, I’ll read through it when I get a chance), do you have any pointers to sections that you think have particularly valuable insights?
My argument was that there were several of “risk factors” that stack. I agree that each one isn’t overwhelmingly strong.
I prefer not to be rude. Are you sure it’s not just that I’m confidently wrong? If I was disagreeing in the same tone with e.g. Yampolskiy’s argument for high confidence AI doom, would this still come across as rude to you?