Agreed on all points. Including that you writing a more detailed version more for the prosaic crowd isn’t probably the best next step. That’s what I was trying to do in LAMRAG, and it was no more successful than this. That’s despite me starting much closer to the standard prosaic alignment/LLM-based model of AGI internals.
I think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren’t that bad, so you can just build it to double the strength it would need on the lake and you’re good to go.
My vague take on this is that what we’re doing in training now is a far cry from what a nascent AGI will see in deployment, so the metaphor holds. I wonder if some well-considered optimists are assuming we’ll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they’re deployed.
If I was confident developers would at least try to do that, I’d be a good bit more optimistic.
Just some thoughts.
More thoughts have arisen. Whatever the next step is, I think this work of clarifying and specifying the arguments, could be critical. I don’t think there’s much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail. The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it, and hey Claude seems to be doing pretty well, so a future better version of it should be fine. That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.
OTOH I do think development might be deployed when the public gets involved. They have less motivated reasoning toward assuming alignment is easy, so the obvious intuition “if nobody knows how dangerous it is, we should stop!” combined with “ummm I’d like to not be permanently jobless” might make a powerful political movement for slowing/stopping. But that’s an entirely separate project from arguing the case for alignment difficulty on its merits. And I don’t know the first thing about public relations/marketing.
ofI think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren’t that bad, so you can just build it to double the strength it would need on the lake and you’re good to go.
Yeah this does seem to be how a lot of people are thinking about it. I think the way to resolve this is to have people meditate on the non-analogy distribution shifts, but yeah doing this well requires having at least one somewhat-detailed model of intelligence, which isn’t that common.
I don’t think there’s much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail.
I think there’s a lot of evidence that AI builders don’t know what they’re doing in the relevant ways, and this evidence will likely get stronger and more widely acknowledged over time (as deployment stakes and capabilities make the occasional OOD weirdness more obvious). I’m sure the game of training against each embarrassing behaviour as it comes up will continue, but I hope that some will notice the pattern and extrapolate. It’s not that I’m wanting arguments to convince people, I think reality can convince people and good clear arguments just smooth the process.
The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it
I think in the particular case of this post, it would be obvious that humans don’t have the corrigibility property and are equally susceptible to distribution shifts.
That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.
Eh, all arguments are equal if you don’t think them through. I think it’s better to think of this kind of argument as setting the stage for the future, rather than winning over large groups of people right now (who you’re assuming aren’t even evaluating the arguments). There are possible futures where world leaders are deciding on a course of action, where a background fact is that it has become extremely obvious that we couldn’t win against a rogue AI. And many other potential futures where different things have become obvious and widely known. Many should provide evidence about alignment competence, but even the ones that don’t directly provide evidence here will provide plenty of motivation to think really carefully. And that plays to the benefit of careful and correct arguments.
combined with “ummm I’d like to not be permanently jobless” might make a powerful political movement for slowing/stopping. But that’s an entirely separate project from arguing the case for alignment difficulty on its merits.
I’m hoping that this, while somewhat misguided, might give politicians more leeway to make the correct choices.
I wonder if some well-considered optimists are assuming we’ll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they’re deployed. If I was confident developers would at least try to do that, I’d be a good bit more optimistic.
I don’t see how “improve training” is an available option even in theory.
Agreed on all points. Including that you writing a more detailed version more for the prosaic crowd isn’t probably the best next step. That’s what I was trying to do in LAMRAG, and it was no more successful than this. That’s despite me starting much closer to the standard prosaic alignment/LLM-based model of AGI internals.
I think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren’t that bad, so you can just build it to double the strength it would need on the lake and you’re good to go.
My vague take on this is that what we’re doing in training now is a far cry from what a nascent AGI will see in deployment, so the metaphor holds. I wonder if some well-considered optimists are assuming we’ll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they’re deployed.
If I was confident developers would at least try to do that, I’d be a good bit more optimistic.
Just some thoughts.
More thoughts have arisen. Whatever the next step is, I think this work of clarifying and specifying the arguments, could be critical. I don’t think there’s much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail. The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it, and hey Claude seems to be doing pretty well, so a future better version of it should be fine. That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.
OTOH I do think development might be deployed when the public gets involved. They have less motivated reasoning toward assuming alignment is easy, so the obvious intuition “if nobody knows how dangerous it is, we should stop!” combined with “ummm I’d like to not be permanently jobless” might make a powerful political movement for slowing/stopping. But that’s an entirely separate project from arguing the case for alignment difficulty on its merits. And I don’t know the first thing about public relations/marketing.
Yeah this does seem to be how a lot of people are thinking about it. I think the way to resolve this is to have people meditate on the non-analogy distribution shifts, but yeah doing this well requires having at least one somewhat-detailed model of intelligence, which isn’t that common.
I think there’s a lot of evidence that AI builders don’t know what they’re doing in the relevant ways, and this evidence will likely get stronger and more widely acknowledged over time (as deployment stakes and capabilities make the occasional OOD weirdness more obvious). I’m sure the game of training against each embarrassing behaviour as it comes up will continue, but I hope that some will notice the pattern and extrapolate. It’s not that I’m wanting arguments to convince people, I think reality can convince people and good clear arguments just smooth the process.
I think in the particular case of this post, it would be obvious that humans don’t have the corrigibility property and are equally susceptible to distribution shifts.
Eh, all arguments are equal if you don’t think them through. I think it’s better to think of this kind of argument as setting the stage for the future, rather than winning over large groups of people right now (who you’re assuming aren’t even evaluating the arguments). There are possible futures where world leaders are deciding on a course of action, where a background fact is that it has become extremely obvious that we couldn’t win against a rogue AI. And many other potential futures where different things have become obvious and widely known. Many should provide evidence about alignment competence, but even the ones that don’t directly provide evidence here will provide plenty of motivation to think really carefully. And that plays to the benefit of careful and correct arguments.
I’m hoping that this, while somewhat misguided, might give politicians more leeway to make the correct choices.
I don’t see how “improve training” is an available option even in theory.