ofI think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren’t that bad, so you can just build it to double the strength it would need on the lake and you’re good to go.
Yeah this does seem to be how a lot of people are thinking about it. I think the way to resolve this is to have people meditate on the non-analogy distribution shifts, but yeah doing this well requires having at least one somewhat-detailed model of intelligence, which isn’t that common.
I don’t think there’s much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail.
I think there’s a lot of evidence that AI builders don’t know what they’re doing in the relevant ways, and this evidence will likely get stronger and more widely acknowledged over time (as deployment stakes and capabilities make the occasional OOD weirdness more obvious). I’m sure the game of training against each embarrassing behaviour as it comes up will continue, but I hope that some will notice the pattern and extrapolate. It’s not that I’m wanting arguments to convince people, I think reality can convince people and good clear arguments just smooth the process.
The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it
I think in the particular case of this post, it would be obvious that humans don’t have the corrigibility property and are equally susceptible to distribution shifts.
That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.
Eh, all arguments are equal if you don’t think them through. I think it’s better to think of this kind of argument as setting the stage for the future, rather than winning over large groups of people right now (who you’re assuming aren’t even evaluating the arguments). There are possible futures where world leaders are deciding on a course of action, where a background fact is that it has become extremely obvious that we couldn’t win against a rogue AI. And many other potential futures where different things have become obvious and widely known. Many should provide evidence about alignment competence, but even the ones that don’t directly provide evidence here will provide plenty of motivation to think really carefully. And that plays to the benefit of careful and correct arguments.
combined with “ummm I’d like to not be permanently jobless” might make a powerful political movement for slowing/stopping. But that’s an entirely separate project from arguing the case for alignment difficulty on its merits.
I’m hoping that this, while somewhat misguided, might give politicians more leeway to make the correct choices.
I wonder if some well-considered optimists are assuming we’ll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they’re deployed. If I was confident developers would at least try to do that, I’d be a good bit more optimistic.
I don’t see how “improve training” is an available option even in theory.
Yeah this does seem to be how a lot of people are thinking about it. I think the way to resolve this is to have people meditate on the non-analogy distribution shifts, but yeah doing this well requires having at least one somewhat-detailed model of intelligence, which isn’t that common.
I think there’s a lot of evidence that AI builders don’t know what they’re doing in the relevant ways, and this evidence will likely get stronger and more widely acknowledged over time (as deployment stakes and capabilities make the occasional OOD weirdness more obvious). I’m sure the game of training against each embarrassing behaviour as it comes up will continue, but I hope that some will notice the pattern and extrapolate. It’s not that I’m wanting arguments to convince people, I think reality can convince people and good clear arguments just smooth the process.
I think in the particular case of this post, it would be obvious that humans don’t have the corrigibility property and are equally susceptible to distribution shifts.
Eh, all arguments are equal if you don’t think them through. I think it’s better to think of this kind of argument as setting the stage for the future, rather than winning over large groups of people right now (who you’re assuming aren’t even evaluating the arguments). There are possible futures where world leaders are deciding on a course of action, where a background fact is that it has become extremely obvious that we couldn’t win against a rogue AI. And many other potential futures where different things have become obvious and widely known. Many should provide evidence about alignment competence, but even the ones that don’t directly provide evidence here will provide plenty of motivation to think really carefully. And that plays to the benefit of careful and correct arguments.
I’m hoping that this, while somewhat misguided, might give politicians more leeway to make the correct choices.
I don’t see how “improve training” is an available option even in theory.