Thanks! I haven’t watched, but I appreciated having something to give me the gist!
Hotz was allowed to drive discussion. In debate terms, he was the con side, raising challenges, while Yudkowsky was the pro side defending a fixed position.
This always seems to be the framing which seems unbelievably stupid given the stakes on each side of the argument. Still, it seems to be the default; I’m guessing this is status quo bias and the historical tendency of everything to stay relatively the same year by year (less so once technology really started happening). I think AI safety outreach needs to break out of this framing or it’s playing a losing game. I feel like, in terms of public communication, whoever’s playing defense has mostly already lost.
The idea that poking a single whole in EY’s reasoning is also a really broken norm around these discussions that we are going to have to move past if we want effective public communication. In particular, the combination of “tell me exactly what an ASI would do” and “if anything you say sounds implausible, then AI is safe” is just ridiculous. Any conversation implicitly operating on that basis is operating in bad faith and borderline not worth having. It’s not a fair framing of the situation.
9. Hotz closes with a vision of ASIs running amok
What a ridiculous thing to be okay with?! Is this representative of his actual stance? Is this stance taken seriously by anyone besides him?
not going to rely on a given argument or pathway because although it was true it would strain credulity. This is a tricky balance, on the whole we likely need more of this.
I take it this means not using certain implausible seeming examples? I agree that we could stand to move away from the “understand the lesson behind this implausible seeming toy example”-style argumentation and more towards an emphasis on something like “a lot of factors point to doom and even very clever people can’t figure out how to make things safe”.
I think it matters that most of the “technical arguments” point strongly towards doom, but I think it’s a mistake for AI safety advocates to try to do all of the work of laying out and defending technical arguments when it comes to public facing communication/debate. If you’re trying to give all the complicated reasons why doom is a real possibility, then you’re implicitly taking on a huge burden of proof and letting your opponent get away with doing nothing more than cause confusion and nitpick.
Like, imagine having to explain general relativity in a debate to an audience who has never heard about it. Your opponent continuously just stops you and disagrees with you; maybe misuses a term here and there and then at the end the debate is judged by whether the audience is convinced that your theory of physics is correct. It just seems like playing a losing game for no reason.
Again, I didn’t see this and I’m sure EY handled himself fine, I just think there’s a lot of room for improvement in the general rhythm that these sorts of discussions tend to fall into.
I think it is okay for AI safety advocates to lay out the groundwork, maybe make a few big-picture arguments, maybe talk about expert opinion (since that alone is enough to perk most sane people’s ears and shift some of the burden of proof), and then mostly let their opponents do the work of stumbling through the briars of technical argumentation if they still want to nitpick whatever thought experiment. In general, a leaner case just argues better and is more easily understood. Thus, I think it’s better to argue the general case than to attempt the standard shuffle of a dozen different analogies; especially when time/audience attention is more acutely limited.
The idea that poking a single whole in EY’s reasoning is also a really broken norm around these discussions that we are going to have to move past if we want effective public communication. In particular, the combination of “tell me exactly what an ASI would do” and “if anything you say sounds implausible, then AI is safe”
Remember that this a three way debate: AI safe; AI causes finite; containable problems; AI kills (almost) everybody. The most extreme scenario is conjunctive because it requires AI with goals; goal stability; rapid self improvement (foom); and means. So nitpicking one stage of Foom Doom actually does refute it, even if it has no impact on the.middle of the road position.
I disagree that rapid self improvement and goal stability are load-bearing arguments here. Even goals are not strictly, 100% required. If we build something with the means to kill everyone, then we should be worried about it. If it has goals that cannot be directed of predicted, then we should be VERY worried about it.
What are the steps? Are we deliberately building a superintelligence with the goal of killing us all? If not, where do the motivation and ability come from?
For me, ability = capability = means. This is one of the two arguments that I said were load bearing. Where will it come from? Well, we are specifically trying to build the most capable systems possible.
Motivation (ie goals) is not actually strictly required. However, there are reasons to think that an AGI could have goals that are not aligned with most humans. The most fundamental is instrumental convergence.
Note that my original comment was not making this case. It was just a meta discussion about what it would take to refute Eliezer’s argument.
Thanks! I haven’t watched, but I appreciated having something to give me the gist!
This always seems to be the framing which seems unbelievably stupid given the stakes on each side of the argument. Still, it seems to be the default; I’m guessing this is status quo bias and the historical tendency of everything to stay relatively the same year by year (less so once technology really started happening). I think AI safety outreach needs to break out of this framing or it’s playing a losing game. I feel like, in terms of public communication, whoever’s playing defense has mostly already lost.
The idea that poking a single whole in EY’s reasoning is also a really broken norm around these discussions that we are going to have to move past if we want effective public communication. In particular, the combination of “tell me exactly what an ASI would do” and “if anything you say sounds implausible, then AI is safe” is just ridiculous. Any conversation implicitly operating on that basis is operating in bad faith and borderline not worth having. It’s not a fair framing of the situation.
What a ridiculous thing to be okay with?! Is this representative of his actual stance? Is this stance taken seriously by anyone besides him?
I take it this means not using certain implausible seeming examples? I agree that we could stand to move away from the “understand the lesson behind this implausible seeming toy example”-style argumentation and more towards an emphasis on something like “a lot of factors point to doom and even very clever people can’t figure out how to make things safe”.
I think it matters that most of the “technical arguments” point strongly towards doom, but I think it’s a mistake for AI safety advocates to try to do all of the work of laying out and defending technical arguments when it comes to public facing communication/debate. If you’re trying to give all the complicated reasons why doom is a real possibility, then you’re implicitly taking on a huge burden of proof and letting your opponent get away with doing nothing more than cause confusion and nitpick.
Like, imagine having to explain general relativity in a debate to an audience who has never heard about it. Your opponent continuously just stops you and disagrees with you; maybe misuses a term here and there and then at the end the debate is judged by whether the audience is convinced that your theory of physics is correct. It just seems like playing a losing game for no reason.
Again, I didn’t see this and I’m sure EY handled himself fine, I just think there’s a lot of room for improvement in the general rhythm that these sorts of discussions tend to fall into.
I think it is okay for AI safety advocates to lay out the groundwork, maybe make a few big-picture arguments, maybe talk about expert opinion (since that alone is enough to perk most sane people’s ears and shift some of the burden of proof), and then mostly let their opponents do the work of stumbling through the briars of technical argumentation if they still want to nitpick whatever thought experiment. In general, a leaner case just argues better and is more easily understood. Thus, I think it’s better to argue the general case than to attempt the standard shuffle of a dozen different analogies; especially when time/audience attention is more acutely limited.
Remember that this a three way debate: AI safe; AI causes finite; containable problems; AI kills (almost) everybody. The most extreme scenario is conjunctive because it requires AI with goals; goal stability; rapid self improvement (foom); and means. So nitpicking one stage of Foom Doom actually does refute it, even if it has no impact on the.middle of the road position.
I disagree that rapid self improvement and goal stability are load-bearing arguments here. Even goals are not strictly, 100% required. If we build something with the means to kill everyone, then we should be worried about it. If it has goals that cannot be directed of predicted, then we should be VERY worried about it.
What are the steps? Are we deliberately building a superintelligence with the goal of killing us all? If not, where do the motivation and ability come from?
For me, ability = capability = means. This is one of the two arguments that I said were load bearing. Where will it come from? Well, we are specifically trying to build the most capable systems possible.
Motivation (ie goals) is not actually strictly required. However, there are reasons to think that an AGI could have goals that are not aligned with most humans. The most fundamental is instrumental convergence.
Note that my original comment was not making this case. It was just a meta discussion about what it would take to refute Eliezer’s argument.