Definitely not, for example, norms to espouse in argumentation (and tbf nowhere does this post claim to be a model for argument, except maybe implicitly under some circumstances).
Yet there’s something to it.
There’s a game of Chicken arising out of the shared responsibility to generate (counter)arguments. If Eliezer commits to Straight, ie. refuses to instantiate the core argument over and over again (either explicitly, by saying “you need to come up with the generator” or implicitly, by refusing to engage with a “please stop.”), then the other will be incentivized to Swerve, ie. put some effort into coming up with their own arguments and thereby stumble upon the generator.
This isn’t my preferred way of coordinating on games of Chicken, since it is somewhat violent and not really coordination. My preferred way is to proportionately share the price of anarchy, which can be loosely estimated with some honest explicitness. But that’s what (part of) this post is, a very explicit presentation of the consequences!
So I recoil less. It feels inviting instead, about a real human issue in reasoning. And bold, given all the possible ways to mischaracterize it as “Eliezer says ‘shut up’ to quantitative models because he has a pet theory about AGI doom”.
But is this an important caveat to the fifth virtue, at least in simulated dialogue? That remains open for me.
The “shut up”s and “please stop”s are jarring.
Definitely not, for example, norms to espouse in argumentation (and tbf nowhere does this post claim to be a model for argument, except maybe implicitly under some circumstances).
Yet there’s something to it.
There’s a game of Chicken arising out of the shared responsibility to generate (counter)arguments. If Eliezer commits to Straight, ie. refuses to instantiate the core argument over and over again (either explicitly, by saying “you need to come up with the generator” or implicitly, by refusing to engage with a “please stop.”), then the other will be incentivized to Swerve, ie. put some effort into coming up with their own arguments and thereby stumble upon the generator.
This isn’t my preferred way of coordinating on games of Chicken, since it is somewhat violent and not really coordination. My preferred way is to proportionately share the price of anarchy, which can be loosely estimated with some honest explicitness. But that’s what (part of) this post is, a very explicit presentation of the consequences!
So I recoil less. It feels inviting instead, about a real human issue in reasoning. And bold, given all the possible ways to mischaracterize it as “Eliezer says ‘shut up’ to quantitative models because he has a pet theory about AGI doom”.
But is this an important caveat to the fifth virtue, at least in simulated dialogue? That remains open for me.