Eliezer, sometimes in a conversation one needs a rapid back and forth, often to clarify what exactly people mean by things they say. In such a situation a format like the one we are using, long daily blog posts, can work particularly badly. In my last post I was trying in part to get you to become clearer about what you meant by what you now call a “winner take all” tech, especially to place it on a continuum with other familiar techs. (And once we are clear on what it means, then I want arguments suggesting that an AI transition would be such a thing.) I suggested talking about outcome variance induced by a transition. If you now want to use that phrase to denote “a local entity tends to end up with the option of becoming one kind of Bostromian singleton”, then we need new terms to refer to the “properties of the technology landscape,” that might lead to such an option.
I am certainly not assuming it is impossible to be “friendly” though I can’t be sure without knowing better what that means. I agree that it is not obvious that we would not want a singleton, if we could choose the sort we wanted. But I am, as you note, quite wary of the sort of total war that might be required to create a singleton. But before we can choose among options we need to get clearer on what the options are.
Carl and Roko, I really wasn’t trying to lay out a moral position, though I was expressing mild horror at encouraging total war, a horror I expected (incorrectly it seems) would be widely shared.
There are people who don’t want to have a total war because of what it would cost them, and then there are people who do want to have a total war so that they can go out and win it.
Eliezer, sometimes in a conversation one needs a rapid back and forth, often to clarify what exactly people mean by things they say. In such a situation a format like the one we are using, long daily blog posts, can work particularly badly. In my last post I was trying in part to get you to become clearer about what you meant by what you now call a “winner take all” tech, especially to place it on a continuum with other familiar techs. (And once we are clear on what it means, then I want arguments suggesting that an AI transition would be such a thing.) I suggested talking about outcome variance induced by a transition. If you now want to use that phrase to denote “a local entity tends to end up with the option of becoming one kind of Bostromian singleton”, then we need new terms to refer to the “properties of the technology landscape,” that might lead to such an option.
I am certainly not assuming it is impossible to be “friendly” though I can’t be sure without knowing better what that means. I agree that it is not obvious that we would not want a singleton, if we could choose the sort we wanted. But I am, as you note, quite wary of the sort of total war that might be required to create a singleton. But before we can choose among options we need to get clearer on what the options are.
Carl and Roko, I really wasn’t trying to lay out a moral position, though I was expressing mild horror at encouraging total war, a horror I expected (incorrectly it seems) would be widely shared.
There are people who don’t want to have a total war because of what it would cost them, and then there are people who do want to have a total war so that they can go out and win it.