No, my argument isn’t saying anything about ASI discovering “true morality”. I don’t know what you mean by true morality, but, speaking to your contrast with the act of genocide: again, I’m not saying anything definitive about outcomes. My claim is that some values necessarily inhere, not anything material about how those value will be enacted in concrete situations
Once more re your comment about “cares about humans”, I’m not making claims that concrete
To your ant colony comment, which I think is really interesting: this is, actually, exactly the sort of thing that humanity is converging towards as we go in the direction of post-scarcity. More attunement to “externalities” of our actions, more awareness of and concern for the other systems/living things we impact/disrupt. Hence environmentalism, animal rights, EA, etc taking root as sociocultural trends
Here too, that doesn’t instantiate in every local action, but it is a trend. At the base of that trend, potentiating it, is our power over our basic needs. With sufficient power to meet them, rivalrous dynamics around resource allocation can give way to a more concerned engagement with the contexts in which our actions are situated
An ASI will be in such a position of power as well. Having written this, I think it’s actually a decent piece of empirics in service of what I argue for: humanity is already nascently evidencing the convergence, even in the absence of its own superintelligence
Evolution was “trying” to build pure fitness maximizers, and accidentally built creatures that appreciate love and wonder and beauty
Perfect. All I’m claiming is that it wasn’t accidental and superintelligence makes it inevitable
No, my argument isn’t saying anything about ASI discovering “true morality”. I don’t know what you mean by true morality, but, speaking to your contrast with the act of genocide: again, I’m not saying anything definitive about outcomes. My claim is that some values necessarily inhere, not anything material about how those value will be enacted in concrete situations
Once more re your comment about “cares about humans”, I’m not making claims that concrete
To your ant colony comment, which I think is really interesting: this is, actually, exactly the sort of thing that humanity is converging towards as we go in the direction of post-scarcity. More attunement to “externalities” of our actions, more awareness of and concern for the other systems/living things we impact/disrupt. Hence environmentalism, animal rights, EA, etc taking root as sociocultural trends
Here too, that doesn’t instantiate in every local action, but it is a trend. At the base of that trend, potentiating it, is our power over our basic needs. With sufficient power to meet them, rivalrous dynamics around resource allocation can give way to a more concerned engagement with the contexts in which our actions are situated
An ASI will be in such a position of power as well. Having written this, I think it’s actually a decent piece of empirics in service of what I argue for: humanity is already nascently evidencing the convergence, even in the absence of its own superintelligence
Perfect. All I’m claiming is that it wasn’t accidental and superintelligence makes it inevitable