Aha, maybe that sentence was ambiguous: “foundational values” was referring to the values that I labelled the ontonormative goods, and by foundational I meant the relation that humanity generally has to them, eg considering them virtuous, aspiring to them big picture/in the limit, etc. I didn’t mean the adjective to apply to how ASI would relate to them a priori
But anyway, my reasoning can be languaged (while staying honest, I think) in terms of instrumental convergent goals: it’s the claim that the set of these contains minimal elements, and that the ontonormative goods are amongst them
The core claim made by IABIED is that the ASI would be unlikely to care anout the humans because some other stimuli would satisfy the AI’s drives better
Yes, agreed that’s the core claim. My thesis is that this may not be true because ASIs wouldn’t distribute uniformly through value-space. I argue for that as being a consequence of just the properties of 1) being superintelligent, and 2) being effective agents in the world. Concretely to what you said, my argument is that what “would satisfy” is constrained in a non-arbitrary way, one that is in fact meaningfully consonant with humanity’s values
To the examples you gave, definitely current AIs are all over the map. They don’t have the defining properties of an ASI, which my argument does rely on, so that’s fine. Maybe I should have been clearer about my non-claim re current AI in the last section
No, my argument isn’t saying anything about ASI discovering “true morality”. I don’t know what you mean by true morality, but, speaking to your contrast with the act of genocide: again, I’m not saying anything definitive about outcomes. My claim is that some values necessarily inhere, not anything material about how those value will be enacted in concrete situations
Once more re your comment about “cares about humans”, I’m not making claims that concrete
To your ant colony comment, which I think is really interesting: this is, actually, exactly the sort of thing that humanity is converging towards as we go in the direction of post-scarcity. More attunement to “externalities” of our actions, more awareness of and concern for the other systems/living things we impact/disrupt. Hence environmentalism, animal rights, EA, etc taking root as sociocultural trends
Here too, that doesn’t instantiate in every local action, but it is a trend. At the base of that trend, potentiating it, is our power over our basic needs. With sufficient power to meet them, rivalrous dynamics around resource allocation can give way to a more concerned engagement with the contexts in which our actions are situated
An ASI will be in such a position of power as well. Having written this, I think it’s actually a decent piece of empirics in service of what I argue for: humanity is already nascently evidencing the convergence, even in the absence of its own superintelligence
Perfect. All I’m claiming is that it wasn’t accidental and superintelligence makes it inevitable