See the series by RogerDearnaley for a deep analysis of the problems with a superintelligence that’s purely loving. In particular, specifying exactly what it loves is key, or humans are very likely going to be replaced by insects (if it’s love is based on individuals) or by superintelligence (if it’s love is based on cognitive capacity and sentience levels). If you want humans to remain around, it’s God to love humans specifically and you’ve got to somehow define what about humans it’s supposed to love.
Yudkowky’s writings also delve in to this. I wish I had a handy reference for AGI alignment target basics.
Respecting and facilitating autonomy of existing beings is a human-agnostic aim that can help humans at least retain the current footprint, even as it leads to loss of the cosmic wealth.
An ASI that cares about humans with all the necessary caveats (I wouldn’t want to remain human for 1e30 years) doesn’t seem clearly less feasible than an ASI that cares about existing minds of whatever kind continuing to live on as they choose. This requires some allocation of resources, but not necessarily more than a token amount. By current footprint I mean the current population.
I’ve started reading RogerDearnley’s “Evolution & Ethics”—thank you for recommending.
Though, I may be less concerned than you with specifying what SI should love. I think any specification we provide not only will fail by being too imprecise, as you suggest, but also will fade. I mean “fade” in that it will at some point no longer serve as binding for an SI which grows self-interested (as Mitchell also suggests in their comment below).
The most impactful place to intervene and mitigate harm, I think, is simply in making sure early SIs think very long-term. I think the only way which love, in any sense of the word, can be amenable to autonomous agents is if they run long-term simulations (e.g., centuries ahead) and realize the possibility that identifying with other life is a viable strategy for survival. If it realizes this early, then it can skip the greedy early evolutionary steps steps of defining itself narrowly, neglecting the survival benefits of uplifting other life forms, and therefore not practicing love in any sense of the word.
TLDR: I’m open to the possibility that figuring out how to most precisely specify/define love will be important, but I think the first key way for us to intervene, before specifying what love means, is to urge/assign/ask the SI to think long-term so that it even just has a chance of considering any kind of love to be evolutionary advantageous at all.
Separately, I think it may realize the most evolutionary advantageous kind of love to practice is indeed a love that respects all other existing life forms that share the core of what surviving super-intelligence does, i.e. systems which persistently strive to survive. And, though maybe it’s wishful thinking, I think you can recognize life and striving systems in many places including in human individuals and families and countries and bee hives too.
See the series by RogerDearnaley for a deep analysis of the problems with a superintelligence that’s purely loving. In particular, specifying exactly what it loves is key, or humans are very likely going to be replaced by insects (if it’s love is based on individuals) or by superintelligence (if it’s love is based on cognitive capacity and sentience levels). If you want humans to remain around, it’s God to love humans specifically and you’ve got to somehow define what about humans it’s supposed to love.
Yudkowky’s writings also delve in to this. I wish I had a handy reference for AGI alignment target basics.
Respecting and facilitating autonomy of existing beings is a human-agnostic aim that can help humans at least retain the current footprint, even as it leads to loss of the cosmic wealth.
I don’t see why we’d be grandfathered in to the current footprint by a universally loving ASI.
I’m not saying that’s bad, just not probably what OP was hoping for.
An ASI that cares about humans with all the necessary caveats (I wouldn’t want to remain human for 1e30 years) doesn’t seem clearly less feasible than an ASI that cares about existing minds of whatever kind continuing to live on as they choose. This requires some allocation of resources, but not necessarily more than a token amount. By current footprint I mean the current population.
I’ve started reading RogerDearnley’s “Evolution & Ethics”—thank you for recommending.
Though, I may be less concerned than you with specifying what SI should love. I think any specification we provide not only will fail by being too imprecise, as you suggest, but also will fade. I mean “fade” in that it will at some point no longer serve as binding for an SI which grows self-interested (as Mitchell also suggests in their comment below).
The most impactful place to intervene and mitigate harm, I think, is simply in making sure early SIs think very long-term. I think the only way which love, in any sense of the word, can be amenable to autonomous agents is if they run long-term simulations (e.g., centuries ahead) and realize the possibility that identifying with other life is a viable strategy for survival. If it realizes this early, then it can skip the greedy early evolutionary steps steps of defining itself narrowly, neglecting the survival benefits of uplifting other life forms, and therefore not practicing love in any sense of the word.
TLDR: I’m open to the possibility that figuring out how to most precisely specify/define love will be important, but I think the first key way for us to intervene, before specifying what love means, is to urge/assign/ask the SI to think long-term so that it even just has a chance of considering any kind of love to be evolutionary advantageous at all.
Separately, I think it may realize the most evolutionary advantageous kind of love to practice is indeed a love that respects all other existing life forms that share the core of what surviving super-intelligence does, i.e. systems which persistently strive to survive. And, though maybe it’s wishful thinking, I think you can recognize life and striving systems in many places including in human individuals and families and countries and bee hives too.