Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
One example of AIs that would count as good successors: ems. Creating a society of highly-accurate human brain emulations would constitute a good successor AI, since they would by definition share human values,
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
Optimistically, if they succeed and find that our value system is algorithmically simple, creating good successor AI might be as simple as copying that algorithm to silicon
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.