I think you’re strawmanning a bit here. Let’s take your example of a forest.
“Okay”, says Forest Succesionist, “but for all those things, we can just build plastic plants, maybe with advanced nanotech, that have those properties but more. Your attachment to real forests is just being a backward luddite hominid. Hell, your stance is basically a moral atrocity—you’re depriving countless future generations from experiencing infinitely plantier plants.”
Obviously the forest will be worse if the plants are made out of plastic! How hard did you try to think of what an actually-better superforest might look like?
If you live in the United States, your local forest has probably gone through several cycles of logging within the last few centuries, and looks very different from how it was before European colonists arrived in North America. On the West coast, 96% of old-growth redwood forests have been logged. On the East coast, less than 1% of the original forests remain, and one of their most important tree species, the American chestnut, is functionally extinct because of an invasive fungus. Fire suppression policies also change the composition of forests. When is the last time you saw a tree this large?
This is an American chestnut. Trees of this size used to be common in the forests of the Eastern United States.
Compared to the American forests of 500 years ago, your local forest probably has much smaller trees, much less biodiversity, and much more underbrush.
Do you approve of the discontinuous logging that produced the forests we have now? If not, why would you disapprove of a hypothetical discontinuous process that could bring back the forests we used to have? You could have larger trees, more animal species, and the forest could be easier to navigate.
We can go even further, and imagine some sort of superforest. The trees would be the size of coastal redwoods, and like redwoods, they would support life not just in the ground but also in the air—soil made from decaying plant matter collects on the branches of these trees and supports plants, worms, and slugs that live their entire lives 200 feet off the ground. The trees naturally grow handholds, so they are easily climbable, and walkways connecting them to each other, so you can explore the forest from off the ground. Some of the trees produce peaches, others produce walnuts. The forest would have fireflies and glowing mushrooms, so that you could navigate it at night without artificial lights. There would be lots of small streams and ponds with plenty of aquatic life. There would be no ticks, mosquitos, or other insects that bite humans, but plenty of ants, bees, worms, spiders, etc. The forest would have moose, wolves, bears and mountain lions (dinosaurs and giant sloths, anyone?), but they would never attack humans.
Obviously nothing like this occurs naturally, but it would be possible with advanced technology. I think this forest would be pretty awesome! I wouldn’t want to replace all forests on earth with this, since there is value in variety, but I would be elated if this forest could exist somewhere, and there are many lower-quality forests that I would be happy to replace with this. I also think that for any forest on earth there exists some improvement that could make it even better.
You argue against maximizing some definition of value like entropy, negentropy, kolmogorov complexity, or some formalization of intelligence, and I agree. If your math says a uniform gas cloud or a giant random matrix maximizes value, you should be extremely skeptical! I also think there is a version of successionism that is not like this. For example, “replace every human with a copy of Claude 3 opus” is a form of successionism that isn’t maximizing something to infinity. Someone who prefers this is crazy, because this system is not even self-sustaining, but I can imagine a point in the future where people think “My instance of Claude 17 is a really compassionate, intelligent, poetic, agentic person, and I prefer him to myself or any of my hypothetical children. I’ll let him continue as my descendant instead of having biological children.”. In this scenario, people’s desire for AIs to replace us is based on normal intuitions, not some out-of-control desire for optimization or moral abstraction. If you think you will correctly believe this in 50 years after interacting with future AIs (and think you won’t be tricked, addicted, etc.), why not update all the way there right now?
I also think your perspective is a bit self-defeating.
The generalized antidote is a certain territoriality. You’re allowed – even obligated – to protect yourself. You’re allowed to take ownership of your slice of the universe and resist assaults on it. You’re allowed to have goals just because you have them, without justifying them in “objective” language. In contrast to the cosmo-eugenicist attitude inherent in successionism, you and your lineage should be allowed to both continue living and wielding power even if someone somewhere “proved” something else was “more optimal”.
What if you’ve “proved” in this post that individual humans existing and holding power in the future is “more optimal” than AI takeover, but founders and employees of AI labs disagree, and continue AI development? Should they refuse to be convinced by your argument, and protect their business model and worldview? Why shouldn’t the AIs adopt territoriality over the whole lightcone, and take it themselves, regardless of what humans say is morally optimal? Even if you argue for some sort of non-aggression principle, there are versions of gradual disempowerment where nobody’s property rights are violated. If you think each entity should pursue their own idiosyncratic desires regardless of grand moral principles, then this predictably results in succession anyways—first, AI researchers achieve their goals of creating powerful AI, regardless of whether this is objectively correct, then the AI achieves its goal of taking over the universe.
I think if you decline to answer the question of what determines value in the universe and retreat to “all desires are equally valid, fulfill yours”, your “might makes right” philosophy (where humans have the “might” of currently existing) advocates successionism anyways. If you do answer the question of what determines value, now you’ve opened yourself up to replacement of humans being objectively optimal. Of course there are many obviously bad versions of successionism, and we should think very carefully before making irreversible changes, but I think it’s illegitimate to rule out all possible forms of successionism right now.
I think you’re strawmanning a bit here. Let’s take your example of a forest.
Obviously the forest will be worse if the plants are made out of plastic! How hard did you try to think of what an actually-better superforest might look like?
If you live in the United States, your local forest has probably gone through several cycles of logging within the last few centuries, and looks very different from how it was before European colonists arrived in North America. On the West coast, 96% of old-growth redwood forests have been logged. On the East coast, less than 1% of the original forests remain, and one of their most important tree species, the American chestnut, is functionally extinct because of an invasive fungus. Fire suppression policies also change the composition of forests. When is the last time you saw a tree this large?
This is an American chestnut. Trees of this size used to be common in the forests of the Eastern United States.
Compared to the American forests of 500 years ago, your local forest probably has much smaller trees, much less biodiversity, and much more underbrush.
Do you approve of the discontinuous logging that produced the forests we have now? If not, why would you disapprove of a hypothetical discontinuous process that could bring back the forests we used to have? You could have larger trees, more animal species, and the forest could be easier to navigate.
We can go even further, and imagine some sort of superforest. The trees would be the size of coastal redwoods, and like redwoods, they would support life not just in the ground but also in the air—soil made from decaying plant matter collects on the branches of these trees and supports plants, worms, and slugs that live their entire lives 200 feet off the ground. The trees naturally grow handholds, so they are easily climbable, and walkways connecting them to each other, so you can explore the forest from off the ground. Some of the trees produce peaches, others produce walnuts. The forest would have fireflies and glowing mushrooms, so that you could navigate it at night without artificial lights. There would be lots of small streams and ponds with plenty of aquatic life. There would be no ticks, mosquitos, or other insects that bite humans, but plenty of ants, bees, worms, spiders, etc. The forest would have moose, wolves, bears and mountain lions (dinosaurs and giant sloths, anyone?), but they would never attack humans.
Obviously nothing like this occurs naturally, but it would be possible with advanced technology. I think this forest would be pretty awesome! I wouldn’t want to replace all forests on earth with this, since there is value in variety, but I would be elated if this forest could exist somewhere, and there are many lower-quality forests that I would be happy to replace with this. I also think that for any forest on earth there exists some improvement that could make it even better.
You argue against maximizing some definition of value like entropy, negentropy, kolmogorov complexity, or some formalization of intelligence, and I agree. If your math says a uniform gas cloud or a giant random matrix maximizes value, you should be extremely skeptical! I also think there is a version of successionism that is not like this. For example, “replace every human with a copy of Claude 3 opus” is a form of successionism that isn’t maximizing something to infinity. Someone who prefers this is crazy, because this system is not even self-sustaining, but I can imagine a point in the future where people think “My instance of Claude 17 is a really compassionate, intelligent, poetic, agentic person, and I prefer him to myself or any of my hypothetical children. I’ll let him continue as my descendant instead of having biological children.”. In this scenario, people’s desire for AIs to replace us is based on normal intuitions, not some out-of-control desire for optimization or moral abstraction. If you think you will correctly believe this in 50 years after interacting with future AIs (and think you won’t be tricked, addicted, etc.), why not update all the way there right now?
I also think your perspective is a bit self-defeating.
What if you’ve “proved” in this post that individual humans existing and holding power in the future is “more optimal” than AI takeover, but founders and employees of AI labs disagree, and continue AI development? Should they refuse to be convinced by your argument, and protect their business model and worldview? Why shouldn’t the AIs adopt territoriality over the whole lightcone, and take it themselves, regardless of what humans say is morally optimal? Even if you argue for some sort of non-aggression principle, there are versions of gradual disempowerment where nobody’s property rights are violated. If you think each entity should pursue their own idiosyncratic desires regardless of grand moral principles, then this predictably results in succession anyways—first, AI researchers achieve their goals of creating powerful AI, regardless of whether this is objectively correct, then the AI achieves its goal of taking over the universe.
I think if you decline to answer the question of what determines value in the universe and retreat to “all desires are equally valid, fulfill yours”, your “might makes right” philosophy (where humans have the “might” of currently existing) advocates successionism anyways. If you do answer the question of what determines value, now you’ve opened yourself up to replacement of humans being objectively optimal. Of course there are many obviously bad versions of successionism, and we should think very carefully before making irreversible changes, but I think it’s illegitimate to rule out all possible forms of successionism right now.