The short answer is yes to both, because of convergent evolution. I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions. Since our descendants, whether biological or something else, will be competing in the same environment, we should expect their behavior to be similar.
So, if assuming convergent evolution:
If valuing paperclip maximization is unlikely for biological descendants, then it’s unlikely for non-biological descendants too. (That addresses your first question.)
In any case, we don’t control the values of our descendants, so the continuation framing isn’t conditioned on their values. (That addresses your second question.)
To be clear, that doesn’t mean I see the long-term future as unchangeable. Two examples:
It still could be the case that we don’t have any long-term descendants at all, for example due to catastrophic asteroid impact.
A decline scenario is also possible, in which our descendants are not flexible enough to respond to the incentive for interstellar colonization, after which civilization declines and eventually ceases to exist.
I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions.
In any case, we don’t control the values of our descendants, so the continuation framing isn’t conditioned on their values.
The word “similar” does a lot of work here. Russians and Ukrainians throughout history have converged to similar solutions to a whole lot of problems, and yet many Ukrainians prefer literal extinction to Russia exercising significant influence on the values of their descendants. I’d say that for the overwhelming majority of people exercising such influence is a necessary condition for the continuation framing to be applicable. E.g. you mentioned Robin Hanson, who’s certainly a very unorthodox contrarian, but even he, when discussing non-AI issues, voices strong preference for the continuation of the culture he belongs to.
Regarding wars, I don’t think that wars in modern times have much to do with controlling the values of descendants. I’d guess that the main reason people fight defensive wars is to protect their loved ones and communities. And there really isn’t any good reason to fight offensive wars (given current conditions—wasn’t always true), so they are started by leaders who are deluded in some way.
Regarding Robin Hanson, I agree that his views are complicated (which is why I’d be hesitant to classify him as “accel”). The main point of his that I’m referring to is his observation that biological descendants would also have differing values from ours.
I’d guess that the main reason people fight defensive wars is to protect their loved ones and communities.
I agree, but the “cultural genocide” also isn’t an obscure notion.
And there really isn’t any good reason to fight offensive wars
According to you. But what if Russia actually wants paperclips?
biological descendants would also have differing values from ours
Sure, but obviously this isn’t an all-or-nothing proposition, with either biological or artificial descendants, and it’s clear to me that most people aren’t indifferent about where on that spectrum those descendants will end up. Do you disagree with that, or think that only “accels” are indifferent (and in some metaphysical sense “correct”)?
I’m afraid that I’m not following the point of the first line of argument. Yes, people sometimes do pointless destructive things for stupid reasons. Such behavior is in the long-term penalized by selective pressures. More-intelligent descendants would be less likely to engage in such behavior, precisely because they are smarter.
Sure, but obviously this isn’t an all-or-nothing proposition, with either biological or artificial descendants, and it’s clear to me that most people aren’t indifferent about where on that spectrum those descendants will end up. Do you disagree with that, or think that only “accels” are indifferent (and in some metaphysical sense “correct”)?
I doubt that most people think about long-term descendants at all, honestly.
Such behavior is in the long-term penalized by selective pressures.
Which ones? Recursive self-improvement is no longer something that only weird contrarians on obscure blogs talk about, it’s the explicit theory of change of leading multibillion AI corps. They might all be deluded of course, but if they happen to be even slightly correct, machine gods of unimaginable power could be among us in short order, with no evolutionary fairies quick enough to punish their destructive stupidity (even assuming that it actually would be long-term maladaptive, which is far from obvious).
I doubt that most people think about long-term descendants at all, honestly.
You only get to long-term descendants through short-term ones.
If an entity does stupid things, it’s disfavored against competitors that don’t do those stupid things, all else being equal. So it needs to adapt by ceasing the stupid behavior or otherwise lose.
machine gods of unimaginable power could be among us in short order, with no evolutionary fairies quick enough to punish their destructive stupidity
Any assumption of the form “super-intelligent AI will take actions that are super-stupid” is dubious.
Any assumption of the form “super-intelligent AI will take actions that are super-stupid” is dubious.
Clearly. The point is that the actions it takes might seem stupidly destructive only according to humanity’s feeble understanding and parochial values. Something involving extermination of all humans, say. My impression is that the “accel”-endorsed attitude to this is to be a good sport and graciously accept the verdict of natural selection.
That just falls back on the common doomer assumption that “evil is optimal” (as Sutton put it). Sure, if evil is optimal and you have an entity that behaves optimally, it’ll act in evil ways.
But there are good reasons to think that evil is not optimal in current conditions. At least as long as a Dyson sphere has not yet been constructed, there are massive gains available from positive-sum cooperation directed towards technological progress. In these conditions, negative-sum conflict is a stupid waste.
This view, that evil is not optimal, ties back into the continuation framing. After all, you can make a philosophical argument either way. But in the continuation framing, we can ask ourselves whether evil is empirically optimal for humans, which will suggest whether evil is optimal for non-biological descendants (since they continue humanity). And in fact we see evil losing a lot, and not coincidentally—WW2 went the way it did in part because the losing side was evil.
After all, you can make a philosophical argument either way.
Indeed, and what baffles me is that many are extremely sure one way or the other, even though philosophy doesn’t exactly have a track record to inspire such confidence. Of course, this also means that nobody is going to stop building stuff because of philosophical arguments, so we’ll have empirical evidence soon enough...
I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions.
Can you give examples of what you have in mind? Because an obvious counterexample is evolution itself. It has produced an enormous variety of different things. There are instances of convergent evolution: “crab” and “tree” are strategies, not monophyletic taxa. But crabs are not similar to trees in any useful sense. If they are solutions to the same problem, they have in common only that they are solutions to the same problem. This does not make them similar solutions.
One might ask whether evolution is or is not a case of “flexible adaptive systems … exposed to the same problems”, but that would just be a debate over definitions, and you already spoke of “our descendants … competing in the same environment”. That sounds like evolution.
I think I agree with everything you wrote. Yes I’d expect there to be multiple niches available in the future, but I’d expect our descendants to ultimately fill all of them, creating an ecosystem of intelligent life. There is a lot of time available for our descendants to diversify, so it’d be surprising if they didn’t.
How much that diversification process resembles Darwinian evolution, I don’t know. Natural selection still applies, since it’s fundamentally the fact that the life we observe today disproportionately descends from past life that was effective at self-reproduction, and that’s essentially tautological. But Darwinian evolution is undirected, whereas our descendants can intelligently direct their own evolution, and that could conceivably matter. I don’t see why it would prevent diversification, though.
Edit:
Here are some thoughts in reply to your request for examples. Though it’s impossible to know what the niches of the long-term future will be, one idea is that there could be an analogue to “plant” and “animal”. A plant-type civilization would occupy a single stellar system, obtaining resources from it via Dyson sphere, mining, etc. An animal-type civilization could move from star to star, taking resources from the locals (which could be unpleasant for the locals, but not necessarily, as with bees pollinating flowers).
I’d expect both those civilizations to descend from ours, much like how crabs and trees both descend from LUCA.
Do you think that paperclipper-style misalignment is extremely unlikely? Or that the continuation framing is appropriate even then?
The short answer is yes to both, because of convergent evolution. I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions. Since our descendants, whether biological or something else, will be competing in the same environment, we should expect their behavior to be similar.
So, if assuming convergent evolution:
If valuing paperclip maximization is unlikely for biological descendants, then it’s unlikely for non-biological descendants too. (That addresses your first question.)
In any case, we don’t control the values of our descendants, so the continuation framing isn’t conditioned on their values. (That addresses your second question.)
To be clear, that doesn’t mean I see the long-term future as unchangeable. Two examples:
It still could be the case that we don’t have any long-term descendants at all, for example due to catastrophic asteroid impact.
A decline scenario is also possible, in which our descendants are not flexible enough to respond to the incentive for interstellar colonization, after which civilization declines and eventually ceases to exist.
The word “similar” does a lot of work here. Russians and Ukrainians throughout history have converged to similar solutions to a whole lot of problems, and yet many Ukrainians prefer literal extinction to Russia exercising significant influence on the values of their descendants. I’d say that for the overwhelming majority of people exercising such influence is a necessary condition for the continuation framing to be applicable. E.g. you mentioned Robin Hanson, who’s certainly a very unorthodox contrarian, but even he, when discussing non-AI issues, voices strong preference for the continuation of the culture he belongs to.
Regarding wars, I don’t think that wars in modern times have much to do with controlling the values of descendants. I’d guess that the main reason people fight defensive wars is to protect their loved ones and communities. And there really isn’t any good reason to fight offensive wars (given current conditions—wasn’t always true), so they are started by leaders who are deluded in some way.
Regarding Robin Hanson, I agree that his views are complicated (which is why I’d be hesitant to classify him as “accel”). The main point of his that I’m referring to is his observation that biological descendants would also have differing values from ours.
I agree, but the “cultural genocide” also isn’t an obscure notion.
According to you. But what if Russia actually wants paperclips?
Sure, but obviously this isn’t an all-or-nothing proposition, with either biological or artificial descendants, and it’s clear to me that most people aren’t indifferent about where on that spectrum those descendants will end up. Do you disagree with that, or think that only “accels” are indifferent (and in some metaphysical sense “correct”)?
I’m afraid that I’m not following the point of the first line of argument. Yes, people sometimes do pointless destructive things for stupid reasons. Such behavior is in the long-term penalized by selective pressures. More-intelligent descendants would be less likely to engage in such behavior, precisely because they are smarter.
I doubt that most people think about long-term descendants at all, honestly.
Which ones? Recursive self-improvement is no longer something that only weird contrarians on obscure blogs talk about, it’s the explicit theory of change of leading multibillion AI corps. They might all be deluded of course, but if they happen to be even slightly correct, machine gods of unimaginable power could be among us in short order, with no evolutionary fairies quick enough to punish their destructive stupidity (even assuming that it actually would be long-term maladaptive, which is far from obvious).
You only get to long-term descendants through short-term ones.
If an entity does stupid things, it’s disfavored against competitors that don’t do those stupid things, all else being equal. So it needs to adapt by ceasing the stupid behavior or otherwise lose.
Any assumption of the form “super-intelligent AI will take actions that are super-stupid” is dubious.
Clearly. The point is that the actions it takes might seem stupidly destructive only according to humanity’s feeble understanding and parochial values. Something involving extermination of all humans, say. My impression is that the “accel”-endorsed attitude to this is to be a good sport and graciously accept the verdict of natural selection.
That just falls back on the common doomer assumption that “evil is optimal” (as Sutton put it). Sure, if evil is optimal and you have an entity that behaves optimally, it’ll act in evil ways.
But there are good reasons to think that evil is not optimal in current conditions. At least as long as a Dyson sphere has not yet been constructed, there are massive gains available from positive-sum cooperation directed towards technological progress. In these conditions, negative-sum conflict is a stupid waste.
This view, that evil is not optimal, ties back into the continuation framing. After all, you can make a philosophical argument either way. But in the continuation framing, we can ask ourselves whether evil is empirically optimal for humans, which will suggest whether evil is optimal for non-biological descendants (since they continue humanity). And in fact we see evil losing a lot, and not coincidentally—WW2 went the way it did in part because the losing side was evil.
Indeed, and what baffles me is that many are extremely sure one way or the other, even though philosophy doesn’t exactly have a track record to inspire such confidence. Of course, this also means that nobody is going to stop building stuff because of philosophical arguments, so we’ll have empirical evidence soon enough...
Can you give examples of what you have in mind? Because an obvious counterexample is evolution itself. It has produced an enormous variety of different things. There are instances of convergent evolution: “crab” and “tree” are strategies, not monophyletic taxa. But crabs are not similar to trees in any useful sense. If they are solutions to the same problem, they have in common only that they are solutions to the same problem. This does not make them similar solutions.
One might ask whether evolution is or is not a case of “flexible adaptive systems … exposed to the same problems”, but that would just be a debate over definitions, and you already spoke of “our descendants … competing in the same environment”. That sounds like evolution.
I think I agree with everything you wrote. Yes I’d expect there to be multiple niches available in the future, but I’d expect our descendants to ultimately fill all of them, creating an ecosystem of intelligent life. There is a lot of time available for our descendants to diversify, so it’d be surprising if they didn’t.
How much that diversification process resembles Darwinian evolution, I don’t know. Natural selection still applies, since it’s fundamentally the fact that the life we observe today disproportionately descends from past life that was effective at self-reproduction, and that’s essentially tautological. But Darwinian evolution is undirected, whereas our descendants can intelligently direct their own evolution, and that could conceivably matter. I don’t see why it would prevent diversification, though.
Edit:
Here are some thoughts in reply to your request for examples. Though it’s impossible to know what the niches of the long-term future will be, one idea is that there could be an analogue to “plant” and “animal”. A plant-type civilization would occupy a single stellar system, obtaining resources from it via Dyson sphere, mining, etc. An animal-type civilization could move from star to star, taking resources from the locals (which could be unpleasant for the locals, but not necessarily, as with bees pollinating flowers).
I’d expect both those civilizations to descend from ours, much like how crabs and trees both descend from LUCA.