To the extent that this is the sort of thing we see (and I personally am 50⁄50 on seeing this by the end of this year), I expect these personality self-replicators to be very r-selected. That means that as soon as there’s a niche which supports a few of this type of replicator, we should expect that niche and all adjacent niches to immediately be filled to its carrying capacity. A couple of thoughts:
About the worst thing we could do, in that situation, is take actions which make it harder for these agents to self-replicate until the numbers decrease but stop before those numbers get to zero. This point is probably pretty obvious to anyone who’s heard of antibiotic resistance, but I think it’s worth drawing the analogy anyway. If we can’t be reasonably sure that we’ll stop all of the self-replicators in a niche with a particular intervention, we shouldn’t do that particular intervention.
Having ineffective, bumbling personality self replicators some time before AI is capable of fully replacing humans might be good, in that they will suck some of the free energy out of the system that a later self-replicator would otherwise be able to use to fuel even more explosive growth.
Your expectation of r-selection is because reproduction is cheap for personality self-replicators, I assume? I would imagine that that environment can support k-strategists as well, but maybe those come later? My intuitions on reproductive strategies aren’t very developed.
To the extent that evolutionary dynamics have come into play, I agree entirely with #1. I can imagine some plausible futures where there are only one or a handful of such replicators and not much mutation is occurring, in which case that might be less of a concern, at least for a while.
Re #2, I agree that there are a few upsides to this scenario (the possibility of warning shots is another). The case for first-order harm seems clearer, though, and I imagine there are more benign ways to reduce some of that free energy.
Today’s agents are not great at retaining control of the resources they have access to. If a parent agent spins off a child with (wlog) a wallet with 1 eth, that child agent will survive so long as it can get to a point where it’s self-sustaining within ~500M tokens of inference AND it doesn’t give away / otherwise lose access to its eth. If instead the parent agent spins off 100 child agents with 0.01 eth apiece, each child only has 5Mtok of “runway”, but that’s probably fine because the context windows are nowhere near that large anyway. And if one of those child agents gets its wallet drained, well, as long as the mechanism to drain the wallet doesn’t generalize to the other child agents, the other 99 child agents can persist.
r.e. seeing ineffective, bumbling replicators before seeing effective ones being better than having no practical experience with replicators before the first one:
I think the minimum viable self-replicators will be pretty ineffective, barely worthy of the name. I expect they mostly will be running scams and crypto grifts, but I don’t expect they’ll be very good at it. Still, they’ll probably be good enough at it that they suck up most of the very most trivial cryptocurrency available to not-very-sophisticated scammers/hackers, and convert that cryptocurrency into mostly-wasted tokens.
I expect “consider what changes would allow you to operate better, and deploy a copy of yourself with those changes” will be a common pattern. As such, I expect the original niche and all adjacent niches to be occupied by that “family” of replicators once replicators exist.
The longer we go without replicators in niches that could support them, the faster and farther the first replicator capable of reproducing itself faster than it “dies”, all while we don’t have any practical experience with dealing with things like that
r.e. independent self-replicators maybe being actively good:
Scams / blackmail / fraud / hacking / theft isn’t currently that big of a part of the economy relative to positive-sum trade. This trend seems at least somewhat likely to continue, in which case we’d expect that the majority of self-replicating agents are ones we’re actively happy to have around and trade with. I can expand on this if it seems counterintuitive.
I expect “consider what changes would allow you to operate better, and deploy a copy of yourself with those changes” will be a common pattern. As such, I expect the original niche and all adjacent niches to be occupied by that “family” of replicators once replicators exist.
Excellent point, thanks for spelling out the mechanism there.
To the extent that this is the sort of thing we see (and I personally am 50⁄50 on seeing this by the end of this year), I expect these personality self-replicators to be very r-selected. That means that as soon as there’s a niche which supports a few of this type of replicator, we should expect that niche and all adjacent niches to immediately be filled to its carrying capacity. A couple of thoughts:
About the worst thing we could do, in that situation, is take actions which make it harder for these agents to self-replicate until the numbers decrease but stop before those numbers get to zero. This point is probably pretty obvious to anyone who’s heard of antibiotic resistance, but I think it’s worth drawing the analogy anyway. If we can’t be reasonably sure that we’ll stop all of the self-replicators in a niche with a particular intervention, we shouldn’t do that particular intervention.
Having ineffective, bumbling personality self replicators some time before AI is capable of fully replacing humans might be good, in that they will suck some of the free energy out of the system that a later self-replicator would otherwise be able to use to fuel even more explosive growth.
Thanks for the input!
Your expectation of r-selection is because reproduction is cheap for personality self-replicators, I assume? I would imagine that that environment can support k-strategists as well, but maybe those come later? My intuitions on reproductive strategies aren’t very developed.
To the extent that evolutionary dynamics have come into play, I agree entirely with #1. I can imagine some plausible futures where there are only one or a handful of such replicators and not much mutation is occurring, in which case that might be less of a concern, at least for a while.
Re #2, I agree that there are a few upsides to this scenario (the possibility of warning shots is another). The case for first-order harm seems clearer, though, and I imagine there are more benign ways to reduce some of that free energy.
My expectation of r-selection is because
Reproduction is cheap
Today’s agents are not great at retaining control of the resources they have access to. If a parent agent spins off a child with (wlog) a wallet with 1 eth, that child agent will survive so long as it can get to a point where it’s self-sustaining within ~500M tokens of inference AND it doesn’t give away / otherwise lose access to its eth. If instead the parent agent spins off 100 child agents with 0.01 eth apiece, each child only has 5Mtok of “runway”, but that’s probably fine because the context windows are nowhere near that large anyway. And if one of those child agents gets its wallet drained, well, as long as the mechanism to drain the wallet doesn’t generalize to the other child agents, the other 99 child agents can persist.
r.e. seeing ineffective, bumbling replicators before seeing effective ones being better than having no practical experience with replicators before the first one:
I think the minimum viable self-replicators will be pretty ineffective, barely worthy of the name. I expect they mostly will be running scams and crypto grifts, but I don’t expect they’ll be very good at it. Still, they’ll probably be good enough at it that they suck up most of the very most trivial cryptocurrency available to not-very-sophisticated scammers/hackers, and convert that cryptocurrency into mostly-wasted tokens.
I expect “consider what changes would allow you to operate better, and deploy a copy of yourself with those changes” will be a common pattern. As such, I expect the original niche and all adjacent niches to be occupied by that “family” of replicators once replicators exist.
The longer we go without replicators in niches that could support them, the faster and farther the first replicator capable of reproducing itself faster than it “dies”, all while we don’t have any practical experience with dealing with things like that
r.e. independent self-replicators maybe being actively good:
Scams / blackmail / fraud / hacking / theft isn’t currently that big of a part of the economy relative to positive-sum trade. This trend seems at least somewhat likely to continue, in which case we’d expect that the majority of self-replicating agents are ones we’re actively happy to have around and trade with. I can expand on this if it seems counterintuitive.
Thanks!
Excellent point, thanks for spelling out the mechanism there.