How will the company paying for using this system identify that their whole compute budget is being eaten by self-replicating patterns? Will it be obvious?
It would be even worse if the self-replicating patterns only involved a small tweak that, aside from the self-replication feature, also happening to still spin-off useful outputs for the company, sort of like HIV allowing hosts to continue to thrive for many years while replicating.
You can think of it as an inner loop. Over enough cycles, the almost aligned system forms a cancer of a non aligned system. Over enough incidents of that happening, the cancer system escapes and is able to survive independently of the host.
If we don’t want this to inevitably happen we have to fill all the ecological space with controllable or aligned systems.
How will the company paying for using this system identify that their whole compute budget is being eaten by self-replicating patterns? Will it be obvious?
It would be even worse if the self-replicating patterns only involved a small tweak that, aside from the self-replication feature, also happening to still spin-off useful outputs for the company, sort of like HIV allowing hosts to continue to thrive for many years while replicating.
“it depends”.
You can think of it as an inner loop. Over enough cycles, the almost aligned system forms a cancer of a non aligned system. Over enough incidents of that happening, the cancer system escapes and is able to survive independently of the host.
If we don’t want this to inevitably happen we have to fill all the ecological space with controllable or aligned systems.