Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.
Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.