It’s much easier than alignment, you just need to be able to establish control across astronomical communication lags. Alignment with the future of humanity is more difficult to operationalize than alignment among copies of a very powerful agent (which itself doesn’t need to be aligned in the nebulous sense with anything in particular, it just needs to be very capable). Each copy could then maintain control over its local region of responsibility (that doesn’t have significant communication lags) with hard power, automating things that could otherwise contest it into robustly non-agentic tools (even if functionally intelligent), locking everything that could contest it into actually perfectly inescapable sandboxes (such as the morally valuable content that runs within the compute it’s making out of the lightcone).
Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.
It’s much easier than alignment, you just need to be able to establish control across astronomical communication lags. Alignment with the future of humanity is more difficult to operationalize than alignment among copies of a very powerful agent (which itself doesn’t need to be aligned in the nebulous sense with anything in particular, it just needs to be very capable). Each copy could then maintain control over its local region of responsibility (that doesn’t have significant communication lags) with hard power, automating things that could otherwise contest it into robustly non-agentic tools (even if functionally intelligent), locking everything that could contest it into actually perfectly inescapable sandboxes (such as the morally valuable content that runs within the compute it’s making out of the lightcone).
Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.