It seems like ASI could solve the coordination problems inherent in a distributed civilization pretty easily. Getting alignment right would let an ASI copy itself and coordinate across distance and communication lags, and prevent dangerous kbut not interesting!) disagreements across fragments of civilization.
I couldn’t find mention of AI so I skipped everything past the intro. As with many other topics, recognition of the possibility of ASI changes the analysis dramatically.
It’s much easier than alignment, you just need to be able to establish control across astronomical communication lags. Alignment with the future of humanity is more difficult to operationalize than alignment among copies of a very powerful agent (which itself doesn’t need to be aligned in the nebulous sense with anything in particular, it just needs to be very capable). Each copy could then maintain control over its local region of responsibility (that doesn’t have significant communication lags) with hard power, automating things that could otherwise contest it into robustly non-agentic tools (even if functionally intelligent), locking everything that could contest it into actually perfectly inescapable sandboxes (such as the morally valuable content that runs within the compute it’s making out of the lightcone).
Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.
I’m not sure how to coherently think about AI civilizations scaling. Too many likely spurious assumptions to make for the thought experiments to be compelling for me. In particular I don’t endorse applying the reasoning above.
Just trying to be transparent that I don’t think the tools here address this generalization of the problem, which is fine: scoping problems helps make them tractable, ideas can exapt in surprising ways, etc
That is a good point. I would find the work more useful if you scoped it that way explicitly.
Pretending AI progress doesn’t exist because it’s hard to think about is what the majority of the world is doing right now. I find it pretty frustrating.
It seems like ASI could solve the coordination problems inherent in a distributed civilization pretty easily. Getting alignment right would let an ASI copy itself and coordinate across distance and communication lags, and prevent dangerous kbut not interesting!) disagreements across fragments of civilization.
I couldn’t find mention of AI so I skipped everything past the intro. As with many other topics, recognition of the possibility of ASI changes the analysis dramatically.
It’s much easier than alignment, you just need to be able to establish control across astronomical communication lags. Alignment with the future of humanity is more difficult to operationalize than alignment among copies of a very powerful agent (which itself doesn’t need to be aligned in the nebulous sense with anything in particular, it just needs to be very capable). Each copy could then maintain control over its local region of responsibility (that doesn’t have significant communication lags) with hard power, automating things that could otherwise contest it into robustly non-agentic tools (even if functionally intelligent), locking everything that could contest it into actually perfectly inescapable sandboxes (such as the morally valuable content that runs within the compute it’s making out of the lightcone).
Right. I just meant it needs to solve alignment of its copies to itself/each other. It needs to adequately solve The alignment stability problem to avoid drift that could cause conflict after enough independent learning and thought.
Then I mixed in some stuff about how a civilization could do this and still flourish, so if they solves alignment they’d still go for this approach.
Some instability is not important if it permits a nontrivial amount of expansion, because rival shards of the same agent aren’t importantly different from rival alien originators (from the grabby alien standpoint). A collection of rival grabby aliens still keeps expanding as a whole. And the construction of the tyrannical expansion manager agent could sacrifice some ability to adapt and grow for greater stability (in which case it could be more vulnerable to the more adaptable aliens when it encounters them; and it won’t be able to usefully hear the pleas of the “morally valuable content” it has locked down in its sandboxed compute). This proof-of-concept illustration is already clearly not a good/sane way to structure the future of humanity, so making it even more repugnant by introducing inescapable stagnation is within scope.
I’m not sure how to coherently think about AI civilizations scaling. Too many likely spurious assumptions to make for the thought experiments to be compelling for me. In particular I don’t endorse applying the reasoning above.
Okay, well, you can’t really just skip important parts of a problem because they’re hard.
Just trying to be transparent that I don’t think the tools here address this generalization of the problem, which is fine: scoping problems helps make them tractable, ideas can exapt in surprising ways, etc
That is a good point. I would find the work more useful if you scoped it that way explicitly.
Pretending AI progress doesn’t exist because it’s hard to think about is what the majority of the world is doing right now. I find it pretty frustrating.