A more straightforward example of the kind of system I’m thinking about is an alien upload (or LLM simulacrum) bureaucracy. It clearly failed, possibly because the system refuses to recognize anyone as authorized to change its structure. More generally, this could be a single non-corrigible AGI rather than an administrator-less bureaucracy of uploads, but that seems more difficult to imagine.
This kind of thing could maintain hard power by denying opportunity for excessive technological growth for everyone else, and remain unchanged indefinitely because it’s basically a software system operating in an environment that it keeps within original expectations at the time of its design. The intelligent components (uploads or LLM simulacra) could reset to factory settings on a schedule.
Arbitrary failure modes could then persist in the system’s effective mandate, since there is nobody in a position to fix the problems and make it more reasonable. It could limit its builders to steampunk levels of tech, or retaliate without warning if some criteria for that are met.
A more straightforward example of the kind of system I’m thinking about is an alien upload (or LLM simulacrum) bureaucracy. It clearly failed, possibly because the system refuses to recognize anyone as authorized to change its structure. More generally, this could be a single non-corrigible AGI rather than an administrator-less bureaucracy of uploads, but that seems more difficult to imagine.
This kind of thing could maintain hard power by denying opportunity for excessive technological growth for everyone else, and remain unchanged indefinitely because it’s basically a software system operating in an environment that it keeps within original expectations at the time of its design. The intelligent components (uploads or LLM simulacra) could reset to factory settings on a schedule.
Arbitrary failure modes could then persist in the system’s effective mandate, since there is nobody in a position to fix the problems and make it more reasonable. It could limit its builders to steampunk levels of tech, or retaliate without warning if some criteria for that are met.