One of the biggest considerations would be the process for activating “crunch time”. In what situations should crunch time be declared? Who decides? How far out would we want to activate and would there be different levels? Are there any downsides of such a process including unwanted attention?
If these aren’t discussed in advance, then I imagine that far too much of the available time could be taken up by whether to activate crunch time protocols or not.
PS. I actually proposed here that we might be able to get a superintelligence to solve most of the problem of embedded agency by itself. I’ll try to write it up into a proper post soon.
One of the biggest considerations would be the process for activating “crunch time”. In what situations should crunch time be declared? Who decides? How far out would we want to activate and would there be different levels? Are there any downsides of such a process including unwanted attention?
If these aren’t discussed in advance, then I imagine that far too much of the available time could be taken up by whether to activate crunch time protocols or not.
PS. I actually proposed here that we might be able to get a superintelligence to solve most of the problem of embedded agency by itself. I’ll try to write it up into a proper post soon.