I think that’s a straw man of the classic AI-related catastrophe scenarios. Bostrom’s “covert preparation” --> “Treacherous turn” --> “takeover” story maps pretty nicely to Paul’s “seek influence via gaming tests” --> “they are now more interested in controlling influence after the resulting catastrophe then continuing to play nice with existing institutions and incentives” --> ” One day leaders may find that despite their nominal authority they don’t actually have control over what these institutions do. For example, military leaders might issue an order and find it is ignored. This might immediately prompt panic and a strong response, but the response itself may run into the same problem, and at that point the game may be up. ”
I think that’s a straw man of the classic AI-related catastrophe scenarios. Bostrom’s “covert preparation” --> “Treacherous turn” --> “takeover” story maps pretty nicely to Paul’s “seek influence via gaming tests” --> “they are now more interested in controlling influence after the resulting catastrophe then continuing to play nice with existing institutions and incentives” --> ” One day leaders may find that despite their nominal authority they don’t actually have control over what these institutions do. For example, military leaders might issue an order and find it is ignored. This might immediately prompt panic and a strong response, but the response itself may run into the same problem, and at that point the game may be up. ”