human control becomes increasingly dependent on a complicated system with lots of moving parts. One day leaders may find that despite their nominal authority they don’t actually have control over what these institutions do. For example, military leaders might issue an order and find it is ignored.
Of course this already often happens where the moving parts are human. I heard a former UK minister say this yesterday—when you get into government you start pulling levers and find they don’t work, and may not seem to be connected to anything. (Less common in the military, due to the emphasis on unquestioning obedience with strong enforcement.)
Current bureaucracies are mechanisms made of meat, with squishy cogs, so it’s easy to see that they can fail in many unexpected ways (someone mistypes a word in a crucial letter, or forgets to send it, or it gets lost in the mail, or the recipient misunderstands the letter, etc.) Hence easier to see that unexpected failures can happen with complex AI mechanisms too. The mistake is regarding them as much more reliable and safe than humans.
Incidentally, I found this post much easier to understand by replacing ‘influence-seeking’ in my head with the (almost synonymous?) ‘power-seeking’, being a well-known phrase & human phenomenon.
Of course this already often happens where the moving parts are human. I heard a former UK minister say this yesterday—when you get into government you start pulling levers and find they don’t work, and may not seem to be connected to anything. (Less common in the military, due to the emphasis on unquestioning obedience with strong enforcement.)
Current bureaucracies are mechanisms made of meat, with squishy cogs, so it’s easy to see that they can fail in many unexpected ways (someone mistypes a word in a crucial letter, or forgets to send it, or it gets lost in the mail, or the recipient misunderstands the letter, etc.) Hence easier to see that unexpected failures can happen with complex AI mechanisms too. The mistake is regarding them as much more reliable and safe than humans.
Incidentally, I found this post much easier to understand by replacing ‘influence-seeking’ in my head with the (almost synonymous?) ‘power-seeking’, being a well-known phrase & human phenomenon.