ASI governed by a value system which, if placed in complete control of the Earth, would still be something we could live with,
That’s exactly my point. However, once the value system is defined, it will either lock mankind in or be corrigible. The former case contains options like my take where the AI only provides everyone with access to education and enforces only universally agreed political opinions or[1] the situations where the AI builds the Deep Utopia or governs the world, criminalising social parasitism in the whole world. The race to ASI might also be motivated with the belief that it will empower the creators’ state or enforce the state’s ethos.
The first example of the ASI’s role, unlike the latter two, might be considered not as a lock-in, but as another system where mankind can govern itself.
That’s exactly my point. However, once the value system is defined, it will either lock mankind in or be corrigible. The former case contains options like my take where the AI only provides everyone with access to education and enforces only universally agreed political opinions or[1] the situations where the AI builds the Deep Utopia or governs the world, criminalising social parasitism in the whole world. The race to ASI might also be motivated with the belief that it will empower the creators’ state or enforce the state’s ethos.
The first example of the ASI’s role, unlike the latter two, might be considered not as a lock-in, but as another system where mankind can govern itself.