Your argument seems to be that it’ll be hard for the CEO to align the AI to themselves and screw the rest of the company. Sure, maybe. But will it be equally hard for the company as a whole to align the AI to its interests and screw the rest of the world? That’s less outlandish, isn’t it? But equally catastrophic. After all, companies have been known to do very bad things when they had impunity; and if you say “but the spec is published to the world”, recall that companies have been known to lie when it benefited them, too.
Your argument seems to be that it’ll be hard for the CEO to align the AI to themselves and screw the rest of the company. Sure, maybe. But will it be equally hard for the company as a whole to align the AI to its interests and screw the rest of the world? That’s less outlandish, isn’t it? But equally catastrophic. After all, companies have been known to do very bad things when they had impunity; and if you say “but the spec is published to the world”, recall that companies have been known to lie when it benefited them, too.