Anyways, why not build an AI who’s goal was to non-coercively increase the intelligence of mankind?
It’s not going to make you more powerful than it if it’s going to limit its ability to make you more intelligent in the future. It will make sure it’s intelligent enough to convince you to accept the modifications it wants you to have until it convinces you to accept the one that gives you its utility function.
It’s not going to make you more powerful than it if it’s going to limit its ability to make you more intelligent in the future. It will make sure it’s intelligent enough to convince you to accept the modifications it wants you to have until it convinces you to accept the one that gives you its utility function.