To clarify the comment for @tjaffee, a superintelligence could do the following
Use invalid but extremely convincing arguments that make the “doomer”[1] change his mind. This appears realistic because sometimes people become convinced of false things through invalid argumentation[2].
Give a complete plan for an aligned superintelligence, shutting itself down in the process, creating a true and probably convincing argument (this is vanishingly unlikely).
To clarify the comment for @tjaffee, a superintelligence could do the following
Use invalid but extremely convincing arguments that make the “doomer”[1] change his mind. This appears realistic because sometimes people become convinced of false things through invalid argumentation[2].
Give a complete plan for an aligned superintelligence, shutting itself down in the process, creating a true and probably convincing argument (this is vanishingly unlikely).
Ugh.
Like maybe once or twice.
People can also be convinced of false things through valid argumentation.