[Not sure if what follows is a blend of “Matrix AI” and “Moral Realism AI” since moral realism is a philosophical stance very common among philosophers, see https://plato.stanford.edu/entries/moral-realism/ and I consider it a misnomer for the scenario described above.]
We are the AGI
Turns out humanity is an experiment to see if moral reasoning can be discovered/sustained by evolutionary means. In the process of recursive self-improvement, a UChicago philosophy professor, Alan Gewirth, learns that there is an objective moral truth which is compelling for all beings capable of reasoning and of having goals (whatever goals, not necessarily benign ones). His views are summarized in a book, “Reason and morality” UChicago Press 1978, and philosophers pay a great deal of attention, see e.g. Edward Regis Jr (ed) “Gewirth’s ethical rationalism” UChicago Press 1984. Gradually, these views spread, and a computer verification of a version of Gewirth’s argument is produced (Fuenmayor and Benzmueller 2019). Silicon-based AGI avails itself of the great discovery made by DNA-based AGI. As the orthogonality thesis is false, it adapts its goal in order to maximize objective goodness in the universe to do no harm.
Dear Roman, you think it’s less likely to get lost in the noise there? In fact I posted this first before the letter was out, and one of the mods set it back to draft status because he was afraid I’m making it public before the embargo expired. But it was up for a few hours and succeeded in gathering about 20 downvotes, so it’s clearly not a popular view...