I think AIs will always be imperfectly rational as they cannot optimize everything and still make decisions, and besides, optimal outcomes are not the same as perfect outcomes. I do think self-improving AI will tend to self-improve toward perfect rationality to the point where there is no significant difference between optimal and perfect.
As for a system in which AIs that disagree on friendliness are not dangerous: I can’t imagine one. If friendliness includes protecting humans from unfriendly AI, a perfectly friendly AI would destroy imperfectly friendly AIs because imperfect friendliness is unfriendly compared to perfect friendliness. Therefore, an imperfectly friendly AI that thought it was perfectly friendly would destroy other AIs it disagreed with. To put it crudely, a “women’s lib” AI could not coexist with a “Sharia” AI. And an imperfectly friendly AI that knew it was imperfectly friendly could not trust itself to make friendly decisions, and so would be acting in an unfriendly way if it tried to make decisions for people. Of course, this is all pretty primitive word vomit that I haven’t spent much time thinking about, and ignores the reality that if imperfectly rational and friendly humans can get along, AIs should be able to as well.
Interestingly, Brin argues from the question Shall we maintain momentum and fealty to the underlying concepts of the Western Enlightenment? What he fails to consider here is the very genuine possibility that an AGI may be created by a non-Enlightenment culture, i.e. Japan or China. Perhaps a likelier possibility yet is an AGI created by a coalition of cultures, researchers from different countries working together, which could further extend to a multinational corporation which may not reflect any particular culture other than a basically capitalist one. While capitalism is a pretty fundamental Enlightenment ideal, it is only a framework in which other cultures may function very differently. The way a Japanese businessman or CEO considers competition is not the same as how a Russian or an American or a Brazilian views it. How different would a Japanese AGI be from an American one, or an AGI built by the Chinese government vs. one built by a private lab in the EU?
I think AIs will always be imperfectly rational as they cannot optimize everything and still make decisions, and besides, optimal outcomes are not the same as perfect outcomes. I do think self-improving AI will tend to self-improve toward perfect rationality to the point where there is no significant difference between optimal and perfect.
As for a system in which AIs that disagree on friendliness are not dangerous: I can’t imagine one. If friendliness includes protecting humans from unfriendly AI, a perfectly friendly AI would destroy imperfectly friendly AIs because imperfect friendliness is unfriendly compared to perfect friendliness. Therefore, an imperfectly friendly AI that thought it was perfectly friendly would destroy other AIs it disagreed with. To put it crudely, a “women’s lib” AI could not coexist with a “Sharia” AI. And an imperfectly friendly AI that knew it was imperfectly friendly could not trust itself to make friendly decisions, and so would be acting in an unfriendly way if it tried to make decisions for people. Of course, this is all pretty primitive word vomit that I haven’t spent much time thinking about, and ignores the reality that if imperfectly rational and friendly humans can get along, AIs should be able to as well.
Interestingly, Brin argues from the question Shall we maintain momentum and fealty to the underlying concepts of the Western Enlightenment? What he fails to consider here is the very genuine possibility that an AGI may be created by a non-Enlightenment culture, i.e. Japan or China. Perhaps a likelier possibility yet is an AGI created by a coalition of cultures, researchers from different countries working together, which could further extend to a multinational corporation which may not reflect any particular culture other than a basically capitalist one. While capitalism is a pretty fundamental Enlightenment ideal, it is only a framework in which other cultures may function very differently. The way a Japanese businessman or CEO considers competition is not the same as how a Russian or an American or a Brazilian views it. How different would a Japanese AGI be from an American one, or an AGI built by the Chinese government vs. one built by a private lab in the EU?