I believe that a fundamental requirement for any rational agent is the motivation to act maximally intelligently and correctly. That requirement seems even more obvious if we are talking about a conjectured artificial general intelligence (AGI) that is able to improve itself to the point where it is substantially better at most activities than humans. Since if it wouldn’t want to be maximally correct then it wouldn’t become superhuman intelligent in the first place.
The standard counterargument is along the lines of:
it won’t care about getting things right per se, it will only employ rationality as a means to other goals.(Or: instrumental rationality is the only kind, because that’s how we define rationality).
What justifies the “will”, the claim of necessity, or at least high probability, brings us back to the title of the original posting: evidence for the Orthogonality Thesis. Is non-instrumental rationality, rationality-as-a-goal impossible? Is no-one trying to build it? Why try to build single minded Artificial Obsessive Compulsives if it is dangerous? Isn’t rationality-as-a-goal a safer architecture?
The standard counterargument is along the lines of: it won’t care about getting things right per se, it will only employ rationality as a means to other goals.(Or: instrumental rationality is the only kind, because that’s how we define rationality).
What justifies the “will”, the claim of necessity, or at least high probability, brings us back to the title of the original posting: evidence for the Orthogonality Thesis. Is non-instrumental rationality, rationality-as-a-goal impossible? Is no-one trying to build it? Why try to build single minded Artificial Obsessive Compulsives if it is dangerous? Isn’t rationality-as-a-goal a safer architecture?