if you say, “The rational belief is X, but the true belief is Y” then you are probably using the word “rational” in a way that means something other than what most of us have in mind
There is nothing wrong with saying “the rational belief given incomplete information is X, but the truth is Y, as follows from more complete information”, and the quote didn’t try to forbid that. Rather, it warned against saying “as far as I know, Y is true, but rationality tells me to believe X”.
Edit: Rationality (the epistemic sort of) is a method which derives true beliefs from given information in the optimal way. Rational beliefs should depend on accessible information and so they can’t always correspond to the truth. The idea of the quote is that we define rationality as the best truth-finding algorithm, and therefore we are not allowed to use any other algorithm to determine the truth. If we do that, we either deliberately use a worse tool in the presence of a better one, or we don’t follow our definition of rationality.
Jaynes used to recommend that no one ever write out an unconditional probability: That you never, ever write simply P(A), but always write P(A|I), where I is your prior information. I’ll use Q instead of I, for ease of reading, but Jaynes used I. Similarly, one would not write P(A|B) for the posterior probability of A given that we learn B, but rather P(A|B,Q), the probability of A given that we learn B and had background information Q.
There is nothing wrong with saying “the rational belief given incomplete information is X, but the truth is Y, as follows from more complete information”, and the quote didn’t try to forbid that. Rather, it warned against saying “as far as I know, Y is true, but rationality tells me to believe X”.
Edit: Rationality (the epistemic sort of) is a method which derives true beliefs from given information in the optimal way. Rational beliefs should depend on accessible information and so they can’t always correspond to the truth. The idea of the quote is that we define rationality as the best truth-finding algorithm, and therefore we are not allowed to use any other algorithm to determine the truth. If we do that, we either deliberately use a worse tool in the presence of a better one, or we don’t follow our definition of rationality.
As Eliezer mentions here: