While KL divergence is a very natural measure of the “goodness of approximation” of a probability distribution, which happens not to talk about the utility function, there is still a strong sense in which only an instrumental rationalist can speak of a “better approximation”, because only an instrumental rationalist can say the word “better”.
KL divergence is an attempt to use a default sort of metric of goodness of approximation, without talking about the utility function, or while knowing as little as possible about the utility function; but in fact, in the absence of a utility function, you actually just can’t say the word “better”, period.
To the extent that this is true, perhaps the very notion of an epistemic rationalist (perhaps also of epistemic rationality) is incoherent. (“Epistemic rationality means acting so as to maximize one’s accuracy.” “Ah, but hidden in that word accuracy is some sort of evaluation, which you aren’t allowed to have.”) But it sure seems like a useful notion.
I propose that there is at least one useful notion of epistemic rationality; in fact, there’s one for each viable notion of what counts as better accuracy; since real people have utility functions, calling a real person an epistemic rationalist is really shorthand for “has a utility function that highly values accuracy-in-some-particular-sense”; that one can usefully talk about epistemic rationality in general, meaning something like “things that are true about anyone who’s an epistemic rationalist in any of that term’s many specific senses”; and that it’s at least a defensible claim that something enough like K-L divergence to make Peter’s argument go through is likely to be part of any viable notion of accuracy.
If epistemic rationalists can’t speak of a “better approximation,” then how can an epistemic rationalist exist in a universe with finite computational resources?
This is basically right, but I guess I think of it in slightly different terms. The KL divergence embodies a particular, implicit utility function, which just happens to be wrong lots of the time. So it can make sense to speak of “better_KL”, it’s just not something that’s necessarily very useful.
Note also that alternative divergence measures, embodying different implicit utility functions, could give different answers. For example, Jensen-Shannon divergence would agree with instrumental rationality here, no? (Though you could obviously construct examples where it too would diverge from our actual utility functions.)
I basically agree with this, although I guess I’d always thought of it in terms of the KL distance incorporating a particular, implicit utility function that happens to be wrong in many cases. It can speak of “better_KL”, but only according to a (sometimes) stupid utility function.
The failure of the KL divergence to incorporate an adequate notion of “betterness” is also demonstrated by the fact that you’d get a different answer if you used an alternative divergence measure. Jensen-Shannon divergence, for example, would give the same answer as instrumental rationality in this example, no? (Though you could obviously construct different examples where it too would diverge from instrumental rationality.)
While KL divergence is a very natural measure of the “goodness of approximation” of a probability distribution, which happens not to talk about the utility function, there is still a strong sense in which only an instrumental rationalist can speak of a “better approximation”, because only an instrumental rationalist can say the word “better”.
KL divergence is an attempt to use a default sort of metric of goodness of approximation, without talking about the utility function, or while knowing as little as possible about the utility function; but in fact, in the absence of a utility function, you actually just can’t say the word “better”, period.
To the extent that this is true, perhaps the very notion of an epistemic rationalist (perhaps also of epistemic rationality) is incoherent. (“Epistemic rationality means acting so as to maximize one’s accuracy.” “Ah, but hidden in that word accuracy is some sort of evaluation, which you aren’t allowed to have.”) But it sure seems like a useful notion.
I propose that there is at least one useful notion of epistemic rationality; in fact, there’s one for each viable notion of what counts as better accuracy; since real people have utility functions, calling a real person an epistemic rationalist is really shorthand for “has a utility function that highly values accuracy-in-some-particular-sense”; that one can usefully talk about epistemic rationality in general, meaning something like “things that are true about anyone who’s an epistemic rationalist in any of that term’s many specific senses”; and that it’s at least a defensible claim that something enough like K-L divergence to make Peter’s argument go through is likely to be part of any viable notion of accuracy.
If epistemic rationalists can’t speak of a “better approximation,” then how can an epistemic rationalist exist in a universe with finite computational resources?
Pure epistemic rationalists with no utility function? Well, they can’t, really. That’s part of the problem with the Oracle AI scenario.
They can speak of a “closer approximation” instead. (But that still needs a metric.)
This is basically right, but I guess I think of it in slightly different terms. The KL divergence embodies a particular, implicit utility function, which just happens to be wrong lots of the time. So it can make sense to speak of “better_KL”, it’s just not something that’s necessarily very useful.
Note also that alternative divergence measures, embodying different implicit utility functions, could give different answers. For example, Jensen-Shannon divergence would agree with instrumental rationality here, no? (Though you could obviously construct examples where it too would diverge from our actual utility functions.)
I basically agree with this, although I guess I’d always thought of it in terms of the KL distance incorporating a particular, implicit utility function that happens to be wrong in many cases. It can speak of “better_KL”, but only according to a (sometimes) stupid utility function.
The failure of the KL divergence to incorporate an adequate notion of “betterness” is also demonstrated by the fact that you’d get a different answer if you used an alternative divergence measure. Jensen-Shannon divergence, for example, would give the same answer as instrumental rationality in this example, no? (Though you could obviously construct different examples where it too would diverge from instrumental rationality.)