An AI consists of a pragmatically computable approximation of a decision theory, not an implementation of a decision theory. This would seem to be the human reflection of that.
In general, I approve; I’ve never told anyone to run calculations they can’t do, including, in some cases, quantifying probabilities.
But the process of going from a simple, pure decision theory, to a collection of approximate rules, is itself subject to distortion in the mind of a less than perfect rationalist—i.e., people will be tempted to endorse just about any rule that has an intuitive appeal, even if it leads to e.g. circular altruism.
What the rules are an approximation to is some purer decision theory—one is still seeking the same pure thing at the center, taking some particular path closer to it. If you’re trying to approximate some incomputable formula, you will nonetheless quite often think of that formula, even if abstractly.
When you paint a sunset you don’t end up with a sunset, you end up with a painting; but you should still be looking at the sunset itself, not just your brushes.
An AI consists of a pragmatically computable approximation of a decision theory, not an implementation of a decision theory. This would seem to be the human reflection of that.
In general, I approve; I’ve never told anyone to run calculations they can’t do, including, in some cases, quantifying probabilities.
But the process of going from a simple, pure decision theory, to a collection of approximate rules, is itself subject to distortion in the mind of a less than perfect rationalist—i.e., people will be tempted to endorse just about any rule that has an intuitive appeal, even if it leads to e.g. circular altruism.
What the rules are an approximation to is some purer decision theory—one is still seeking the same pure thing at the center, taking some particular path closer to it. If you’re trying to approximate some incomputable formula, you will nonetheless quite often think of that formula, even if abstractly.
When you paint a sunset you don’t end up with a sunset, you end up with a painting; but you should still be looking at the sunset itself, not just your brushes.