Ah, MIRI summer fellows! Maybe that’s why there’s so many posts today.

I think that if there’s a dichotomy, it’s “abstract/ideal agents” vs. “physical ‘agents’”.

Physical agents, like humans, don’t have to be anything like agent clusters—there doesn’t have to be any ideal agent hiding inside them. Instead, thinking about them as agents is a descriptive step taken by us, the people modeling them. The key philosophical technology is the intentional stance.

(Yeah, I do feel like “read about the intentional stance” is this year’s “read the sequences”)

On to the meat of the post—agents are already very general, especially if you allow preferences over world-histories, at which point they become *really* general. Maybe it makes more sense to think of these things as languages in which some things are simple and others are complicated? At which point I think you have a straightforward distance function between languages (how surprising is one language one average to another), but no sense of equivalency aside from identical rankings.

To elaborate, A->B is an operation with a truth table:

The only thing that falsifies A->B is if A is true but B is false. This is different from how we usually think about implication, because it’s not like there’s any requirement that you can deduce B from A. It’s just a truth table.

But it is relevant to probability, because if A->B, then you’re not allowed to assign high probability to A but low probability to B.

EDIT: Anyhow I think that paragraph is a really quick and dirty way of phrasing the incompatibility of logical uncertainty with normal probability. The issue is that in normal probability, logical steps are things that are allowed to happen inside the parentheses of the P() function. No matter how complicated the proof of φ, as long as the proof follows logically from premises, you can’t doubt φ more than you doubt the premises, because the P() function thinks that P(premises) and P(logical equivalent of premises according to Boolean algebra) are “the same thing.”