The goal of functional decision theory is to maximize the utility of all agents with similar policies to your own, weighted by exp(-KL(their policy||your policy)).
Huh, may I have a source on this? I thought you could point FDT at maximizing any utility function you like.
The issue with saying, “this agent,” is you do not actually know its policy. The best anyone can do is generate all programs that output the seen distribution of actions, using error-correction codes for nondeterministic policies. Now you have many theories of varying description lengths for the agent, which you weight according to the Solomonoff prior. We can always describe another agent’s policy with a fixed KL(their policy||your policy) extra error-correction bits, so the utils under a given theory are
Why do you need to know the policy in order to figure out the utility function? I thought you could point FDT at, like, maximizing Chaitin’s constant. I am hoping to look at whatever reference document you are getting your definitions from, is there no such thing?
Huh, may I have a source on this? I thought you could point FDT at maximizing any utility function you like.
The issue with saying, “this agent,” is you do not actually know its policy. The best anyone can do is generate all programs that output the seen distribution of actions, using error-correction codes for nondeterministic policies. Now you have many theories of varying description lengths for the agent, which you weight according to the Solomonoff prior. We can always describe another agent’s policy with a fixed KL(their policy||your policy) extra error-correction bits, so the utils under a given theory are
sum_{policy} exp(-|theory| - KL(policy||your policy)) utils(policy)
and the total utils are
sum_{theory} sum_{policy} … = constant * sum_{policy} exp(-KL(policy||your policy) utils(policy)
I assume you mean Arithmetic coding.
Why do you need to know the policy in order to figure out the utility function? I thought you could point FDT at, like, maximizing Chaitin’s constant. I am hoping to look at whatever reference document you are getting your definitions from, is there no such thing?
There is no such thing.