since the actual calculation of that fact takes more than one observer-moment, i.e. I can’t verify it all at once
As far as I can tell, this is not allowed by UDT 1.1[1].
(UDT 1.1 also needs a hard coded search order (iteration order) in order to self-cooperate properly, an additional reason why I don’t think humans can run it.)
As for other agents, Wei Dai’s original UDT 1.0 is logically omniscient in the sense that it will never notice its “thoughts” (such as they are) taking any noticeable amount of time. It features an “intuition module” for mathematics, but this is always exactly the same and has no appreciable origin. The agent definition hardcodes the mathematical intuition subroutine along with all the agent’s other code, with that including its prior and utility function. This means that any modification there results in an entirely different agent.
Any agent that takes time to think is going to be weaker than Wei Dai’s original designs, since it will have to keep state around, state that can be deleted or tampered with, such as is effectively the case with The Absent-Minded Driver[2]if it doesn’t have enough time at each intersection to recompute the plan from scratch. Plausibly, in Driver the policy may be fully forgotten (along with the agent’s current location) or incorrectly remembered/corrupted, leading a computationally weak UDT 1.0 agent to do badly (I don’t think computationally weak UDT 1.1 agents make much sense, at least without absurd ergodicity and memory assumptions).
Any additional state a weak agent may try to bring along would have to be re-checked at every point, making it useless, in the same sense that the “map” in the map explanation of UDT[3] must be recalculated from scratch at each time, otherwise it could be tampered with just like the agent’s memory and sense data. (Note that the unbounded agent reaches, at each step, the same map as the one it reaches at every other step (if it even has such a map at any point), so the prior is still all the epistemics it has.)
Slow agents would need more forgiving fairness criteria, maybe a list of previously used cryptographic public keys that the environment isn’t able to tamper with (only delete from). Assuming some things in computer science, the agent could check its previous state faster than just re-calculating it all. If ordering matters, Merkle chains can be used by the agent, with results written into its standard, tamper-vulnerable memory. Note that in each step, the agent would need to generate a private key, use it to sign what it thought about that step, add the corresponding public key to its public key list, and then delete the private key. The private keys would need to be opaque to the environment during the step, otherwise none of this accomplishes anything. Presumably, the agent would be required to have only one chance to generate a key pair each step, and only have the option to add that public key into the key list, otherwise it could try to brute-force a pattern into the public key it adds. Preventing this isn’t particularly realistic, but it’s a good research direction to avoid a trivial tiling result. (If the environment could tamper with the agent’s public key list, it could sign anything it wanted in the agent’s tampered-with memory using newly generated private keys and then write the corresponding public keys into the agent’s key list. Proof-of-work is a no-go because the environment is assumed to be stronger than the agent.)
As far as I can tell, this is not allowed by UDT 1.1[1].
(UDT 1.1 also needs a hard coded search order (iteration order) in order to self-cooperate properly, an additional reason why I don’t think humans can run it.)
As for other agents, Wei Dai’s original UDT 1.0 is logically omniscient in the sense that it will never notice its “thoughts” (such as they are) taking any noticeable amount of time. It features an “intuition module” for mathematics, but this is always exactly the same and has no appreciable origin. The agent definition hardcodes the mathematical intuition subroutine along with all the agent’s other code, with that including its prior and utility function. This means that any modification there results in an entirely different agent.
Any agent that takes time to think is going to be weaker than Wei Dai’s original designs, since it will have to keep state around, state that can be deleted or tampered with, such as is effectively the case with The Absent-Minded Driver[2] if it doesn’t have enough time at each intersection to recompute the plan from scratch. Plausibly, in Driver the policy may be fully forgotten (along with the agent’s current location) or incorrectly remembered/corrupted, leading a computationally weak UDT 1.0 agent to do badly (I don’t think computationally weak UDT 1.1 agents make much sense, at least without absurd ergodicity and memory assumptions).
Any additional state a weak agent may try to bring along would have to be re-checked at every point, making it useless, in the same sense that the “map” in the map explanation of UDT[3] must be recalculated from scratch at each time, otherwise it could be tampered with just like the agent’s memory and sense data.
(Note that the unbounded agent reaches, at each step, the same map as the one it reaches at every other step (if it even has such a map at any point), so the prior is still all the epistemics it has.)
Slow agents would need more forgiving fairness criteria, maybe a list of previously used cryptographic public keys that the environment isn’t able to tamper with (only delete from). Assuming some things in computer science, the agent could check its previous state faster than just re-calculating it all. If ordering matters, Merkle chains can be used by the agent, with results written into its standard, tamper-vulnerable memory. Note that in each step, the agent would need to generate a private key, use it to sign what it thought about that step, add the corresponding public key to its public key list, and then delete the private key. The private keys would need to be opaque to the environment during the step, otherwise none of this accomplishes anything. Presumably, the agent would be required to have only one chance to generate a key pair each step, and only have the option to add that public key into the key list, otherwise it could try to brute-force a pattern into the public key it adds. Preventing this isn’t particularly realistic, but it’s a good research direction to avoid a trivial tiling result.
(If the environment could tamper with the agent’s public key list, it could sign anything it wanted in the agent’s tampered-with memory using newly generated private keys and then write the corresponding public keys into the agent’s key list. Proof-of-work is a no-go because the environment is assumed to be stronger than the agent.)
https://www.lesswrong.com/posts/2ew3chEabxf8YySR5/functional-decision-theory-not-even-wrong-also-wrong?commentId=jv3hHkraTkHC6zqpp
https://www.lesswrong.com/posts/GfHdNfqxe3cSCfpHL/the-absent-minded-driver
https://www.lesswrong.com/posts/2ew3chEabxf8YySR5/functional-decision-theory-not-even-wrong-also-wrong?commentId=ecizaBhcp7rwSwe7D