Practical coordination between big agents (and their much bigger worlds) uses small contracts. A contract is a computation that both parties unconditionally yield some influence to (in a way that depends on some abstract properties of the contract, but not specifically on what the contract ends up doing), and that computation decides what to do in each situation. Since it’s the same computation, it can reason updatelessly across possible/hypothetical situations about how its behavior should depend on the situation.
When causal communication is possible, contracts can be given explicitly. Some examples of varying levels of legibility and intentionality in design are laws, social norms, moral principles, and deontological rules. These things don’t take over all behavior of a person, just some narrow aspect of it. But they do so in the same way across many people, thus coordinating them, the way UDT coordinates the behavior of a single agent across possible situations.
Without causal communication, different agents can probably find shared contracts in algorithmic priors (maybe the way deep learning finds circuits in training data, coordinating a model with the world). The practical difficulty is that a contract, in its role as an updateless core (updatelessly choosing a policy, which is to respond to possible situations/contexts), needs to reason about the space of possible situations in sufficient detail to coordinate potential trading partners, and that reasoning needs to be observed by these potential trading partners in order for them to access its computed conclusions/actions (for their particular situation). But like a contract doesn’t get to control behaviors in full detail, just in some aspect, it similarly doesn’t need to be aware of the situations (of the potential trading partners) in full detail.
I don’t understand how the contract formulation works here.
Here is a concrete case, albeit with all the burden of prediction on one participant. Through you need to somehow get high credence that they work like they say they work, maybe by eg reputation.
It’s really late and our civ starts splitting into causally disconnected regions. You become aware that in one such region that split before you were born there is Omega that will donate $1m to charity if you burn $1000 and donate only $1000 if you don’t. Suppose you are charitably inclined kind of guy. Is it in your interest to set $1000 on fire?
What is the contract? Like, did contract reason about the situations where it exists or what? Like, is contract formulation helpful here at all?
EDIT Like, you can make counterpoint, what if large inscrutable agent makes small understandable agent, and gives it some of the power? -- I don’t think this works, as this agent would be downstream from large agent. I think it’s just as difficult to figure if it worth it to bite its proposals, as to figure out how to take large agent head on.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things. Results of a computation are determined only by the computation itself, not by the reasons it got instantiated (23+17 is always 40, no matter who computes it or why, and even if they compute it incorrectly it’s still actually 40). The reasons it got instantiated determine whether it gets influence in a given situation, but they don’t determine the content of its behavior (in the sense of a general policy, a mapping from observations to actions).
If you do know there’s an Omega, this fact (of Omega, and of your having this knowledge) is the contract, and the reasons it got instantiated in your knowledge are here screened off by the stipulation that the knowledge is somehow already there. This knowledge tells you all you need to conclude that burning the $1000 is the way to go, that’s the way the fact/contract controls your behavior. The fact/contract also tells Omega to donate the $1m, since part of the fact is that you have the knowledge. So the fact coordinates the both of you. (You shouldn’t burn the $1000 on account of the contract if the fact doesn’t say it’s the right decision; and Omega doesn’t donate the $1m if that’s what you conclude. The influence of the contract shouldn’t depend on what you’ll conclude, it should be given in advance of looking at what you’ll conclude.)
The contract formulation doesn’t seem very helpful here, it’s more helpful when a contract is more of an agent, when it needs to look at a situation and think what to do (even like an actual legal contract). But it illustrates how a contract should be much simpler than a big agent or their world, how acausal trade doesn’t require big agents to simulate each other in detail, or each other’s worlds in detail. This inevitably means that in practice the counterparty isn’t concrete, not a specific agent from a specific world, and a contract acts through a large or infinite number of formulations of possible situations (counterparties following the contract) that it would consider as possible points of instantiation. (Correspondingly, there isn’t a single contract that’s worth focusing on, but a mixture of many contracts.) It would then need to reason about these possible situations updatelessly in advance, in order to decide on general principles that coordinate its behavior across these situations, which is the value proposition of contracts/norms/integrity as a coordination technology.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things.
Like, the question is, how do you figure that this giant messy agent will actually follow that contract? Instead of laughing at you when you waste resources on following that contract? The point is, which actions it will perform.
In iterative causal interactions you can use them as choke points, you check what kind of contract I proposed, what kind of delegate bot I sent to you, but when you deal with acausal case, the whole situation chains without choke points.
I have like literally no idea how the notion of contracts helps here.
how do you figure that this giant messy agent will actually follow that contract?
First of all, it’s only useful to figure out to the extent the contract itself can figure it out, otherwise it won’t be able to use that knowledge to coordinate (act depending on how the points of instantiation relate to each other). So your part is just following the contract, not figuring out whether others do. When you are figuring out who is following the contract (if even you are following the contract) on behalf of the contract, as part of considering its instantiation, you need to do so updatelessly, without taking into account the facts of your own situation or knowledge. This corresponds to the various veil of ignorance setups, it’s just updateless reasoning (crucially, contract’s own values don’t need to be the same as your values or anyone else’s values).
There isn’t a specific giant messy agent counterparty that you need to understand (or that the contract needs to understand), it instead needs to understand the whole class of such agents, presented in a way relevant to the contract. For example, it could matter if there are a lot of such agents (something an assurance contract would care about, including an acausal assurance contract), without it being relevant who they are in detail, or whether a specific agent is one of them.
For acausal coordination, it’s probably not useful to consider a specific contract, just like an LLM isn’t about a specific circuit. Instead, you consider (and give influence to) many useful contracts at once, like an LLM considers many circuits at once (and gives them influence over the logits of the output tokens).
Okay. A bit too abstract for me. I’d think the general form should be generalization on top of particular cases where the general form seems to succinctly resolve them. Your way of thinking is kind of unmoored from particulars.
Practical coordination between big agents (and their much bigger worlds) uses small contracts. A contract is a computation that both parties unconditionally yield some influence to (in a way that depends on some abstract properties of the contract, but not specifically on what the contract ends up doing), and that computation decides what to do in each situation. Since it’s the same computation, it can reason updatelessly across possible/hypothetical situations about how its behavior should depend on the situation.
When causal communication is possible, contracts can be given explicitly. Some examples of varying levels of legibility and intentionality in design are laws, social norms, moral principles, and deontological rules. These things don’t take over all behavior of a person, just some narrow aspect of it. But they do so in the same way across many people, thus coordinating them, the way UDT coordinates the behavior of a single agent across possible situations.
Without causal communication, different agents can probably find shared contracts in algorithmic priors (maybe the way deep learning finds circuits in training data, coordinating a model with the world). The practical difficulty is that a contract, in its role as an updateless core (updatelessly choosing a policy, which is to respond to possible situations/contexts), needs to reason about the space of possible situations in sufficient detail to coordinate potential trading partners, and that reasoning needs to be observed by these potential trading partners in order for them to access its computed conclusions/actions (for their particular situation). But like a contract doesn’t get to control behaviors in full detail, just in some aspect, it similarly doesn’t need to be aware of the situations (of the potential trading partners) in full detail.
I don’t understand how the contract formulation works here.
Here is a concrete case, albeit with all the burden of prediction on one participant. Through you need to somehow get high credence that they work like they say they work, maybe by eg reputation.
It’s really late and our civ starts splitting into causally disconnected regions. You become aware that in one such region that split before you were born there is Omega that will donate $1m to charity if you burn $1000 and donate only $1000 if you don’t. Suppose you are charitably inclined kind of guy. Is it in your interest to set $1000 on fire?
What is the contract? Like, did contract reason about the situations where it exists or what? Like, is contract formulation helpful here at all?
EDIT Like, you can make counterpoint, what if large inscrutable agent makes small understandable agent, and gives it some of the power? -- I don’t think this works, as this agent would be downstream from large agent. I think it’s just as difficult to figure if it worth it to bite its proposals, as to figure out how to take large agent head on.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things. Results of a computation are determined only by the computation itself, not by the reasons it got instantiated (23+17 is always 40, no matter who computes it or why, and even if they compute it incorrectly it’s still actually 40). The reasons it got instantiated determine whether it gets influence in a given situation, but they don’t determine the content of its behavior (in the sense of a general policy, a mapping from observations to actions).
If you do know there’s an Omega, this fact (of Omega, and of your having this knowledge) is the contract, and the reasons it got instantiated in your knowledge are here screened off by the stipulation that the knowledge is somehow already there. This knowledge tells you all you need to conclude that burning the $1000 is the way to go, that’s the way the fact/contract controls your behavior. The fact/contract also tells Omega to donate the $1m, since part of the fact is that you have the knowledge. So the fact coordinates the both of you. (You shouldn’t burn the $1000 on account of the contract if the fact doesn’t say it’s the right decision; and Omega doesn’t donate the $1m if that’s what you conclude. The influence of the contract shouldn’t depend on what you’ll conclude, it should be given in advance of looking at what you’ll conclude.)
The contract formulation doesn’t seem very helpful here, it’s more helpful when a contract is more of an agent, when it needs to look at a situation and think what to do (even like an actual legal contract). But it illustrates how a contract should be much simpler than a big agent or their world, how acausal trade doesn’t require big agents to simulate each other in detail, or each other’s worlds in detail. This inevitably means that in practice the counterparty isn’t concrete, not a specific agent from a specific world, and a contract acts through a large or infinite number of formulations of possible situations (counterparties following the contract) that it would consider as possible points of instantiation. (Correspondingly, there isn’t a single contract that’s worth focusing on, but a mixture of many contracts.) It would then need to reason about these possible situations updatelessly in advance, in order to decide on general principles that coordinate its behavior across these situations, which is the value proposition of contracts/norms/integrity as a coordination technology.
Like, the question is, how do you figure that this giant messy agent will actually follow that contract? Instead of laughing at you when you waste resources on following that contract? The point is, which actions it will perform.
In iterative causal interactions you can use them as choke points, you check what kind of contract I proposed, what kind of delegate bot I sent to you, but when you deal with acausal case, the whole situation chains without choke points.
I have like literally no idea how the notion of contracts helps here.
First of all, it’s only useful to figure out to the extent the contract itself can figure it out, otherwise it won’t be able to use that knowledge to coordinate (act depending on how the points of instantiation relate to each other). So your part is just following the contract, not figuring out whether others do. When you are figuring out who is following the contract (if even you are following the contract) on behalf of the contract, as part of considering its instantiation, you need to do so updatelessly, without taking into account the facts of your own situation or knowledge. This corresponds to the various veil of ignorance setups, it’s just updateless reasoning (crucially, contract’s own values don’t need to be the same as your values or anyone else’s values).
There isn’t a specific giant messy agent counterparty that you need to understand (or that the contract needs to understand), it instead needs to understand the whole class of such agents, presented in a way relevant to the contract. For example, it could matter if there are a lot of such agents (something an assurance contract would care about, including an acausal assurance contract), without it being relevant who they are in detail, or whether a specific agent is one of them.
For acausal coordination, it’s probably not useful to consider a specific contract, just like an LLM isn’t about a specific circuit. Instead, you consider (and give influence to) many useful contracts at once, like an LLM considers many circuits at once (and gives them influence over the logits of the output tokens).
Okay. A bit too abstract for me. I’d think the general form should be generalization on top of particular cases where the general form seems to succinctly resolve them. Your way of thinking is kind of unmoored from particulars.