The most possible way acausal trade turns out to be infeasible is probably like ascertaining that other party has decision procedure that is actually dependent on yours. And finding such parties, that would in turn look back at you.
Like, let’s say the basic case for the trade is like that. You want A to exist and have B-suitable opportunity. You can make A out of it, but at 0.5 efficiency. Someone else has this situation but reversed, they want B to exist and have A-suitable opportunity.
You can trade and both will be better off. Of course even better would be the situation when you look like you are going do your end of the deal, but then not do it, you get 150% out of that.
So, you should do your end of the deal only if that makes the other party to switch from not-A to A, with enough credence that gain outweighs the cost.
But, how do you ascertain that? That sounds super difficult, and there incentives to be deceptive in the impression you give off.
Okay, here is one way to look at this.
Let’s say there are large agents that you can’t effectively simulate, because their policies depend on large fraction of compute you yourself have. But some of them were not always like that, some of them were small and then grew. You probably can exploit that? Like, you can look into them and see that they commit to become particular type of large agent. So, you can do actions, that their later self would be able to see, even if they can’t now.
Small visible agent commits to something → You in mid stage see them and do your part → Small agent becomes large and able too see you in mid stage, and verifies your execution, and does its part too.
It’s plausible a lot of this stuff demands very early commitments, and sprouts from that. Of course as you are small, you can make very bad choices, so you should do less choices. Unfortunately.
Practical coordination between big agents (and their much bigger worlds) uses small contracts. A contract is a computation that both parties unconditionally yield some influence to (in a way that depends on some abstract properties of the contract, but not specifically on what the contract ends up doing), and that computation decides what to do in each situation. Since it’s the same computation, it can reason updatelessly across possible/hypothetical situations about how its behavior should depend on the situation.
When causal communication is possible, contracts can be given explicitly. Some examples of varying levels of legibility and intentionality in design are laws, social norms, moral principles, and deontological rules. These things don’t take over all behavior of a person, just some narrow aspect of it. But they do so in the same way across many people, thus coordinating them, the way UDT coordinates the behavior of a single agent across possible situations.
Without causal communication, different agents can probably find shared contracts in algorithmic priors (maybe the way deep learning finds circuits in training data, coordinating a model with the world). The practical difficulty is that a contract, in its role as an updateless core (updatelessly choosing a policy, which is to respond to possible situations/contexts), needs to reason about the space of possible situations in sufficient detail to coordinate potential trading partners, and that reasoning needs to be observed by these potential trading partners in order for them to access its computed conclusions/actions (for their particular situation). But like a contract doesn’t get to control behaviors in full detail, just in some aspect, it similarly doesn’t need to be aware of the situations (of the potential trading partners) in full detail.
I don’t understand how the contract formulation works here.
Here is a concrete case, albeit with all the burden of prediction on one participant. Through you need to somehow get high credence that they work like they say they work, maybe by eg reputation.
It’s really late and our civ starts splitting into causally disconnected regions. You become aware that in one such region that split before you were born there is Omega that will donate $1m to charity if you burn $1000 and donate only $1000 if you don’t. Suppose you are charitably inclined kind of guy. Is it in your interest to set $1000 on fire?
What is the contract? Like, did contract reason about the situations where it exists or what? Like, is contract formulation helpful here at all?
EDIT Like, you can make counterpoint, what if large inscrutable agent makes small understandable agent, and gives it some of the power? -- I don’t think this works, as this agent would be downstream from large agent. I think it’s just as difficult to figure if it worth it to bite its proposals, as to figure out how to take large agent head on.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things. Results of a computation are determined only by the computation itself, not by the reasons it got instantiated (23+17 is always 40, no matter who computes it or why, and even if they compute it incorrectly it’s still actually 40). The reasons it got instantiated determine whether it gets influence in a given situation, but they don’t determine the content of its behavior (in the sense of a general policy, a mapping from observations to actions).
If you do know there’s an Omega, this fact (of Omega, and of your having this knowledge) is the contract, and the reasons it got instantiated in your knowledge are here screened off by the stipulation that the knowledge is somehow already there. This knowledge tells you all you need to conclude that burning the $1000 is the way to go, that’s the way the fact/contract controls your behavior. The fact/contract also tells Omega to donate the $1m, since part of the fact is that you have the knowledge. So the fact coordinates the both of you. (You shouldn’t burn the $1000 on account of the contract if the fact doesn’t say it’s the right decision; and Omega doesn’t donate the $1m if that’s what you conclude. The influence of the contract shouldn’t depend on what you’ll conclude, it should be given in advance of looking at what you’ll conclude.)
The contract formulation doesn’t seem very helpful here, it’s more helpful when a contract is more of an agent, when it needs to look at a situation and think what to do (even like an actual legal contract). But it illustrates how a contract should be much simpler than a big agent or their world, how acausal trade doesn’t require big agents to simulate each other in detail, or each other’s worlds in detail. This inevitably means that in practice the counterparty isn’t concrete, not a specific agent from a specific world, and a contract acts through a large or infinite number of formulations of possible situations (counterparties following the contract) that it would consider as possible points of instantiation. (Correspondingly, there isn’t a single contract that’s worth focusing on, but a mixture of many contracts.) It would then need to reason about these possible situations updatelessly in advance, in order to decide on general principles that coordinate its behavior across these situations, which is the value proposition of contracts/norms/integrity as a coordination technology.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things.
Like, the question is, how do you figure that this giant messy agent will actually follow that contract? Instead of laughing at you when you waste resources on following that contract? The point is, which actions it will perform.
In iterative causal interactions you can use them as choke points, you check what kind of contract I proposed, what kind of delegate bot I sent to you, but when you deal with acausal case, the whole situation chains without choke points.
I have like literally no idea how the notion of contracts helps here.
how do you figure that this giant messy agent will actually follow that contract?
First of all, it’s only useful to figure out to the extent the contract itself can figure it out, otherwise it won’t be able to use that knowledge to coordinate (act depending on how the points of instantiation relate to each other). So your part is just following the contract, not figuring out whether others do. When you are figuring out who is following the contract (if even you are following the contract) on behalf of the contract, as part of considering its instantiation, you need to do so updatelessly, without taking into account the facts of your own situation or knowledge. This corresponds to the various veil of ignorance setups, it’s just updateless reasoning (crucially, contract’s own values don’t need to be the same as your values or anyone else’s values).
There isn’t a specific giant messy agent counterparty that you need to understand (or that the contract needs to understand), it instead needs to understand the whole class of such agents, presented in a way relevant to the contract. For example, it could matter if there are a lot of such agents (something an assurance contract would care about, including an acausal assurance contract), without it being relevant who they are in detail, or whether a specific agent is one of them.
For acausal coordination, it’s probably not useful to consider a specific contract, just like an LLM isn’t about a specific circuit. Instead, you consider (and give influence to) many useful contracts at once, like an LLM considers many circuits at once (and gives them influence over the logits of the output tokens).
Okay. A bit too abstract for me. I’d think the general form should be generalization on top of particular cases where the general form seems to succinctly resolve them. Your way of thinking is kind of unmoored from particulars.
The most possible way acausal trade turns out to be infeasible is probably like ascertaining that other party has decision procedure that is actually dependent on yours. And finding such parties, that would in turn look back at you.
Like, let’s say the basic case for the trade is like that. You want A to exist and have B-suitable opportunity. You can make A out of it, but at 0.5 efficiency. Someone else has this situation but reversed, they want B to exist and have A-suitable opportunity.
You can trade and both will be better off. Of course even better would be the situation when you look like you are going do your end of the deal, but then not do it, you get 150% out of that.
So, you should do your end of the deal only if that makes the other party to switch from not-A to A, with enough credence that gain outweighs the cost.
But, how do you ascertain that? That sounds super difficult, and there incentives to be deceptive in the impression you give off.
Okay, here is one way to look at this.
Let’s say there are large agents that you can’t effectively simulate, because their policies depend on large fraction of compute you yourself have. But some of them were not always like that, some of them were small and then grew. You probably can exploit that? Like, you can look into them and see that they commit to become particular type of large agent. So, you can do actions, that their later self would be able to see, even if they can’t now.
Small visible agent commits to something → You in mid stage see them and do your part → Small agent becomes large and able too see you in mid stage, and verifies your execution, and does its part too.
It’s plausible a lot of this stuff demands very early commitments, and sprouts from that. Of course as you are small, you can make very bad choices, so you should do less choices. Unfortunately.
Also something something successor alignment.
Practical coordination between big agents (and their much bigger worlds) uses small contracts. A contract is a computation that both parties unconditionally yield some influence to (in a way that depends on some abstract properties of the contract, but not specifically on what the contract ends up doing), and that computation decides what to do in each situation. Since it’s the same computation, it can reason updatelessly across possible/hypothetical situations about how its behavior should depend on the situation.
When causal communication is possible, contracts can be given explicitly. Some examples of varying levels of legibility and intentionality in design are laws, social norms, moral principles, and deontological rules. These things don’t take over all behavior of a person, just some narrow aspect of it. But they do so in the same way across many people, thus coordinating them, the way UDT coordinates the behavior of a single agent across possible situations.
Without causal communication, different agents can probably find shared contracts in algorithmic priors (maybe the way deep learning finds circuits in training data, coordinating a model with the world). The practical difficulty is that a contract, in its role as an updateless core (updatelessly choosing a policy, which is to respond to possible situations/contexts), needs to reason about the space of possible situations in sufficient detail to coordinate potential trading partners, and that reasoning needs to be observed by these potential trading partners in order for them to access its computed conclusions/actions (for their particular situation). But like a contract doesn’t get to control behaviors in full detail, just in some aspect, it similarly doesn’t need to be aware of the situations (of the potential trading partners) in full detail.
I don’t understand how the contract formulation works here.
Here is a concrete case, albeit with all the burden of prediction on one participant. Through you need to somehow get high credence that they work like they say they work, maybe by eg reputation.
It’s really late and our civ starts splitting into causally disconnected regions. You become aware that in one such region that split before you were born there is Omega that will donate $1m to charity if you burn $1000 and donate only $1000 if you don’t. Suppose you are charitably inclined kind of guy. Is it in your interest to set $1000 on fire?
What is the contract? Like, did contract reason about the situations where it exists or what? Like, is contract formulation helpful here at all?
EDIT Like, you can make counterpoint, what if large inscrutable agent makes small understandable agent, and gives it some of the power? -- I don’t think this works, as this agent would be downstream from large agent. I think it’s just as difficult to figure if it worth it to bite its proposals, as to figure out how to take large agent head on.
What a small computation does is not downstream of a large agent that runs it and gives it authority to do things. Results of a computation are determined only by the computation itself, not by the reasons it got instantiated (23+17 is always 40, no matter who computes it or why, and even if they compute it incorrectly it’s still actually 40). The reasons it got instantiated determine whether it gets influence in a given situation, but they don’t determine the content of its behavior (in the sense of a general policy, a mapping from observations to actions).
If you do know there’s an Omega, this fact (of Omega, and of your having this knowledge) is the contract, and the reasons it got instantiated in your knowledge are here screened off by the stipulation that the knowledge is somehow already there. This knowledge tells you all you need to conclude that burning the $1000 is the way to go, that’s the way the fact/contract controls your behavior. The fact/contract also tells Omega to donate the $1m, since part of the fact is that you have the knowledge. So the fact coordinates the both of you. (You shouldn’t burn the $1000 on account of the contract if the fact doesn’t say it’s the right decision; and Omega doesn’t donate the $1m if that’s what you conclude. The influence of the contract shouldn’t depend on what you’ll conclude, it should be given in advance of looking at what you’ll conclude.)
The contract formulation doesn’t seem very helpful here, it’s more helpful when a contract is more of an agent, when it needs to look at a situation and think what to do (even like an actual legal contract). But it illustrates how a contract should be much simpler than a big agent or their world, how acausal trade doesn’t require big agents to simulate each other in detail, or each other’s worlds in detail. This inevitably means that in practice the counterparty isn’t concrete, not a specific agent from a specific world, and a contract acts through a large or infinite number of formulations of possible situations (counterparties following the contract) that it would consider as possible points of instantiation. (Correspondingly, there isn’t a single contract that’s worth focusing on, but a mixture of many contracts.) It would then need to reason about these possible situations updatelessly in advance, in order to decide on general principles that coordinate its behavior across these situations, which is the value proposition of contracts/norms/integrity as a coordination technology.
Like, the question is, how do you figure that this giant messy agent will actually follow that contract? Instead of laughing at you when you waste resources on following that contract? The point is, which actions it will perform.
In iterative causal interactions you can use them as choke points, you check what kind of contract I proposed, what kind of delegate bot I sent to you, but when you deal with acausal case, the whole situation chains without choke points.
I have like literally no idea how the notion of contracts helps here.
First of all, it’s only useful to figure out to the extent the contract itself can figure it out, otherwise it won’t be able to use that knowledge to coordinate (act depending on how the points of instantiation relate to each other). So your part is just following the contract, not figuring out whether others do. When you are figuring out who is following the contract (if even you are following the contract) on behalf of the contract, as part of considering its instantiation, you need to do so updatelessly, without taking into account the facts of your own situation or knowledge. This corresponds to the various veil of ignorance setups, it’s just updateless reasoning (crucially, contract’s own values don’t need to be the same as your values or anyone else’s values).
There isn’t a specific giant messy agent counterparty that you need to understand (or that the contract needs to understand), it instead needs to understand the whole class of such agents, presented in a way relevant to the contract. For example, it could matter if there are a lot of such agents (something an assurance contract would care about, including an acausal assurance contract), without it being relevant who they are in detail, or whether a specific agent is one of them.
For acausal coordination, it’s probably not useful to consider a specific contract, just like an LLM isn’t about a specific circuit. Instead, you consider (and give influence to) many useful contracts at once, like an LLM considers many circuits at once (and gives them influence over the logits of the output tokens).
Okay. A bit too abstract for me. I’d think the general form should be generalization on top of particular cases where the general form seems to succinctly resolve them. Your way of thinking is kind of unmoored from particulars.