Is the argument that firms run by homeostatic agents will outcompete firms run by consequentialist agents because homeostatic agents can more reliably follow long-term contracts?
I would phrase it as “the conditions under which homeostatic agents will renege on long-term contracts are more predictable than those under which consequentialist agents will do so”. Taking into account the actions of the counterparties would take to reduce the chance of such contract breaking, though, yes.
Homeostatic ones exclusively. I think the number of agents in the world as it exists today that behave as long-horizon consequentialists of the sort Eliezer and company seem to envision is either zero or very close to zero. FWIW I expect that most people in that camp would agree that no true consequentialist agents exist in the world as it currently is, but would disagree with my “and I expect that to remain true” assessment.
Edit: on reflection some corporations probably do behave more like unbounded infinite-horizon consequentialists in the sense that they have drives to acquire resources where acquiring those resources doesn’t reduce the intensity of the drive. This leads to behavior that in many cases would be the same behavior as an agent that was actually trying to maximize its future resources through any available means. And I have ever bought Chiquita bananas, so maybe not homeostatic agents exclusively.
Maybe different definitions are being used, can you list some people or institutions that you trade with which come to mind who you don’t think have long-term goals?
Again, homeostatic agents exhibit goal-directed behavior. “Unbounded consequentialist” was a poor choice of term to use for this on my part. Digging through the LW archives uncovered Nostalgebraist’s post Why Assume AGIs Will Optimize For Fixed Goals, which coins the term “wrapper-mind”.
When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted:
An AGI has a single terminal goal[1].
The goal is a fixed part of the AI’s structure. The internal dynamics of the AI, if left to their own devices, will never modify the goal.
The “outermost loop” of the AI’s internal dynamics is an optimization process aimed at the goal, or at least the AI behaves just as though this were true.
This “outermost loop” or “fixed-terminal-goal-directed wrapper” chooses which of the AI’s specific capabilities to deploy at any given time, and how to deploy it[2].
The AI’s capabilities will themselves involve optimization for sub-goals that are not the same as the goal, and they will optimize for them very powerfully (hence “capabilities”). But it is “not enough” that the AI merely be good at optimization-for-subgoals: it will also have a fixed-terminal-goal-directed wrapper.
In terms of which agents I trade with which do not have the wrapper structure, I will go from largest to smallest in terms of expenses
My country: I pay taxes to it. In return, I get a stable place to live with lots of services and opportunities. I don’t expect that I get these things because my country is trying to directly optimize for my well-being, or directly trying to optimize for any other specific unbounded goal. My country a FPTP democracy, the leaders do have drives to make sure that at least half of voters vote for them over the opposition—but once that “half” is satisfied, they don’t have a drive to get approval high as possible no matter what or maximize the time their party is in power or anything like that.
My landlord: He is renting the place to me because he wants money, and he wants money because it can be exchanged for goods and services, which can satisfy his drives for things like food and social status. I expect that if all of his money-satisfiable drives were satisfied, he would not seek to make money by renting the house out. I likewise don’t expect that there is any fixed terminal goal I could ascribe to him that would lead me to predict his behavior better than “he’s a guy with the standard set of human drives, and will seek to satisfy those drives”.
My bank: … you get the idea
Publicly traded companies do sort of have the wrapper structure from a legal perspective, but in terms of actual behavior they are usually (with notable exceptions) not asking “how do we maximize market cap” and then making explicit subgoals and subsubgoals with only that in mind.
on reflection some corporations probably do behave more like unbounded infinite-horizon consequentialists in the sense that they have drives to acquire resources where acquiring those resources doesn’t reduce the intensity of the drive. This leads to behavior that in many cases would be the same behavior as an agent that was actually trying to maximize its future resources through any available means. And I have ever bought Chiquita bananas, so maybe not homeostatic agents exclusively.
On average, do those corporations have more or less money or power than the heuristic based firms & individuals you trade with?
Is the argument that firms run by homeostatic agents will outcompete firms run by consequentialist agents because homeostatic agents can more reliably follow long-term contracts?
I would phrase it as “the conditions under which homeostatic agents will renege on long-term contracts are more predictable than those under which consequentialist agents will do so”. Taking into account the actions of the counterparties would take to reduce the chance of such contract breaking, though, yes.
Cool, I want to know also whether you think you’re currently (eg in day to day life) trading with consequentialist or homeostatic agents.
Homeostatic ones exclusively. I think the number of agents in the world as it exists today that behave as long-horizon consequentialists of the sort Eliezer and company seem to envision is either zero or very close to zero. FWIW I expect that most people in that camp would agree that no true consequentialist agents exist in the world as it currently is, but would disagree with my “and I expect that to remain true” assessment.
Edit: on reflection some corporations probably do behave more like unbounded infinite-horizon consequentialists in the sense that they have drives to acquire resources where acquiring those resources doesn’t reduce the intensity of the drive. This leads to behavior that in many cases would be the same behavior as an agent that was actually trying to maximize its future resources through any available means. And I have ever bought Chiquita bananas, so maybe not homeostatic agents exclusively.
I think this is false, eg John Wentworth often gives Ben Pace as a prototypical example of a consequentialist agent. [EDIT]: Also Eliezer talks about consequentialism being “ubiquitous”.
Maybe different definitions are being used, can you list some people or institutions that you trade with which come to mind who you don’t think have long-term goals?
Again, homeostatic agents exhibit goal-directed behavior. “Unbounded consequentialist” was a poor choice of term to use for this on my part. Digging through the LW archives uncovered Nostalgebraist’s post Why Assume AGIs Will Optimize For Fixed Goals, which coins the term “wrapper-mind”.
In terms of which agents I trade with which do not have the wrapper structure, I will go from largest to smallest in terms of expenses
My country: I pay taxes to it. In return, I get a stable place to live with lots of services and opportunities. I don’t expect that I get these things because my country is trying to directly optimize for my well-being, or directly trying to optimize for any other specific unbounded goal. My country a FPTP democracy, the leaders do have drives to make sure that at least half of voters vote for them over the opposition—but once that “half” is satisfied, they don’t have a drive to get approval high as possible no matter what or maximize the time their party is in power or anything like that.
My landlord: He is renting the place to me because he wants money, and he wants money because it can be exchanged for goods and services, which can satisfy his drives for things like food and social status. I expect that if all of his money-satisfiable drives were satisfied, he would not seek to make money by renting the house out. I likewise don’t expect that there is any fixed terminal goal I could ascribe to him that would lead me to predict his behavior better than “he’s a guy with the standard set of human drives, and will seek to satisfy those drives”.
My bank: … you get the idea
Publicly traded companies do sort of have the wrapper structure from a legal perspective, but in terms of actual behavior they are usually (with notable exceptions) not asking “how do we maximize market cap” and then making explicit subgoals and subsubgoals with only that in mind.
Yeah seems reasonable. You link the enron scandal, on your view do all unbounded consequentialists die in such a scandal or similar?
On average, do those corporations have more or less money or power than the heuristic based firms & individuals you trade with?