But the economy is ultimately about what humans value.
I don’t see how this contradicts the notion that market and capitalism forces will drive people to cede control to AIs, which you acknowledge a few paragraphs earlier.
The dynamic can change. The economy is currently about what humans value, but if sufficiently capable and independent non-human agents exist, the economy will ultimately also be about what they value. And at some capability level, they could produce most of the value, and carry most of the weight.
Even if humans remain nominally in charge, if AI systems are better decision makers, market forces will drive people to cede decision making to them. You could imagine a world where 99% of the resource utilization goes to growing the economy, and 1% is distributed to humans (which may resemble the current world). If the humans are uninvolved in the decisions about where and how to expand, in what sense is the economy about them?
It’s a choice what level of independence or autonomy to give to agents. I think we can make the choice of a human centered world, where the economy remains about what humans value. And as more of their material needs are satisfied, what humans value might move more towards relations with other humans.
My point is that there is single objective function that corresponds to “making the economy better”- humans get to choose their own objectives. Yes, if our goal was to advance on the Kardashev scale as fast as possible, and maximize energy production, then it might make sense to just hand over control to the AIs. But we don’t have to make it our goal.
Not all goals have equal fitness, some are self reinforcing. Your goal may be to have great relationships, my goal may be to accumulate money and power. My thing compounds. Reproduction, growth, and propagation feed themselves.
If energy and intelligence are the currency, someone will task an AI system to make more of those things and reinvest.
We could coordinate to suppress those dynamics, but that coordination seems harder in the decentralized futures. I agree that it’s desirable but I don’t see how that world persists.
Do you agree that you’re not describing a stable state? Or is there a stabilizing component that I’m missing?
Looking for examples in nature, you might need something like an immune system auditing nodes in the network to make sure they’re not cancerous. Empathy /deontological tendencies in humans may be another example of a stabilizer. Otherwise a garden without a gardener will become all weeds.
I don’t see how this contradicts the notion that market and capitalism forces will drive people to cede control to AIs, which you acknowledge a few paragraphs earlier.
The dynamic can change. The economy is currently about what humans value, but if sufficiently capable and independent non-human agents exist, the economy will ultimately also be about what they value. And at some capability level, they could produce most of the value, and carry most of the weight.
Even if humans remain nominally in charge, if AI systems are better decision makers, market forces will drive people to cede decision making to them. You could imagine a world where 99% of the resource utilization goes to growing the economy, and 1% is distributed to humans (which may resemble the current world). If the humans are uninvolved in the decisions about where and how to expand, in what sense is the economy about them?
It’s a choice what level of independence or autonomy to give to agents. I think we can make the choice of a human centered world, where the economy remains about what humans value. And as more of their material needs are satisfied, what humans value might move more towards relations with other humans.
My point is that there is single objective function that corresponds to “making the economy better”- humans get to choose their own objectives. Yes, if our goal was to advance on the Kardashev scale as fast as possible, and maximize energy production, then it might make sense to just hand over control to the AIs. But we don’t have to make it our goal.
Not all goals have equal fitness, some are self reinforcing. Your goal may be to have great relationships, my goal may be to accumulate money and power. My thing compounds. Reproduction, growth, and propagation feed themselves.
If energy and intelligence are the currency, someone will task an AI system to make more of those things and reinvest.
We could coordinate to suppress those dynamics, but that coordination seems harder in the decentralized futures. I agree that it’s desirable but I don’t see how that world persists.
Do you agree that you’re not describing a stable state? Or is there a stabilizing component that I’m missing?
Looking for examples in nature, you might need something like an immune system auditing nodes in the network to make sure they’re not cancerous. Empathy /deontological tendencies in humans may be another example of a stabilizer. Otherwise a garden without a gardener will become all weeds.