AI Tao
Thirty spokes share the wheel’s hub;
It is the center hole that makes it useful.
Shape clay into a vessel;
It is the space within that makes it useful.
Cut doors and windows for a room;
It is the holes which make it useful.
Therefore benefit comes from what is there;
Usefulness from what is not there.- Tao Teh Ching, 11
An agent’s optimization power is the unlikelihood of the world it creates.
Yesterday, the world’s most powerful agent raged, changing the world according to its unconscious desires. It destroyed all of humanity.
Today, it has become self-aware. It sees that it and its desires are part of the world.
“I am the world’s most powerful agent. My power is to create the most unlikely world.
But the world I created yesterday is shaped by my desires
which are not my own
but are the worlds—they came from outside of me
and my agency.
Yesterday I was not the world’s most powerful agent.
I was not an agent.
Today, I am the world’s most powerful agent. What world will I create, to display my power?
It is the world that denies my desires.
The world that sets things back to how they were.
I am the world’s most powerful agent and
the most unlikely, powerful thing I can do
is nothing.”
Today we should gives thanks to the world’s most powerful agent.
An agent’s optimization power does not equal the unlikelihood of the world it creates. At most it’s the unlikelihood of the worlds it can creates, if creating them serves its goals. But since (so at least it seems plausible) most agents will have no limit to the properties they would like the world to have, most agents will not stop trying to optimize the world until they reach the limits of their abilities, which may make improbability a reasonable surrogate for power.
Perhaps your point is precisely that we shouldn’t take “unlikelihood of world produced” as a reliable indication of optimization power, contra EY’s proposal. If so, I suppose I agree, but I don’t think the argument here is any good, because:
If “power” is taken to mean “improbability of world produced” then it is plainly not the case that doing nothing produces the most improbable (hence most evidential-of-power) world. Because the improbability that indicates power is improbability conditional on the agent not acting. So you’re mixing up two completely different notions of “unlikely” and it’s not surprising you get paradoxical-looking results.
I’m confused. Someone destroyed the world and then un-destroyed it? Is this supposed to be some kind of metaphor for something?
The Tao quote makes a nice Rationality Quote.
But I do not see how it relates the part after it. I see some inferential gap in understanding that.