Vladimir, if I understand both you and Eliezer correctly you’re saying that Eliezer is saying not “intelligence is reality-steering ability” but “intelligence is reality-steering ability modulo available resources”. That makes good sense, but that definition is only usable in so far as you have some separate way of estimating an agent’s available resources, and comparing the utility of what might be very different sets of available resources. (Compare a nascent superintelligent AI, with no ability to influence the world directly other than by communicating with people, with someone carrying a whole lot of powerful weapons. Who has the better available resources? Depends on context—and on the intelligence of the two.) Eliezer, I think, is proposing a way of evaluating the “intelligence” of an agent about which we know very little, including (perhaps) very little about what resources it has.
Put differently: I think Eliezer’s given a definition of “intelligence” that could equally be given as a definition of “power”, and I suspect that in practice using it to evaluate intelligence involves applying some other notion of what counts as intelligence and what counts as something else. (E.g., we’ve already decided that how much money you have, or how many nuclear warheads you have at your command, don’t count as “intelligence”.)
If you have two agent source codes A and B both provided with resource amount R (detailing computational architecture and tools such as telepresence robots and nanotech) then you will observe that if A is a stronger optimizer than B, A will get more done in less time, or alternately hit a much higher preferred world.
If you have one agent source code C and run two instances, one with resources M and one with resources N, then if M > N, then agent running on M will dominate the agent running on N.
“Intelligence” is cleverness of source code, “Power” is the available resources. A really clever agent can outdo a stupid one even if ludicrously handicapped, resource wise. A stupid agent with powerful nanotech dominates a clever agent with human servants.
Vladimir, if I understand both you and Eliezer correctly you’re saying that Eliezer is saying not “intelligence is reality-steering ability” but “intelligence is reality-steering ability modulo available resources”. That makes good sense, but that definition is only usable in so far as you have some separate way of estimating an agent’s available resources, and comparing the utility of what might be very different sets of available resources. (Compare a nascent superintelligent AI, with no ability to influence the world directly other than by communicating with people, with someone carrying a whole lot of powerful weapons. Who has the better available resources? Depends on context—and on the intelligence of the two.) Eliezer, I think, is proposing a way of evaluating the “intelligence” of an agent about which we know very little, including (perhaps) very little about what resources it has.
Put differently: I think Eliezer’s given a definition of “intelligence” that could equally be given as a definition of “power”, and I suspect that in practice using it to evaluate intelligence involves applying some other notion of what counts as intelligence and what counts as something else. (E.g., we’ve already decided that how much money you have, or how many nuclear warheads you have at your command, don’t count as “intelligence”.)
If you have two agent source codes A and B both provided with resource amount R (detailing computational architecture and tools such as telepresence robots and nanotech) then you will observe that if A is a stronger optimizer than B, A will get more done in less time, or alternately hit a much higher preferred world.
If you have one agent source code C and run two instances, one with resources M and one with resources N, then if M > N, then agent running on M will dominate the agent running on N.
“Intelligence” is cleverness of source code, “Power” is the available resources. A really clever agent can outdo a stupid one even if ludicrously handicapped, resource wise. A stupid agent with powerful nanotech dominates a clever agent with human servants.