The “meat” is clearly implementing a computation of the type I described, whereas a tree or rock isn’t. Do you dispute that?
I not so much dispute that as don’t know of a way to make this judgment precise.
Your point seems to be that part of FAI theory should be a general and rigorous theory of how to extract preferences from any given object. Only then could we have sufficient theoretical support for any specific procedures for extracting preferences from human beings.
Right, although I’m not sure that “objects” are the right scope of such theory. I suspect that you also need enough subjective specification of preference to initiate the process of interpretation (preference-extraction). This will make preference of rocks arbitrary, because the process of their interpretation can start in too many arbitrary ways and won’t converge to the same result from different starting points. At the same time, the structure of humans possibly creates a strong attractor, so that you have enough freedom in choosing the initial interpretation to specify something manually, while knowing that the end result depends very little on the initial specification.
I think that’s a separate question from “why one might design [a could/should] agent”, which is what started this thread. For that, the informal definition of “agent” that I gave seems to be sufficient, at least to understand the question.
On the level of informal understanding, of course. When you classify systems on agents and non-agents informally, you are using your own brain to interpret the system. This is not strong enough mechanism to extract preference, while a mechanism that can extract preference presumably would be able to see agents in configurations that people can’t interpret as agents, and what those mechanisms can see as agents is a more rigorous definition of what an agent is, hence my remark.
I not so much dispute that as don’t know of a way to make this judgment precise.
Right, although I’m not sure that “objects” are the right scope of such theory. I suspect that you also need enough subjective specification of preference to initiate the process of interpretation (preference-extraction). This will make preference of rocks arbitrary, because the process of their interpretation can start in too many arbitrary ways and won’t converge to the same result from different starting points. At the same time, the structure of humans possibly creates a strong attractor, so that you have enough freedom in choosing the initial interpretation to specify something manually, while knowing that the end result depends very little on the initial specification.
On the level of informal understanding, of course. When you classify systems on agents and non-agents informally, you are using your own brain to interpret the system. This is not strong enough mechanism to extract preference, while a mechanism that can extract preference presumably would be able to see agents in configurations that people can’t interpret as agents, and what those mechanisms can see as agents is a more rigorous definition of what an agent is, hence my remark.