Is anyone else bothered by the apparent dichotomy into “natural” and “unnatural” categories? There’s room for a matter of degree here—and that’s probably all the argument ultimately needs—but the terminology is binary.
Suppose there is at least one utility function, and an agent who has it. Then all categories the agent ultimately finds useful are ipso facto “constructed [...] in an observable, testable way from categories themselves simple”. Namely, from the categories “this agent” and “utility function”. And (I think—objections welcome) you don’t have to share this agent’s values to construct it.
Is there any physical object or property that absolutely must be used (not merely recognizable in principle, but actually used) in the ontology of all conceivable rational beings, no matter their values? I doubt it. If that is the standard of “natural category” then all categories are unnatural. But if it is enough that the category be recognizable in principle, then (see the paragraph above) all categories are natural.
Is anyone else bothered by the apparent dichotomy into “natural” and “unnatural” categories? There’s room for a matter of degree here—and that’s probably all the argument ultimately needs—but the terminology is binary.
Suppose there is at least one utility function, and an agent who has it. Then all categories the agent ultimately finds useful are ipso facto “constructed [...] in an observable, testable way from categories themselves simple”. Namely, from the categories “this agent” and “utility function”. And (I think—objections welcome) you don’t have to share this agent’s values to construct it.
Is there any physical object or property that absolutely must be used (not merely recognizable in principle, but actually used) in the ontology of all conceivable rational beings, no matter their values? I doubt it. If that is the standard of “natural category” then all categories are unnatural. But if it is enough that the category be recognizable in principle, then (see the paragraph above) all categories are natural.