tl;dr: If yes, then the problem of modelling the intentional outlook of a representational system may be reducible to the problem of modelling a subset of that system’s context structuring questions under sufficient counterfactual variation.
Introduction:
I argue that questions (on a standard interpretation of question semantics) sometimes function like a halfway house between descriptive and normative content and that the mechanism mediating this relationship strongly resembles rigid designation. Descriptive content covers facts and objective reality and normative content covers shoulds and oughts. Hume taught us all that we can’t get an ought from an is, but shall the twain never meet? Wilfrid Sellars tells us that the world perceived by an observer (even and especially a purely scientific observer) is fraught with ought. Here is my attempt to use one of 20th century analytic philosophy’s cleanest insights—Krikpe’s idea of rigid designation—plus some less well-known (but quite sturdy) work on question semantics to give formal character to the idea that the manifest image of the world is shot through with normativity. And by “manifest image” I mean the world represented by any perceiver down to and probably including some thermostats.[1]
What is rigid designation?
Saul Kripke defines a rigid designator as a term that refers to the same object in every world in which that object exists. Names are rigid designators. The Morning Star (Phosphorus) and the Evening Star (Hesperus) both name the planet Venus, and the statement “Hesperus is Phosphorus” is a necessary a posteriori truth on Kripke’s widely accepted account. It’s an a posteriori truth because it’s possible to imagine a world in which Venus did not exist, and so it’s true in virtue of some feature of the world. It’s a necessary truth because in any world in which Venus exists, two different names for it are necessarily co-extensive. Other examples of necessary a posteriori truths include “Water is H2O” and “Gold has atomic number 79”.
Definite descriptions on the other hand are typically non-rigid designators. For example, “The winner of the 2025 Nobel Peace Prize” is a non-rigid designator and a definite description. The assertion “The winner of the 2025 Nobel Peace Price is María Corina Machado” is true, but it’s not necessarily true. If we indulge in some possible world semantics,[2] there are many worlds in which Machado exists but someone else won the award; that she won is only a contingent fact about the world we live in. Fun tidbit, if we add an indexical like “actually” to the definite description it rigidifies it, but it’s the indexical not the definite description doing that work.
To sum up: Rigid designators refer to the same object in every world in which that object exists whereas non-rigid designators can refer to different objects across worlds.
Why think that questions can rigidly designate intentions?
Imagine an organism with one intention: find food. Now suppose that organism—call it Fred—has to query its (controlled) environment to achieve this goal. The ability to query or sample an environment and store information as a result of the sampling process is a minimal representational capacity.[3] Fred must differentiate food from non-food, and given the ability to represent, sample, and store (even imperfectly) information there are basically two methods for doing so: a query that directly bears on Fred’s intention or a query that is a proxy for Fred’s intention.
Direct query: Fred queries the environment for food itself . Questions like “what things have nutritional value to Fred?” or “is this food?”, when given full scope over the objects in Fred’s environment and answered accurately, sort the environment into edible and non-edible for Fred.
Proxy query: Fred queries the environment for a proxy for food. For example, suppose that all and only food is colored red where Fred is, and so the question “what things are red?” when answered accurately sorts the environment into red and non-red categories and by stipulation into edible and non-edible for Fred.
You probably see the connection already. These two methods closely approximate rigid and non-rigid designation as a relationship that holds between queries and intentions. Direct queries function like rigid designators and proxy queries function like non-rigid designators insofar as they bear on the intention find food. In every world in which Fred has the goal of finding food, the direct query “what things have nutritional value to Fred?” bears on this goal in that a complete answer to the question is equivalent to satisfying the goal. By contrast, proxy queries fail in certain counterfactual environments as they only track contingent correlates of the satisfaction conditions for the goal. If we imagined a world in which all and only food was blue, the proxy query “what things are red?” would no longer bear on Fred’s intention to find food. Much like a non-rigid designator, a proxy query tracks a contingent fact of the environment that happens to bear on an intention but does not necessarily do so.
Now, what does this bears on relationship actually look like? Kripke is talking about reference—a rigid designator refers to the same object in every world in which that object exists. Are we to understand direct queries as referring to some intention in every world in which that intention exists? Sortof. Reference assumes a descriptive, truth-functional relationship between the referring term (like a name) and the thing referred to (like the named object). And while we can of course refer to intentional attitudes (things like “Fred wants to find food”), intentions themselves have a normative character—they are not descriptive. We often express intentions with imperatives like “shut the door!” or “find food!” which are not truth functional the way declaratives like “the door is shut” or “Fred wants to find food” are truth functional. What we need is a reliable way to transpose non-truth functional intentional (normative) content into a truth functional space. Enter question semantics.
Question semantics differ from the semantics of declarative utterances. On the standard reading, declarative utterances (assertions) have propositional content. Propositions can be understood as functions from a set of possible worlds to truth valuations at each of those worlds. A proposition tokened by an assertive utterance bisects the set of all possible worlds, sorting those worlds into ones in which the utterance is true and worlds in which the utterance is false. By contrast standard question semantics represents questions as a partition over a set of worlds. The partition, induced by the question, is a mutually exclusive and exhaustive set of alternatives (or cells)--each alternative itself a set of worlds. A partial answer to a question returns a truth valuation for every world in at least one cell of the partition. A complete answer satisfies this condition for every cell in the question partition. A key feature of question semantics is that differences between worlds that are in the same cell of a question partition by definition do not directly contribute to answering the question. For example, if I ask “who talked to who at the party last night?” the partition denoted by this question would not track differences in shirt colors of the party-goers. Conversely, “who wore pink to the party last night?” would be indifferent to conversational pairings. In short, questions impose structure on the context. I believe this structure allows for (is?) an interface between descriptive and normative content and that counterfactual variation reveals which context structuring questions map to intentions across every world in which that intention exists.
An organism like Fred could have a context-structuring question partition that models its context by way of direct query or proxy query. Only by systematically varying Fred’s environment could the observer determine which partition maps to Fred intention. Without access to Fred’s subjective states and making no assumptions about Fred’s biology, find the red things or find food are both viable candidate intentions for Fred. Of course, the observer can only get asymmetric information about whether or not a question structuring partition is a direct query or a proxy query insofar as it relates to Fred’s intentional outlook; you can falsify the claim that some query is a direct query but you cannot confirm a query directly bears on an intention unless every possible environmental variation is tested (an impossible task, just ask Popper). If the environmental variation results in a behavior change, say the observer turns the food blue and Fred keeps going after red stuff for a while but then changes behavior or dies, we learn that Fred was tracking red but as a proxy for something else. The observer should adopt a guilty until proven innocent attitude towards the possible context structuring questions for Fred, assuming that all possible queries consistent with Fred’s behavior directly bear on an intention until an intervention proves otherwise.
If it’s true that direct queries stand in this special relationship to intentions, then the problem of modelling the intentional outlook (or normative content) of a system capable of representation (whether that’s Fred or a frontier AI) gets reduced to the problem of modelling that representational system’s context structuring questions under sufficient counterfactual variation. Fraught with ought indeed.
There are all sorts of reasons why a formal characterization of the relationship between descriptive content and normative content is valuable for AI safety research, and this is my reason for posting on LW. The argument below turns on a structured representation of context which ought to be machine learnable with a suitable architecture and a healthy dose of statistical learning theory (I’m not a Bayesian, sue me). Future posts will explore this idea in more detail, but while I can hand-roll metaethics, I cannot hand-roll a neural net. Help would be appreciated on this front.
I don’t have any strong commitments about the metaphysical status of possible worlds. They are a useful and standard analytic frame to think about assertions, questions, and counterfactuals. All talk of possible worlds in this post could be replaced with a pointless topology that would have sparser metaphysical commitments but involve a lot more math that I don’t know how to do.
I will use goal and intention interchangeably throughout this example. I’m aware that the terms are not always used synonymously, but I promise that I’m not doing any arbitrage on possible differences in interpretation (at least that’s not my goal/not my intention).
Can questions rigidly designate intentions? File under: Rolling my own metaethics
tl;dr: If yes, then the problem of modelling the intentional outlook of a representational system may be reducible to the problem of modelling a subset of that system’s context structuring questions under sufficient counterfactual variation.
Introduction:
I argue that questions (on a standard interpretation of question semantics) sometimes function like a halfway house between descriptive and normative content and that the mechanism mediating this relationship strongly resembles rigid designation. Descriptive content covers facts and objective reality and normative content covers shoulds and oughts. Hume taught us all that we can’t get an ought from an is, but shall the twain never meet? Wilfrid Sellars tells us that the world perceived by an observer (even and especially a purely scientific observer) is fraught with ought. Here is my attempt to use one of 20th century analytic philosophy’s cleanest insights—Krikpe’s idea of rigid designation—plus some less well-known (but quite sturdy) work on question semantics to give formal character to the idea that the manifest image of the world is shot through with normativity. And by “manifest image” I mean the world represented by any perceiver down to and probably including some thermostats.[1]
What is rigid designation?
Saul Kripke defines a rigid designator as a term that refers to the same object in every world in which that object exists. Names are rigid designators. The Morning Star (Phosphorus) and the Evening Star (Hesperus) both name the planet Venus, and the statement “Hesperus is Phosphorus” is a necessary a posteriori truth on Kripke’s widely accepted account. It’s an a posteriori truth because it’s possible to imagine a world in which Venus did not exist, and so it’s true in virtue of some feature of the world. It’s a necessary truth because in any world in which Venus exists, two different names for it are necessarily co-extensive. Other examples of necessary a posteriori truths include “Water is H2O” and “Gold has atomic number 79”.
Definite descriptions on the other hand are typically non-rigid designators. For example, “The winner of the 2025 Nobel Peace Prize” is a non-rigid designator and a definite description. The assertion “The winner of the 2025 Nobel Peace Price is María Corina Machado” is true, but it’s not necessarily true. If we indulge in some possible world semantics,[2] there are many worlds in which Machado exists but someone else won the award; that she won is only a contingent fact about the world we live in. Fun tidbit, if we add an indexical like “actually” to the definite description it rigidifies it, but it’s the indexical not the definite description doing that work.
To sum up: Rigid designators refer to the same object in every world in which that object exists whereas non-rigid designators can refer to different objects across worlds.
Why think that questions can rigidly designate intentions?
Imagine an organism with one intention: find food. Now suppose that organism—call it Fred—has to query its (controlled) environment to achieve this goal. The ability to query or sample an environment and store information as a result of the sampling process is a minimal representational capacity.[3] Fred must differentiate food from non-food, and given the ability to represent, sample, and store (even imperfectly) information there are basically two methods for doing so: a query that directly bears on Fred’s intention or a query that is a proxy for Fred’s intention.
Direct query: Fred queries the environment for food itself . Questions like “what things have nutritional value to Fred?” or “is this food?”, when given full scope over the objects in Fred’s environment and answered accurately, sort the environment into edible and non-edible for Fred.
Proxy query: Fred queries the environment for a proxy for food. For example, suppose that all and only food is colored red where Fred is, and so the question “what things are red?” when answered accurately sorts the environment into red and non-red categories and by stipulation into edible and non-edible for Fred.
You probably see the connection already. These two methods closely approximate rigid and non-rigid designation as a relationship that holds between queries and intentions. Direct queries function like rigid designators and proxy queries function like non-rigid designators insofar as they bear on the intention find food. In every world in which Fred has the goal of finding food, the direct query “what things have nutritional value to Fred?” bears on this goal in that a complete answer to the question is equivalent to satisfying the goal. By contrast, proxy queries fail in certain counterfactual environments as they only track contingent correlates of the satisfaction conditions for the goal. If we imagined a world in which all and only food was blue, the proxy query “what things are red?” would no longer bear on Fred’s intention to find food. Much like a non-rigid designator, a proxy query tracks a contingent fact of the environment that happens to bear on an intention but does not necessarily do so.
Now, what does this bears on relationship actually look like? Kripke is talking about reference—a rigid designator refers to the same object in every world in which that object exists. Are we to understand direct queries as referring to some intention in every world in which that intention exists? Sortof. Reference assumes a descriptive, truth-functional relationship between the referring term (like a name) and the thing referred to (like the named object). And while we can of course refer to intentional attitudes (things like “Fred wants to find food”), intentions themselves have a normative character—they are not descriptive. We often express intentions with imperatives like “shut the door!” or “find food!” which are not truth functional the way declaratives like “the door is shut” or “Fred wants to find food” are truth functional. What we need is a reliable way to transpose non-truth functional intentional (normative) content into a truth functional space. Enter question semantics.
Question semantics differ from the semantics of declarative utterances. On the standard reading, declarative utterances (assertions) have propositional content. Propositions can be understood as functions from a set of possible worlds to truth valuations at each of those worlds. A proposition tokened by an assertive utterance bisects the set of all possible worlds, sorting those worlds into ones in which the utterance is true and worlds in which the utterance is false. By contrast standard question semantics represents questions as a partition over a set of worlds. The partition, induced by the question, is a mutually exclusive and exhaustive set of alternatives (or cells)--each alternative itself a set of worlds. A partial answer to a question returns a truth valuation for every world in at least one cell of the partition. A complete answer satisfies this condition for every cell in the question partition. A key feature of question semantics is that differences between worlds that are in the same cell of a question partition by definition do not directly contribute to answering the question. For example, if I ask “who talked to who at the party last night?” the partition denoted by this question would not track differences in shirt colors of the party-goers. Conversely, “who wore pink to the party last night?” would be indifferent to conversational pairings. In short, questions impose structure on the context. I believe this structure allows for (is?) an interface between descriptive and normative content and that counterfactual variation reveals which context structuring questions map to intentions across every world in which that intention exists.
An organism like Fred could have a context-structuring question partition that models its context by way of direct query or proxy query. Only by systematically varying Fred’s environment could the observer determine which partition maps to Fred intention. Without access to Fred’s subjective states and making no assumptions about Fred’s biology, find the red things or find food are both viable candidate intentions for Fred. Of course, the observer can only get asymmetric information about whether or not a question structuring partition is a direct query or a proxy query insofar as it relates to Fred’s intentional outlook; you can falsify the claim that some query is a direct query but you cannot confirm a query directly bears on an intention unless every possible environmental variation is tested (an impossible task, just ask Popper). If the environmental variation results in a behavior change, say the observer turns the food blue and Fred keeps going after red stuff for a while but then changes behavior or dies, we learn that Fred was tracking red but as a proxy for something else. The observer should adopt a guilty until proven innocent attitude towards the possible context structuring questions for Fred, assuming that all possible queries consistent with Fred’s behavior directly bear on an intention until an intervention proves otherwise.
If it’s true that direct queries stand in this special relationship to intentions, then the problem of modelling the intentional outlook (or normative content) of a system capable of representation (whether that’s Fred or a frontier AI) gets reduced to the problem of modelling that representational system’s context structuring questions under sufficient counterfactual variation. Fraught with ought indeed.
There are all sorts of reasons why a formal characterization of the relationship between descriptive content and normative content is valuable for AI safety research, and this is my reason for posting on LW. The argument below turns on a structured representation of context which ought to be machine learnable with a suitable architecture and a healthy dose of statistical learning theory (I’m not a Bayesian, sue me). Future posts will explore this idea in more detail, but while I can hand-roll metaethics, I cannot hand-roll a neural net. Help would be appreciated on this front.
I don’t have any strong commitments about the metaphysical status of possible worlds. They are a useful and standard analytic frame to think about assertions, questions, and counterfactuals. All talk of possible worlds in this post could be replaced with a pointless topology that would have sparser metaphysical commitments but involve a lot more math that I don’t know how to do.
I will use goal and intention interchangeably throughout this example. I’m aware that the terms are not always used synonymously, but I promise that I’m not doing any arbitrage on possible differences in interpretation (at least that’s not my goal/not my intention).