I think the other natural approach is to simply make decisions based on the current estimated preferences, but to learn instrumental preferences of the user (including desire for the agent to learn more), as described here. Of course this also doesn’t have the problem from the OP.
Yeah, this seems like the most natural way to deal with things like “chocolate” that aren’t yet well-defined. In this case, the instrumental preferences themselves will be treated as historical facts (it’s assumed that they’re already well-defined enough to learn).
I think the other natural approach is to simply make decisions based on the current estimated preferences, but to learn instrumental preferences of the user (including desire for the agent to learn more), as described here. Of course this also doesn’t have the problem from the OP.
Yeah, this seems like the most natural way to deal with things like “chocolate” that aren’t yet well-defined. In this case, the instrumental preferences themselves will be treated as historical facts (it’s assumed that they’re already well-defined enough to learn).