how do we take those notions and turn them into something mathematically precise enough that we could instruct a machine to do them and then evaluate whether or not what it did was in fact what we intended
Yep, that’s the project! I think the main utility of Callard’s work here is (1) pointing out the phenomenon (a phenomenon that is strikingly similar to some of the abilities we want AI’s to have), and (2) noticing that the most prominent theories of decision theory, moral psychology, and moral responsibility make assumptions that we have to break if we want to allow room for aspiration (assumptions that we who are trying to build safe AI are probably also accidentally making insofar as we take over those standard theories). IDK whether she provides alternate assumptions to make instead, but if she does these might also be useful. But the main point is just noticing that we need different theories of these things.
Once we’ve noticed the phenomenon of aspiration, and that it requires breaking some of these assumptions, I agree that the hard bit is coming up with a mathematical theory of aspiration (or the AI equivalent).
Yep, that’s the project! I think the main utility of Callard’s work here is (1) pointing out the phenomenon (a phenomenon that is strikingly similar to some of the abilities we want AI’s to have), and (2) noticing that the most prominent theories of decision theory, moral psychology, and moral responsibility make assumptions that we have to break if we want to allow room for aspiration (assumptions that we who are trying to build safe AI are probably also accidentally making insofar as we take over those standard theories). IDK whether she provides alternate assumptions to make instead, but if she does these might also be useful. But the main point is just noticing that we need different theories of these things.
Once we’ve noticed the phenomenon of aspiration, and that it requires breaking some of these assumptions, I agree that the hard bit is coming up with a mathematical theory of aspiration (or the AI equivalent).