False assumptions and leaky abstractions in machine learning and AI safety
The problems of embedded agency are due to the notion of agency implicit in reinforcement learning being a leaky abstraction.
Machine learning problem statements often makes assumptions that are known to be false, for example, assuming i.i.d. data.
Examining failure modes that result from false assumptions and leaky abstractions is important for safety, (at least) because they create additional possibilities for convergent rationality.
Attempting to enforce the assumptions implicit in machine learning problem statements is another important topic for safety research, since we do not fully understand the failure modes.
In practice, most machine learning research is done in settings where unrealistic assumptions are trivially enforced to a sufficiently high extent that it is reasonable to assume they are not violated (e.g. by the use of a fixed train/valid/test set, generated via pseudo-random uniform sampling from a fixed dataset).
We can (and probably should) do machine learning research that targets failure modes of common assumptions and methods of enforcing assumptions by (instead) creating settings in which these assumptions have the potential to be violated.
If you haven’t already seen it, you might find this paper I wrote (preprint, in review) relevant to your interest in this topic, since I suspect the issue runs even deeper than technical assumptions down to philosophical assumptions; or more precisely, hinge assumptions one must adopt to deal with unknowability (or if you don’t believe in unknowability, then just pragmatic assumptions made to deal with epistemic uncertainty).
I’m curious about your thoughts around the problems of embedded agency. My view is that most of what’s meaningful about understanding embedded agency is that it exposes the problems of machine learning models that make strong assumptions about the world that don’t hold up, such that the problems of embedded agency are the problems of making overly strong assumptions (cf. AIXI and the anvil problem). Is this what you are pointing to or were your trying to say something different; your words are short enough that there’s some ambiguity to me about where you suspect the problem-causing assumptions lie.
IIUC, yes, that’s basically what I was trying to say about embedded agency.
A few more important examples of important leaky abstractions that we might worry about protecting/enforcing:
Casual interventions (as “uncaused causes”, ala free will).
Boxes that don’t leak information (BoMAI)
Making a more complete list would be a good project