For the “Crappy Optimizer Theorem”, I don’t understand why condition 4, that if , then , isn’t just a tautology^{[1]}. Surely if , then no-matter what is being used,

as , then letting , then , and so .

I guess if the 4 conditions are seen as conditions on a function (where they are written for ), then it no-longer is automatic, and it is just when specifying that for some , that condition 4 becomes automatic?

______________

[start of section spitballing stuff based on the crappy optimizer theorem]

Spitball 1:

What if instead of saying , we had ? would we still get the results of the crappy optimizer theorem?

If we define if s(f) is now a distribution over X, then, I suppose instead of writing Q(s)(f)=f(s(f)) should write Q(s)(f) = s(f)(f) , and, in this case, the first 2 and 4th conditions seem just as reasonable. The third condition… seems like it should also be satisfied?

Spitball 2:

While I would expect that the 4 conditions might not be *exactly* satisfied by, e.g. gradient descent, I would kind of expect basically any reasonable deterministic optimization process to at least “almost” satisfy them? (like, maybe gradient-descent-in-practice would fail condition 1 due to floating point errors, but not too badly in reasonable cases).

Do you think that a modification of this theorem for functions Q(s) which only approximately satisfy conditions 1-3, would be reasonably achievable?

______________

- ^
I might be stretching the meaning of “tautology” here. I mean something provable in our usual background mathematics, and which therefore adding it as an additional hypothesis to a theorem, doesn’t let us show anything that we couldn’t show without it being an explicit hypothesis.

Well, I was kinda thinking of ν as being, say, a distribution of human behaviors in a certain context (as filtered through a particular user interface), though, I guess that way of doing it would only make sense within limited contexts, not general contexts where whether the agent is physically a human or something else, would matter. And in this sort of situation, well, the action of “modify yourself to no-longer be a quantilizer” would not be in the human distribution, because the actions to do that are not applicable to humans (as humans are, presumably, not quantilizers, and the types of self-modification actions that would be available are not the same). Though, “create a successor agent” could still be in the human distribution.

Of course, one doesn’t have practical access to “the true probability distribution of human behaviors in context M”, so I guess I was imagining a trained approximation to this distribution.

Hm, well, suppose that the distribution over human-like behaviors includes both making an agent which is a quantilizer and making one which isn’t, both of equal probability. Hm. I don’t see why a general quantilizer in this case would pick the quantilizer over the plain optimizer, as the utility...

Hm...

I get the idea that the “quantilizers correspond to optimizing an infra-function of form [...]” thing is maybe dealing with a distribution over a single act?

Or.. if we have a utility function over histories until the end of the episode, then, if one has a model of how the environment will be and how one is likely to act in all future steps, given each of one’s potential actions in the current step, one gets an expected utility conditioned on each of the potential actions in the current step, and this works as a utility function over actions for the current step,

and if one acts as a quantilizer over that, each step.. does that give the same behavior as an agent optimizing an infra-function defined using the condition with the L1 norm described in the post, in terms of the utility function over histories for an entire episode, and reference distributions for the whole episode?

argh, seems difficult...