Is there a reason you think a reflective oracle (or equivalent) can’t just be selected “arbitrarily”, and will likely be selected to maximize some score?
The gradient descent is not being done over the reflective oracles, it’s being done over some general computational model like a neural net. Any highly-performing solution will necessarily look like a fixed-point-finding computation of some kind, due to the self-referential nature of the predictions. Then, since this fixed-point-finder is *internal* to the model, it will be optimized for log loss just like everything else in the model.
That is, the global optimization of the model is distinct from whatever internal optimization the fixed-point-finder uses to choose the reflective oracle. The global optimization will favor internal optimizers that produce fixed-points with good score. So while fixed-point-finders in general won’t optimize for anything in particular, the one this model uses will.
I think the fixed point finder won’t optimize the fixed point for minimizing expected log loss. I’m going to give a concrete algorithm and show that it doesn’t exhibit this behavior. If you disagree, can you present an alternative algorithm?
Here’s the algorithm. Start with some oracle (not a reflective oracle). Sample ~1000000 universes based on this oracle, getting 1000000 data points for what the reflective oracle outputs. Move the oracle 1% of the way from its current position towards the oracle that would answer queries correctly given the distribution over universes implied by the data points. Repeat this procedure a lot of times (~10,000). This procedure is similar to gradient descent.
Here’s an example universe:
M:=if O(M,0.3)=1 then flip(0.9) else 0
Note the presence of two reflective oracles that are stable equilibria: one where P(O(M,0.3)=1)=0, and one where P(O(M,0.3)=1)=1. Notice that the first has lower expected log loss than the second.
Let’s parameterize oracles by numbers in [0,1] representing P(O(M,0.3)=1) (since this is the only relevant query). Start with oracle 0.5. If we sample 1000000 universes, about 45% of them have outcome 1. So, based on these data points, P(M())=0.45, so the oracle based on these data points will say P(O(M,0.3)=1)=1, i.e. it is parameterized by 1. So we move our current oracle (0.5) 1% of the way towards the oracle 1, yielding oracle 0.505. We repeat this a bunch of times, eventually getting an oracle parameterized by a number very close to 1.
So, this procedure yields an oracle with suboptimal expected log loss. It is not the case that the fixed point finder minimizes expected log loss. The neural net case is different, but not that much; it would give the same answer in this particular case, since the model can just be parameterized by a single real number.
Reflective Oracles are a bit of a weird case case because their ‘loss’ is more like a 0⁄1 loss than a log loss, so all of the minima are exactly the same(If we take a sample of 100000 universes to score them, the difference is merely incredibly small instead of 0). I was being a bit glib referencing them in the article; I had in mind something more like a model parameterizing a distribution over outputs, whose only influence on the world is via a random sample from this distribution. I think that such models should in general have fixed points for similar reasons, but am not sure. Regardless, these models will, I believe, favour fixed points whose distributions are easy to compute(But not fixed points with low entropy, that is they will punish logical uncertainty but not intrinsic uncertainy). I’m planning to run some experiments with VAEs and post the results later.
The gradient descent is not being done over the reflective oracles, it’s being done over some general computational model like a neural net. Any highly-performing solution will necessarily look like a fixed-point-finding computation of some kind, due to the self-referential nature of the predictions. Then, since this fixed-point-finder is *internal* to the model, it will be optimized for log loss just like everything else in the model.
That is, the global optimization of the model is distinct from whatever internal optimization the fixed-point-finder uses to choose the reflective oracle. The global optimization will favor internal optimizers that produce fixed-points with good score. So while fixed-point-finders in general won’t optimize for anything in particular, the one this model uses will.
I think the fixed point finder won’t optimize the fixed point for minimizing expected log loss. I’m going to give a concrete algorithm and show that it doesn’t exhibit this behavior. If you disagree, can you present an alternative algorithm?
Here’s the algorithm. Start with some oracle (not a reflective oracle). Sample ~1000000 universes based on this oracle, getting 1000000 data points for what the reflective oracle outputs. Move the oracle 1% of the way from its current position towards the oracle that would answer queries correctly given the distribution over universes implied by the data points. Repeat this procedure a lot of times (~10,000). This procedure is similar to gradient descent.
Here’s an example universe:
M:=if O(M,0.3)=1 then flip(0.9) else 0
Note the presence of two reflective oracles that are stable equilibria: one where P(O(M,0.3)=1)=0, and one where P(O(M,0.3)=1)=1. Notice that the first has lower expected log loss than the second.
Let’s parameterize oracles by numbers in [0,1] representing P(O(M,0.3)=1) (since this is the only relevant query). Start with oracle 0.5. If we sample 1000000 universes, about 45% of them have outcome 1. So, based on these data points, P(M())=0.45, so the oracle based on these data points will say P(O(M,0.3)=1)=1, i.e. it is parameterized by 1. So we move our current oracle (0.5) 1% of the way towards the oracle 1, yielding oracle 0.505. We repeat this a bunch of times, eventually getting an oracle parameterized by a number very close to 1.
So, this procedure yields an oracle with suboptimal expected log loss. It is not the case that the fixed point finder minimizes expected log loss. The neural net case is different, but not that much; it would give the same answer in this particular case, since the model can just be parameterized by a single real number.
Reflective Oracles are a bit of a weird case case because their ‘loss’ is more like a 0⁄1 loss than a log loss, so all of the minima are exactly the same(If we take a sample of 100000 universes to score them, the difference is merely incredibly small instead of 0). I was being a bit glib referencing them in the article; I had in mind something more like a model parameterizing a distribution over outputs, whose only influence on the world is via a random sample from this distribution. I think that such models should in general have fixed points for similar reasons, but am not sure. Regardless, these models will, I believe, favour fixed points whose distributions are easy to compute(But not fixed points with low entropy, that is they will punish logical uncertainty but not intrinsic uncertainy). I’m planning to run some experiments with VAEs and post the results later.