That’s very helpful, thanks. I think the key here is this part:
The short-term predictor is a learning algorithm. Given infinite time, we normally expect learning algorithms to settle into some steady-state configuration where they no longer update. We can think of this configuration as “what we are training it to do”. So, in this toy model, what are we training the short term predictor to do?
I’m trying to talk about what will happen given infinite time, in steady state, i.e. when it gets to a fixed-point / self-consistent solution. In steady-state / at a self-consistent fixed point, the STP output approximates the expectation of the next override (and/or the expectation of the STP output at the next change of context data). So we can call that “prediction”, in the sense that “it’s a signal which tells us that a certain thing will happen later”. But it’s not a “prediction” in the sense of “passively predicting an independent, exogenous event”.
Rather, the LTP is one component of a machine (that also includes the stomach or whatever), and we’re narrowly zooming into that one component and seeing that its outputs can be interpreted as predictions, at the fixed point.
Relatedly, in logical induction (cf. here, or Theorem 4.11.2 in the original paper), they formulate a seemingly-paradoxical sentence “this sentence is true if you predict it to be true with probability LESS than 50%”. Does it have a fixed-point / steady-state? Yes, 50%. And that’s what it converges to.
Another pathological case would be to make everything a fixed point: in the case at hand, we could set up a LTP where the override is by definition whatever the STP is outputting at that moment. So then the LTP will stably output whatever random value the STP was spitting out when it was randomly initialized. Nevertheless, we can still call that a kind of “prediction”, I think. Like, we look at the STP output of 1.3 and say “it’s predicting that the next override will be 1.3”, and then the next override comes, and indeed it is 1.3, which validates that point of view.
Anyway, the text didn’t make any of this very clear, because I wasn’t really thinking about this aspect of it, so again I appreciate your comments.
Indeed, I’m now questioning a bit whether “prediction” is the right word here, since the word “prediction” does usually have a connotation of “passively predicting an exogenous event”, and I have previously criticized people for using the word “prediction” in weird situations where that connotation does not apply, so it might be hypocritical if I’m doing that myself. Hmm, I think the word “prediction” is still OK here, but I would want to add some clarifying text for sure.
Compare R=1 to R=2. If these are predictions, then R=1 is a higher [predicted probability that the next override is R=0] than R=2. But R=1 produces fewer enzymes, which makes the R=0 override less likely.
Think of this in terms of what the fixed point / steady-state / self-consistent solution would be.
Let’s say context is always the same, and on day 0, the STP outputs R=2, and that’s almost always too much digestive enzymes, so there’s R=0 overrides 90% of the time (and R=10 the other 10%). Over the next week, the STP weights update to make the predictions incrementally lower, and now R=1.8, and there’s R=0 overrides 85% of the time. Over the next week, the STP weights continue to update in the same direction, until now R=1.7, and there’s R=0 overrides 83% of the time, and R=10 overrides the other 17%. And now we’re at the fixed point! And indeed the STP output is now the expectation value of the next override, so we can (maybe slightly dubiously) use the word “prediction” to describe this output.
That’s very helpful, thanks. I think the key here is this part:
I’m trying to talk about what will happen given infinite time, in steady state, i.e. when it gets to a fixed-point / self-consistent solution. In steady-state / at a self-consistent fixed point, the STP output approximates the expectation of the next override (and/or the expectation of the STP output at the next change of context data). So we can call that “prediction”, in the sense that “it’s a signal which tells us that a certain thing will happen later”. But it’s not a “prediction” in the sense of “passively predicting an independent, exogenous event”.
Rather, the LTP is one component of a machine (that also includes the stomach or whatever), and we’re narrowly zooming into that one component and seeing that its outputs can be interpreted as predictions, at the fixed point.
Relatedly, in logical induction (cf. here, or Theorem 4.11.2 in the original paper), they formulate a seemingly-paradoxical sentence “this sentence is true if you predict it to be true with probability LESS than 50%”. Does it have a fixed-point / steady-state? Yes, 50%. And that’s what it converges to.
Another pathological case would be to make everything a fixed point: in the case at hand, we could set up a LTP where the override is by definition whatever the STP is outputting at that moment. So then the LTP will stably output whatever random value the STP was spitting out when it was randomly initialized. Nevertheless, we can still call that a kind of “prediction”, I think. Like, we look at the STP output of 1.3 and say “it’s predicting that the next override will be 1.3”, and then the next override comes, and indeed it is 1.3, which validates that point of view.
Anyway, the text didn’t make any of this very clear, because I wasn’t really thinking about this aspect of it, so again I appreciate your comments.
Indeed, I’m now questioning a bit whether “prediction” is the right word here, since the word “prediction” does usually have a connotation of “passively predicting an exogenous event”, and I have previously criticized people for using the word “prediction” in weird situations where that connotation does not apply, so it might be hypocritical if I’m doing that myself. Hmm, I think the word “prediction” is still OK here, but I would want to add some clarifying text for sure.
Think of this in terms of what the fixed point / steady-state / self-consistent solution would be.
Let’s say context is always the same, and on day 0, the STP outputs R=2, and that’s almost always too much digestive enzymes, so there’s R=0 overrides 90% of the time (and R=10 the other 10%). Over the next week, the STP weights update to make the predictions incrementally lower, and now R=1.8, and there’s R=0 overrides 85% of the time. Over the next week, the STP weights continue to update in the same direction, until now R=1.7, and there’s R=0 overrides 83% of the time, and R=10 overrides the other 17%. And now we’re at the fixed point! And indeed the STP output is now the expectation value of the next override, so we can (maybe slightly dubiously) use the word “prediction” to describe this output.