Here is why I think the iterated-automated-ontology-identification approach cannot work: You cannot create information out of nothing. In more detail:
The safety constraint that you need to be 100% sure if you answer “Yes” is impossible to fulfill, since you can never be 100% sure.
So let’s say we take the safety constraint that you need to be 99% sure if you answer “Yes”. So now you run your automated ontology identifier to get a new example where it is 99% sure that the answer there is “Yes”.
Now you have two options:
You add that new example to the training set with a label “only 99% sure about that one” and train on. If you always do it like this, it seems very plausible that the automated ontology identifier cannot generate new examples until you can answer all questions correctly (aka with 99% probability), since the new training set doesn’t actually contain new information, just sth that could be inferred out of the original training set.
You just assume the answer “Yes” was correct and add the new example to the training set and train on. Then it may be plausible that the process continues on finding new “99% Yes” examples for a long time, but the problem is that it probably goes completely off the rails, since some of the “Yes” labeled examples were not true, and making predictions with those makes it much more likely that you label other “No” examples as “Yes”.
In short: For every example that your process can identify as “Yes”, the example must already be identifiable by only looking at the initial training set, since you cannot generate information out of nothing.
Your process only seems like it could work, because you assume you can find a new example that is not in the training set where you can be 100% sure that the answer is “Yes”, but this would already require an infinite amount of evidence, i.e. is impossible.
Well keep in mind that we are not proposing “iterated ontology identification” as a solution to the ELK problem, but rather as a reductio ad absurdum of the existence of any algorithm fulfilling the safety and generalization guarantees that we have given. Now here is why I don’t think it’s quite so easy to show a contradiction:
In the 99% safety guarantee, you can just train a bunch of separate predictor/reporter pairs on the same initial training data and take the intersection of their decision boundaries to get a 99.9% guarantee. Then you can sample more data from that region and do the iteration like that.
Now this assumes that each of the predictor/reporter pairs has an independent 99% safety guarantee, and you might say that they are trained on the same training data, so this independence won’t hold. But then we can use completely different sensor data—camera data, lidar data, microphone data—for each of the pairs, and proceed that way. We can still iterate the overall scheme.
The basic issue here is that it is just extremely challenging to get generalization with a safety guarantee. It is hard to see how it could be accomplished! We suspect that this is actually formally impossible, and that’s what we set out to show, though we came up short of a formal impossibility result.
I might just be repeating what he said, but he is right. Iteration doesn’t work. Assuming that you have performed an optimal Bayesian update on the training set and therefore have a probability distribution over models, generating new data from those models can’t improve your probability distribution over them, it’s just the law of conservation of expected evidence, if you had that information you would have already updated on it. Any scheme that violates this law simply can’t work.
Yeah we certainly can’t do better than the optimal Bayes update, and you’re right that any scheme violating that law can’t work. Also, I share your intuition that “iteration can’t work”—that intuition is the main driver of this write-up.
As far as I’m concerned, the central issue is: what actually is the extent of the optimal Bayes update in concept extrapolation? Is it possible that a training set drawn from some limited regime might contain enough information to extrapolate the relevant concept to situations that humans don’t yet understand? The conservation of expected evidence isn’t really sufficient to settle that question, because the iteration might just be a series of computational steps towards a single Bayes update (we do not require that each individual step optimally utilize all available information).
This makes me think of what a probabilistic version of defining this extrapolation procedure would look like. It seems like you could very easily give something a bunch of labeled data in a region with high confidence, and it could extend this with progressively lower confidence as you extrapolate more, with no paradoxes. The problem entirely seems to come from the fact that the generalizer’s guesses are treated as just as good as the original examples in every way.
Forget iteration. All you can do is to take the training data, do Bayesian inference and get from it the probability that the diamond is in the room for some situation.
Trying to prove some impossibility result here seems useless.
Well just so you know, the point of the write-up is that iteration makes no sense. We are saying “hey suppose you have an automated ontology identifier with a safety guarantee and a generalization guarantee, then uh oh it looks like this really counter-intuitive iteration thing becomes possible”.
However it’s not quite as simple as to rule out iteration as appealing to conservation of expected evidence, because it’s not clear exactly how much evidence is in the training data. Perhaps there is enough information in the training data to extrapolate all the way to C. In this case the iteration scheme would just be a series of computational steps that implement a single Bayes update. Yet for the reasons discussed under “implications” I don’t think this is reasonable.
Well just so you know, the point of the write-up is that iteration makes no sense.
True, not sure what I was thinking when I wrote the last sentence of my comment.
“hey suppose you have an automated ontology identifier with a safety guarantee and a generalization guarantee, then uh oh it looks like this really counter-intuitive iteration thing becomes possible”
For an automated ontology identifier with a possible safety guarantee (like 99.9% certainty), I don’t agree with your intuition that iteration seems like it could work significantly better than just doing predictions with the original training set. Iteration simply doesn’t seem promising to me, but maybe I’m overlooking something.
If your intuition that iteration might work doesn’t come from the sense that the new predicted training examples are basically certain (as I described in the main comment of that comment thread), then where does it come from? (I do still think that you are probably confused because of the reason I described, but maybe I’m wrong and there is another reason.)
Perhaps there is enough information in the training data to extrapolate all the way to C. In this case the iteration scheme would just be a series of computational steps that implement a single Bayes update.
Actually, in the case that the training data includes enough information to extrapolate all the way to C (which I think is rarely the case for most applications), it does seem plausible to me that the iteration approach finds the perfect decision boundary, but in this case, it seems also plausible to me that a normal classifier that only uses extrapolation from the training set also finds the perfect boundary.
I don’t see a reason why a normal classifier should perform a lot worse than an optimal Bayes update from the training set. Do you think it does perform a lot worse, and if so, why? (If we don’t think that it performs much worse than optimal, then it quite trivially follows that the iteration approach cannot be much better, since it cannot be better than the optimal Bayes error.)
In the 99% safety guarantee, you can just train a bunch of separate predictor/reporter pairs on the same initial training data and take the intersection of their decision boundaries to get a 99.9% guarantee.
Counterexample: here is an infinite set of unique predictors that each have a 99% safety guarantee that when combined together have a… 99% safety guarantee.
Ground truth:
0≤x≤1,x∈R
f(x)={YES,if x≤0.5,NO,if x>0.5.
Predictor n:
pn(x)=⎧⎨⎩YES,if (x≤0.50)∧(Random oracle queried on (n, x) returns True)YES,if (0.50<x≤0.51)NO,otherwise
(If you want to make this more rigorous, replace the Random oracle query with e.g. digits of Normal numbers.)
(Analogous arguments apply in finite domains, so long as the number of possible predictors is relatively large compared to the number of actual predictors.)
But then we can use completely different sensor data—camera data, lidar data, microphone data—for each of the pairs, and proceed that way. We can still iterate the overall scheme.
No two sets of sensor data are truly ‘completely different’. Among many other things, the laws of Physics remain the same.
Here is why I think the iterated-automated-ontology-identification approach cannot work: You cannot create information out of nothing. In more detail:
The safety constraint that you need to be 100% sure if you answer “Yes” is impossible to fulfill, since you can never be 100% sure.
So let’s say we take the safety constraint that you need to be 99% sure if you answer “Yes”. So now you run your automated ontology identifier to get a new example where it is 99% sure that the answer there is “Yes”.
Now you have two options:
You add that new example to the training set with a label “only 99% sure about that one” and train on. If you always do it like this, it seems very plausible that the automated ontology identifier cannot generate new examples until you can answer all questions correctly (aka with 99% probability), since the new training set doesn’t actually contain new information, just sth that could be inferred out of the original training set.
You just assume the answer “Yes” was correct and add the new example to the training set and train on. Then it may be plausible that the process continues on finding new “99% Yes” examples for a long time, but the problem is that it probably goes completely off the rails, since some of the “Yes” labeled examples were not true, and making predictions with those makes it much more likely that you label other “No” examples as “Yes”.
In short: For every example that your process can identify as “Yes”, the example must already be identifiable by only looking at the initial training set, since you cannot generate information out of nothing.
Your process only seems like it could work, because you assume you can find a new example that is not in the training set where you can be 100% sure that the answer is “Yes”, but this would already require an infinite amount of evidence, i.e. is impossible.
Well keep in mind that we are not proposing “iterated ontology identification” as a solution to the ELK problem, but rather as a reductio ad absurdum of the existence of any algorithm fulfilling the safety and generalization guarantees that we have given. Now here is why I don’t think it’s quite so easy to show a contradiction:
In the 99% safety guarantee, you can just train a bunch of separate predictor/reporter pairs on the same initial training data and take the intersection of their decision boundaries to get a 99.9% guarantee. Then you can sample more data from that region and do the iteration like that.
Now this assumes that each of the predictor/reporter pairs has an independent 99% safety guarantee, and you might say that they are trained on the same training data, so this independence won’t hold. But then we can use completely different sensor data—camera data, lidar data, microphone data—for each of the pairs, and proceed that way. We can still iterate the overall scheme.
The basic issue here is that it is just extremely challenging to get generalization with a safety guarantee. It is hard to see how it could be accomplished! We suspect that this is actually formally impossible, and that’s what we set out to show, though we came up short of a formal impossibility result.
I might just be repeating what he said, but he is right. Iteration doesn’t work. Assuming that you have performed an optimal Bayesian update on the training set and therefore have a probability distribution over models, generating new data from those models can’t improve your probability distribution over them, it’s just the law of conservation of expected evidence, if you had that information you would have already updated on it. Any scheme that violates this law simply can’t work.
Yeah we certainly can’t do better than the optimal Bayes update, and you’re right that any scheme violating that law can’t work. Also, I share your intuition that “iteration can’t work”—that intuition is the main driver of this write-up.
As far as I’m concerned, the central issue is: what actually is the extent of the optimal Bayes update in concept extrapolation? Is it possible that a training set drawn from some limited regime might contain enough information to extrapolate the relevant concept to situations that humans don’t yet understand? The conservation of expected evidence isn’t really sufficient to settle that question, because the iteration might just be a series of computational steps towards a single Bayes update (we do not require that each individual step optimally utilize all available information).
This makes me think of what a probabilistic version of defining this extrapolation procedure would look like. It seems like you could very easily give something a bunch of labeled data in a region with high confidence, and it could extend this with progressively lower confidence as you extrapolate more, with no paradoxes. The problem entirely seems to come from the fact that the generalizer’s guesses are treated as just as good as the original examples in every way.
Yep, I approve of that answer!
Forget iteration. All you can do is to take the training data, do Bayesian inference and get from it the probability that the diamond is in the room for some situation.
Trying to prove some impossibility result here seems useless.
Well just so you know, the point of the write-up is that iteration makes no sense. We are saying “hey suppose you have an automated ontology identifier with a safety guarantee and a generalization guarantee, then uh oh it looks like this really counter-intuitive iteration thing becomes possible”.
However it’s not quite as simple as to rule out iteration as appealing to conservation of expected evidence, because it’s not clear exactly how much evidence is in the training data. Perhaps there is enough information in the training data to extrapolate all the way to C. In this case the iteration scheme would just be a series of computational steps that implement a single Bayes update. Yet for the reasons discussed under “implications” I don’t think this is reasonable.
True, not sure what I was thinking when I wrote the last sentence of my comment.
For an automated ontology identifier with a possible safety guarantee (like 99.9% certainty), I don’t agree with your intuition that iteration seems like it could work significantly better than just doing predictions with the original training set. Iteration simply doesn’t seem promising to me, but maybe I’m overlooking something.
If your intuition that iteration might work doesn’t come from the sense that the new predicted training examples are basically certain (as I described in the main comment of that comment thread), then where does it come from? (I do still think that you are probably confused because of the reason I described, but maybe I’m wrong and there is another reason.)
Actually, in the case that the training data includes enough information to extrapolate all the way to C (which I think is rarely the case for most applications), it does seem plausible to me that the iteration approach finds the perfect decision boundary, but in this case, it seems also plausible to me that a normal classifier that only uses extrapolation from the training set also finds the perfect boundary.
I don’t see a reason why a normal classifier should perform a lot worse than an optimal Bayes update from the training set. Do you think it does perform a lot worse, and if so, why? (If we don’t think that it performs much worse than optimal, then it quite trivially follows that the iteration approach cannot be much better, since it cannot be better than the optimal Bayes error.)
Counterexample: here is an infinite set of unique predictors that each have a 99% safety guarantee that when combined together have a… 99% safety guarantee.
Ground truth:
0≤x≤1,x∈R
f(x)={YES,if x≤0.5,NO,if x>0.5.
Predictor n:
pn(x)=⎧⎨⎩YES,if (x≤0.50)∧(Random oracle queried on (n, x) returns True)YES,if (0.50<x≤0.51)NO,otherwise
(If you want to make this more rigorous, replace the Random oracle query with e.g. digits of Normal numbers.)
(Analogous arguments apply in finite domains, so long as the number of possible predictors is relatively large compared to the number of actual predictors.)
No two sets of sensor data are truly ‘completely different’. Among many other things, the laws of Physics remain the same.