I don’t think so. An optimal predictor produces estimates which are as good as any efficient algorithm can produce, so if this problem indeed admits an optimal predictor (e.g. because it is generatable) then my agent will succeed at value inference as much as anything can succeed at it.
This is true asymptotically. The same argument can be applied to training an algorithm for pursuing symbolically defined goals, even using conventional techniques. The problem is that quantitatively these arguments have no teeth in the applications we care about, unless we can train on instances as complex as the one we actually care about.
I don’t know if we disagree here. We probably reach different conclusions because of our disagreement about whether you can generate instances of value inferences that are as complex as the real world.
This is true asymptotically. The same argument can be applied to training an algorithm for pursuing symbolically defined goals, even using conventional techniques. The problem is that quantitatively these arguments have no teeth in the applications we care about, unless we can train on instances as complex as the one we actually care about.
I don’t know if we disagree here. We probably reach different conclusions because of our disagreement about whether you can generate instances of value inferences that are as complex as the real world.