I agree that Luke here overstates the significance of my result, but I do think you miss the point a bit or are uncharitable. Regardless of whether making predictions about your own behavior is fundamentally difficult, we don’t yet understand any formal framework that can capture reasoning of the form “my decisions are good because my beliefs correspond to reality.” Assuming there is a natural formal framework capturing human reasoning (I think the record so far suggests optimism) then there is something interesting that we don’t yet understand. It seems like you are applying the argument: “We know that humans can do X, so why do you think that X is an important problem?” The comment about undecidability issues not applying in practice also seems a bit unfair; for programs that do proof search we know that we cannot prove claims of the desired type based on simple Godelian arguments, and almost all interesting frameworks for reasoning are harder to prove things about than a simple proof search. (Of course the game is that we don’t want to prove things about the algorithms in question, we are happy to form justified beliefs about them in whatever way we can, including inductive inference. But the point is that there are things we don’t understand.)
There are further questions about whether any work at MIRI is a meaningful contribution to this problem or any other. I think that the stuff I’ve worked on is plausibly but not obviously a significant contribution (basically the same status as the other work I’m doing).
Regarding the modal agents stuff, I agree that it’s a toy problem where you should expect progress to be fast (if there was a nice theorem at the end of it, then it wouldn’t be too unusual as a paper in theoretical CS, except for the unfashionable use of mathematical logic). Regarding updateless/timeless/ambient decision theory, it’s a clear step forward for a very idiosyncratic problem, but one for which I think you can make a reasonable case that it’s worthwhile.
I think you shouldn’t be too surprised to make meaningful headway on theoretically interesting questions, even those which will plausibly be important. It seems like in theoretical research today things are still developing reasonably rapidly, and the ratio between plausibly important problems and human capital is very large. I expect that given effort and success at recruiting human capital MIRI can make good headway, in the same sort of way that other theorists do. Optimistically they would be distinguished primarily by working on a class of problems which is unusually important given their values and model of the world (a judgment with which you might disagree).
Of course the game is that we don’t want to prove things about the algorithms in question, we are happy to form justified beliefs about them in whatever way we can, including inductive inference. But the point is that there are things we don’t understand.
And the question is: who cares? The mechanism by which human beings predict their future behavior is not logical inference. Similar ad-hoc Bayesian extrapolation techniques can be used in any general AI without worry about Löbian obstacles. So why is it such a pressing issue?
I don’t wish to take away from the magnitude of your accomplishment. It is an important achievement. But in the long run I don’t think it’s going to be a very useful result in the construction of superhuman AGIs, specifically. And it’s reasonable to ask why MIRI is assigning strategic importance to these issues.
I think you shouldn’t be too surprised to make meaningful headway on theoretically interesting questions, even those which will plausibly be important. It seems like in theoretical research today things are still developing reasonably rapidly, and the ratio between plausibly important problems and human capital is very large.
I agree with this. Luke seems to be making a much stronger claim than the above, though.
It seems like you are applying the argument: “We know that humans can do X, so why do you think that X is an important problem?”
I agree that that would be a bad argument. That was not the argument I intended to make, though I can see why it has been interpreted that way and I should have put more effort into explaining myself. I am rather saying that human reasoning looks so far away from even getting close to running into issues with Godel / Lob, that it seems like a rather abstruse starting point for investigation.
The rest of your comment seems most easily discussed in person, so I’ll do that and hopefully we’ll remember to update the thread with our conclusion.
I agree that Luke here overstates the significance of my result, but I do think you miss the point a bit or are uncharitable. Regardless of whether making predictions about your own behavior is fundamentally difficult, we don’t yet understand any formal framework that can capture reasoning of the form “my decisions are good because my beliefs correspond to reality.” Assuming there is a natural formal framework capturing human reasoning (I think the record so far suggests optimism) then there is something interesting that we don’t yet understand. It seems like you are applying the argument: “We know that humans can do X, so why do you think that X is an important problem?” The comment about undecidability issues not applying in practice also seems a bit unfair; for programs that do proof search we know that we cannot prove claims of the desired type based on simple Godelian arguments, and almost all interesting frameworks for reasoning are harder to prove things about than a simple proof search. (Of course the game is that we don’t want to prove things about the algorithms in question, we are happy to form justified beliefs about them in whatever way we can, including inductive inference. But the point is that there are things we don’t understand.)
There are further questions about whether any work at MIRI is a meaningful contribution to this problem or any other. I think that the stuff I’ve worked on is plausibly but not obviously a significant contribution (basically the same status as the other work I’m doing).
Regarding the modal agents stuff, I agree that it’s a toy problem where you should expect progress to be fast (if there was a nice theorem at the end of it, then it wouldn’t be too unusual as a paper in theoretical CS, except for the unfashionable use of mathematical logic). Regarding updateless/timeless/ambient decision theory, it’s a clear step forward for a very idiosyncratic problem, but one for which I think you can make a reasonable case that it’s worthwhile.
I think you shouldn’t be too surprised to make meaningful headway on theoretically interesting questions, even those which will plausibly be important. It seems like in theoretical research today things are still developing reasonably rapidly, and the ratio between plausibly important problems and human capital is very large. I expect that given effort and success at recruiting human capital MIRI can make good headway, in the same sort of way that other theorists do. Optimistically they would be distinguished primarily by working on a class of problems which is unusually important given their values and model of the world (a judgment with which you might disagree).
And the question is: who cares? The mechanism by which human beings predict their future behavior is not logical inference. Similar ad-hoc Bayesian extrapolation techniques can be used in any general AI without worry about Löbian obstacles. So why is it such a pressing issue?
I don’t wish to take away from the magnitude of your accomplishment. It is an important achievement. But in the long run I don’t think it’s going to be a very useful result in the construction of superhuman AGIs, specifically. And it’s reasonable to ask why MIRI is assigning strategic importance to these issues.
I agree with this. Luke seems to be making a much stronger claim than the above, though.
I agree that that would be a bad argument. That was not the argument I intended to make, though I can see why it has been interpreted that way and I should have put more effort into explaining myself. I am rather saying that human reasoning looks so far away from even getting close to running into issues with Godel / Lob, that it seems like a rather abstruse starting point for investigation.
The rest of your comment seems most easily discussed in person, so I’ll do that and hopefully we’ll remember to update the thread with our conclusion.
What makes you say that? Did you see what I said about this here?