Hi. I’m a long time lurker (a few years now), and I finally joined so that I could participate in the community and the discussions. This was borne partly out of a sense that I’m at a place in my life where I could really benefit from this community (and it could benefit from me), and partly out of a specific interest in some of the things that have been posted recently: the MIRI technical research agenda.
In particular, once I’ve had more time to digest it, I want to post comments and questions about Reasoning Under Logical Uncertainty.
More about me: I’m currently working as a postdoctoral fellow in mathematics. My professional work is in physics-y differential geometry, so only connected to the LW material indirectly via things like quantum mechanics. I practice Buhddist meditation, without definitively endorsing any of the doctrines. I’m surprised meditation hasn’t gotten more airtime in the rationalist community.
My IRL exposure to the LWverse is limited (hi Critch!), but I gather there’s a meetup group in Utrecht, where I’m living now.
Anyway, I look forward to good discussions. Hello everyone!
I see a fair amount of back-and-forth where someone says “What about this?” and you say “I addressed that in several places; clearly you didn’t read it.” Unfortunately, while you may think you have addressed the various issues, I don’t think you did (and presumably your interlocutors don’t). Perhaps you will humor me in responding to my comment. Let me try and make the issue as sharp as possible by pointing out what I think is an out-and-out mistake made by you. In the section you call the heart of your argument, you say.
Yes, the outcome is clearly the result of a “programming error” (in some sense). However, you then ask how a superintelligent machine could ignore such an “inconsistency in its reasoning.” But a programming error is not the same thing as an inconsistency in reasoning.
Note: I want to test your argument (at least at first), so I would rather not get a response from you claiming I’ve failed to take into account other arguments or other evidence, therefore my objection is invalid. Let me propose that you either 1) dispute that this was, in fact, a mistake, 2) explain how I have misunderstood, 3) grant that it was a mistake, and reformulate the claim here, or 4) state that this claim is not necessary for your argument.
If you can help me understand this point, I would be happy to continue to engage.