I see a fair amount of back-and-forth where someone says “What about this?” and you say “I addressed that in several places; clearly you didn’t read it.” Unfortunately, while you may think you have addressed the various issues, I don’t think you did (and presumably your interlocutors don’t). Perhaps you will humor me in responding to my comment. Let me try and make the issue as sharp as possible by pointing out what I think is an out-and-out mistake made by you. In the section you call the heart of your argument, you say.
If the AI is superintelligent (and therefore unstoppable), it will be smart enough to know all about its own limitations when it comes to the business of reasoning about the world and making plans of action. But if it is also programmed to utterly ignore that fallibility—for example, when it follows its compulsion to put everyone on a dopamine drip, even though this plan is clearly a result of a programming error—then we must ask the question: how can the machine be both superintelligent and able to ignore a gigantic inconsistency in its reasoning?
Yes, the outcome is clearly the result of a “programming error” (in some sense). However, you then ask how a superintelligent machine could ignore such an “inconsistency in its reasoning.” But a programming error is not the same thing as an inconsistency in reasoning.
Note: I want to test your argument (at least at first), so I would rather not get a response from you claiming I’ve failed to take into account other arguments or other evidence, therefore my objection is invalid. Let me propose that you either 1) dispute that this was, in fact, a mistake, 2) explain how I have misunderstood, 3) grant that it was a mistake, and reformulate the claim here, or 4) state that this claim is not necessary for your argument.
If you can help me understand this point, I would be happy to continue to engage.
Hi. I’m a long time lurker (a few years now), and I finally joined so that I could participate in the community and the discussions. This was borne partly out of a sense that I’m at a place in my life where I could really benefit from this community (and it could benefit from me), and partly out of a specific interest in some of the things that have been posted recently: the MIRI technical research agenda.
In particular, once I’ve had more time to digest it, I want to post comments and questions about Reasoning Under Logical Uncertainty.
More about me: I’m currently working as a postdoctoral fellow in mathematics. My professional work is in physics-y differential geometry, so only connected to the LW material indirectly via things like quantum mechanics. I practice Buhddist meditation, without definitively endorsing any of the doctrines. I’m surprised meditation hasn’t gotten more airtime in the rationalist community.
My IRL exposure to the LWverse is limited (hi Critch!), but I gather there’s a meetup group in Utrecht, where I’m living now.
Anyway, I look forward to good discussions. Hello everyone!