Thanks for your reply. I welcome an object-level discussion, and appreciate people reading my thoughts and showing me where they think I went wrong.
The hidden complexity of wishes stuff is not persuasive to me in the context of an argument that AI will literally kill everyone. If we wish for it not to, there might be some problems with the outcome, but it won’t kill everyone. In terms of Bay Area Lab 9324 doing something stupid, I think by the time thousands of labs are doing this, if we have been able to successfully wish for stuff without catastrophe being triggered, it will be relatively easy to wish for universal controls on the wishing technology.
“Infinite number of possible mesa-optimizers”. This feels like just invoking an unknown unknown to me, and then asserting that we’re all going to die, and feels like it’s missing some steps.
You’re wrong about Eliezer’s assertions about hacking, he 100% does believe by dint of a VR headset. I quote: “—Hack a human brain—in the sense of getting the human to carry out any desired course of action, say—given a full neural wiring diagram of that human brain, and full A/V I/O with the human (eg high-resolution VR headset), unsupervised and unimpeded, over the course of a day: DEFINITE YES—Hack a human, given a week of video footage of the human in its natural environment; plus an hour of A/V exposure with the human, unsupervised and unimpeded: YES ”
I get the analogy of all roads leading to doom, but it’s just very obviously not like that, because it depends on complex systems that are very hard to understand, and AI x-risk proponents are some of the biggest advocates of that opacity.
It’s very strange to me that there isn’t a central, accessible “101” version of the argument given how much has been written.
I don’t think anyone should make false claims, and this is an uncharitable mischaracterization of what I wrote. I am telling you that, from the outside view, what LW/rationalism gets attention for is the “I am sure we are all going to die”, which I don’t think is a claim most of its members hold, and this repels the average person because it violates common sense.
The object level responses you gave are so minimal and dismissive that I think they highlight the problem. “You’re missing the point, no one thinks that anymore.” Responses like this turn discussion into an inside-view only affair. Your status as a LW admin sharpens this point.