I’m still not sure why exactly people (I’m thinking of a few in particular, but this applies to many in the field) tell very detailed stories of AI domination like “AI will use protein nanofactories to embed tiny robots in our bodies to destroy all of humanity at the press of a button.” This seems like a classic use of the conjunction fallacy, and it doesn’t seem like those people really flinch from the word “and” like the Sequences tell them they should.
Furthermore, it seems like people within AI alignment aren’t taking the “sci-fi” criticism as seriously as they could. I don’t think most people who have that objection are saying “this sounds like science fiction, therefore it’s wrong.” I think they’re more saying “these hypothetical scenarios are popular because they make good science fiction, not because they’re likely.” And I have yet to find a strong argument against the latter form of that point.
Please let me know if I’m doing an incorrect “steelman,” or if I’m missing something fundamental here.
What’s preventing MIRI from making massive investments into human intelligence augmentation? If I recall correctly, MIRI is most constrained on research ideas, but human intelligence augmentation is a huge research idea that other grantmakers, for whatever reason, aren’t funding. There are plenty of shovel-ready proposals already, e.g. https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing; why doesn’t MIRI fund them?