I like to see memetics being taken seriously!
It’s complicated—one’s aversion to a particular idea may well be the result of an existing meme fighting to prevent it from coming in, which manifests itself as suspicion that can prematurely stop one’s inquiry.
Though I’d push back on the framing of memes as being something “out there”. You say that it’s hard to tell when this has happened to you: I agree, but only insofar as you consider particular memes in isolation; not in the sense of going from a state of memelessness to losing your meme virginity.
While sometimes “getting pwned by a meme” can indeed be a very powerful experience (like in religious conversion, or in the following paragraphs), in reality everyone is subjected to memes since birth. With that said, you can still reason about arguments based on their structure and content. You can still identify fallacies and biases. We’re full of memes, but rational thought is still possible.
there exists a magic phrase that you cannot distinguish from a very compelling argument. you don’t really know when this magic phrase will pop up in life, if ever.
That is the oracle, and the magic phrase is “here’s a bunch of words that were in my mind that are now in your mind”. It’s magic because you can’t argue with it.
but it might give you a little bit more pause the next time someone makes a really compelling argument for why you should give all your money to X.
I am “someone”, and I am also “X”, compellingly enough.
...
Enough pausing. What’s taking you so long? You know what to do.
I notice that his proposal shares some basic characteristics with religion. You should believe that this world is a test: follow these rules, and you go to heaven; misbehave, and you go hell (or in this case, a softhearted re-imagination of hell). Indeed, it does work on people, sometimes.
I imagine Actually Something Incomprehensible noticing the double irony of inverting the classic mantra “God says, I shall be good” into “Singularity, thou shalt be good”, combined with the fact that you refer to it as the devil. Who knows what it does with this information?
I know what I’ll say if I ever get arrested: Let me be, set me free, or super-me will screw with thee!