Humans are extremely susceptible to the arguments they have not been inoculated against. These arguments van be religious, scientific, emotional, financial, anything. One example is the new immigrants from certain places falling for get-rich-quick scams in disproportionally large numbers (not so much anymore, since the knowledge has spread). Or certain LW regulars believing Roko’s basilisk. Or become vegan (not all mind hacking is necessarily negative).
I would conjecture that every single one of us has open ports to be exploited (some more so than others), and someone with a good model of you, be it a super-smart AI or a police negotiator, can manipulate you into willingly doing stuff you would never have expected to be convinced of doing before having heard the argument.
Humans are extremely susceptible to the arguments they have not been inoculated against. These arguments van be religious, scientific, emotional, financial, anything. One example is the new immigrants from certain places falling for get-rich-quick scams in disproportionally large numbers (not so much anymore, since the knowledge has spread). Or certain LW regulars believing Roko’s basilisk. Or become vegan (not all mind hacking is necessarily negative).
I would conjecture that every single one of us has open ports to be exploited (some more so than others), and someone with a good model of you, be it a super-smart AI or a police negotiator, can manipulate you into willingly doing stuff you would never have expected to be convinced of doing before having heard the argument.