Computer security researcher working on evaluations for LLMs’ capability to attack hardware, software, and system users.
Personal page: https://fredheiding.com/
Google Scholar: https://scholar.google.se/citations?user=BJnWJVQAAAAJ
Fred Heiding
Karma: 116
Great flag. In our prior paper on email phishing, we show how AI models are more cost-efficient than even scammers in lower-income countries (https://arxiv.org/abs/2412.00586).
AI Voice Phishing Performs on Par With Human Scammers at a Fraction of the Cost
Jailbreaking AI models to Phish Elderly Victims
Great discussion. I’d add that it’s context-dependent and somewhat ambiguous. It’s noteworthy that our work shows that all tested AI models conflict with at least three of the eight prohibited AI practices outlined in the EU’s AI Act.
It’s also worth noting that the only real difference between sophisticated phishing and marketing can be the intention, making mitigation difficult. Actions from AI companies to prevent phishing might restrict legitimate use cases too much to be interesting.
Thanks for your feedback! It’s just a matter of time before scammers maximize their use of AI. Hopefully, the defense community can use this time to optimize our head start. Stay tuned for our coming work!
Voice phishing is improving quickly, but a Philippine worker (say earning $2/hour) would still be more profitable than the AI voice models, but I expect that to change in the coming years.