I strongly suspect significant fractions of Magnifica Humanitas, the Pope’s new encyclical on AI, is AI-generated or AI-assisted.
Evidence includes:
Pangram (Some examples include paragraphs 7-8. To a lesser degree, paragraphs 120-128)
I think you notice it too if you read it compared to other paragraphs
Also on a statistical level, the em-dash (“—”) was used 127 times in the recent encyclical.
Checking the last five encyclicals, it varies from 0 times to 23 times previously.
“genuinely” was used 9 times and “genuine” overall 22 times in today’s encyclical.
compared to 0 and 5 times, respectively, in the previous encyclical from 2024 (which appears to be of similar length).
In total, maybe 10-15% of the final draft was written by AI. Because the level of AI-writing varies significantly from section to section (from ~0% to ~100%), and because the styles are quite different, I suspect some cardinals who ghostwrote/contributed to it use AI-assistance much more than other people.
One complication here—the working language of the Vatican is ordinarily Italian and this document was probably drafted first in Italian (or possibly in a mish-mash of languages with intermediate translation steps). The clearest tells are probably in the Italian version. I don’t know how much LLM-based translation ends up rendering things in LLM voice, but it seems possible that part of the explanation comes from translation rather than drafting steps.
The same problems I observed in English appears to be in Italian. I did not do a systematic sweep (and also am not confident I can catch them in other languages; had to rely on Claude for the Italian analysis. Also Italian Pangram flags it more than English Pangram[1] (I dunno if Italian Pangram is over-sensitive though, I’ve only tested it before in English, and the published research on Pangram is also only on English text afaict).
I strongly suspect significant fractions of Magnifica Humanitas, the Pope’s new encyclical on AI, is AI-generated or AI-assisted.
Evidence includes:
Pangram (Some examples include paragraphs 7-8. To a lesser degree, paragraphs 120-128)
I think you notice it too if you read it compared to other paragraphs
Also on a statistical level, the em-dash (“—”) was used 127 times in the recent encyclical.
Checking the last five encyclicals, it varies from 0 times to 23 times previously.
“genuinely” was used 9 times and “genuine” overall 22 times in today’s encyclical.
compared to 0 and 5 times, respectively, in the previous encyclical from 2024 (which appears to be of similar length).
In total, maybe 10-15% of the final draft was written by AI. Because the level of AI-writing varies significantly from section to section (from ~0% to ~100%), and because the styles are quite different, I suspect some cardinals who ghostwrote/contributed to it use AI-assistance much more than other people.
Interesting.
One complication here—the working language of the Vatican is ordinarily Italian and this document was probably drafted first in Italian (or possibly in a mish-mash of languages with intermediate translation steps). The clearest tells are probably in the Italian version. I don’t know how much LLM-based translation ends up rendering things in LLM voice, but it seems possible that part of the explanation comes from translation rather than drafting steps.
Supposedly it’s higher in the Italian version. https://x.com/0xkartr/status/2058993778925490596
longer post forthcoming.
Is this language dependent?
The same problems I observed in English appears to be in Italian. I did not do a systematic sweep (and also am not confident I can catch them in other languages; had to rely on Claude for the Italian analysis. Also Italian Pangram flags it more than English Pangram[1] (I dunno if Italian Pangram is over-sensitive though, I’ve only tested it before in English, and the published research on Pangram is also only on English text afaict).
[1] https://x.com/0xkartr/status/2058993778925490596
There are not only these signs, but also around 10-15 occurrences of the “not only X, but also Y” pattern.
Fair point! Looking back at the 2024 encyclical there are also ~10 occurrences of this pattern, so this is not much evidence of AI-writing.