I’m ignorant, but my prior would’ve been more like 50⁄50 on “original was in English” / “original was in Italian”, since Pope Leo has English as his native/fluent language according to Wikipedia, and my ignorant prior guesses that this may also be true for a significant fraction of whoever else would’ve been involved in the drafting.
(edit: retracting this part.) So… I think the headline on the OP overstates the evidence, unless there’s reason to put more prior on Italian-first. (Vs “Many portions of Magnifica Humanitas appear to be partially AI-translated,” say). (Assuming Util’s comment is correct.)
Why I feel more confident that the Italian was original: I have some background impression from following Vatican news that these things are written by a team of people, and not the actual Pope himself (who is super busy and needs to delegate stuff, altho I assume he gives some outline to people and maybe does some edits). That team of people are presumably Vatican city staff, who presumably (a) use Italian as a working language due to being in Italy, and (b) are presumably not native English speakers just because most Catholics are not.
Why I feel more confident that the AI-generation flags are not due to AI translation of human-generated English-language content:
The English text does actually have some paragraphs that Pangram flags as AI written (e.g. the contrast between Nehemiah rebuilding Jerusalem vs the Tower of Babel)
when I take human-written text and ask AI to translate it, it does not get flagged as AI-generated
Example I just ran to double-check this: I asked Claude to translate a photo of a page from my Italian-language Ancient Greek textbook (which I don’t think has a human-written English translation available that Claude could be plagiarizing), and ran it through Pangram, which flags it as 100% human-written.
not the actual Pope himself (who is super busy and needs to delegate stuff
So, the Pope delegates to a Cardinal, the Cardinal delegates to an Archbishop, the Archbishop delegates to a Bishop, the Bishop delegates to a Priest, the Priest delegates to a Deacon, and the Deacon uses ChatGPT.
Hmm; your second point is pretty compelling to me (“when I take human-written text and ask AI to translate it, it does not get flagged as AI-generated”). I’m curious how robust that is.
I don’t find your argument about probably-Italian-first compelling yet: it seems plausible/likely to me that various folks contributed to the writing who are e.g. Pope Leo himself, or others high up in the Catholic church (perhaps selecting partly for tech-familiarity), and English fluency seems as likely there as Italian fluency to me. I would guess folks who grew up in (Rome, especially Rome outside the Vatican) are more Italian-fluent, but don’t see why I should expect authorship-average to be folks native to Rome. Maybe I’m being dense.
Hmm; your second point is pretty compelling to me (“when I take human-written text and ask AI to translate it, it does not get flagged as AI-generated”). I’m curious how robust that is.
Literally 100% for me asking Claude to take text and translate it (including old Papal encyclicals), but that’s small n and maybe it is copying the style of publicly-available human-written translations. That said Pangram gives you a couple of free checks a day and I’d encourage people to play around with this!
I’ve just tried this once, getting Claude to write a paragraph in Latin, translating it to English myself, and plugging it into Pangram. Surprisingly to me, Pangram flags it as 100% AI-generated, even though it doesn’t sound all that AI-ish to my ear (but admittedly I don’t have the most acute ear for such things). Also note that the translation was kind of a rush, and I could probably have made it sound more pleasing (e.g. “a big rock” could probably have been “a large rock”, and “I’m in the habit of walking” is not obviously the best way of rendering “ambulare soleo”).
I agree that the authors are probably not native to Rome. That said, my guess is most Vatican officials are either literally Italian (recall that John Paul II was the first non-Italian pope in a long time) or from a bunch of random countries such that the natural language to work in is that of the country that you’re in (which in this case is basically Italian).
I also agree these arguments aren’t air-tight, and would like to hear from people who know more (e.g. Vatican reporters).
Hmm, Daniel Gallagher, a strong Latinist who’s worked in the Curia, writes:
Latin is still the “official” language of the Roman Catholic Church and the Magisterium, and I have little doubt anything will ever take its place. But it is not the “working” language of the Roman Curia, and likely never will be again. Italian has had that status for some time, though English has practically replaced it over the last two decades. A few Cardinals have mentioned the importance of having had English-speaking electors on hand during the recent conclave, as many members of the College who needed help did not have Italian as a second language.
(h/t Linch Zhang for finding this article)
I think I’m now falling back to “if it wasn’t composed in Italian, I’m confused why the Italian version would show as more AI-generated”.
From the same piece, this paragraph makes it sound like Italian is the working language of encyclicals (albeit not definitively). Bolding mine:
Whereas a hundred years ago the Latin version was produced first and then the vernaculars were rendered as translations of it, now the Latin version is the last to be completed and is essentially a fine-tuned and carefully raked Latin translation of the Italian.
I’m ignorant, but my prior would’ve been more like 50⁄50 on “original was in English” / “original was in Italian”, since Pope Leo has English as his native/fluent language according to Wikipedia, and my ignorant prior guesses that this may also be true for a significant fraction of whoever else would’ve been involved in the drafting.
Kartr says Pangram says the English version was basically not AI-authored/assisted.
(edit: retracting this part.)
So…
I think the headline on the OP overstates the evidence, unless there’s reason to put more prior on Italian-first. (Vs “Many portions of Magnifica Humanitas appear to be partially AI-translated,” say). (Assuming Util’s comment is correct.)Why I feel more confident that the Italian was original: I have some background impression from following Vatican news that these things are written by a team of people, and not the actual Pope himself (who is super busy and needs to delegate stuff, altho I assume he gives some outline to people and maybe does some edits). That team of people are presumably Vatican city staff, who presumably (a) use Italian as a working language due to being in Italy, and (b) are presumably not native English speakers just because most Catholics are not.
Why I feel more confident that the AI-generation flags are not due to AI translation of human-generated English-language content:
The English text does actually have some paragraphs that Pangram flags as AI written (e.g. the contrast between Nehemiah rebuilding Jerusalem vs the Tower of Babel)
when I take human-written text and ask AI to translate it, it does not get flagged as AI-generated
Example I just ran to double-check this: I asked Claude to translate a photo of a page from my Italian-language Ancient Greek textbook (which I don’t think has a human-written English translation available that Claude could be plagiarizing), and ran it through Pangram, which flags it as 100% human-written.
Here’s me asking Claude to translate a random page of Money-Pump Arguments into Italian. This is a niche enough monograph that I’m pretty sure a translation is not available. Pangram thinks it’s 100% human-written.
So, the Pope delegates to a Cardinal, the Cardinal delegates to an Archbishop, the Archbishop delegates to a Bishop, the Bishop delegates to a Priest, the Priest delegates to a Deacon, and the Deacon uses ChatGPT.
God works in mysterious ways.
Hmm; your second point is pretty compelling to me (“when I take human-written text and ask AI to translate it, it does not get flagged as AI-generated”). I’m curious how robust that is.
I don’t find your argument about probably-Italian-first compelling yet: it seems plausible/likely to me that various folks contributed to the writing who are e.g. Pope Leo himself, or others high up in the Catholic church (perhaps selecting partly for tech-familiarity), and English fluency seems as likely there as Italian fluency to me. I would guess folks who grew up in (Rome, especially Rome outside the Vatican) are more Italian-fluent, but don’t see why I should expect authorship-average to be folks native to Rome. Maybe I’m being dense.
I have 3 lines of arguments against the translation artifact hypothesis (though my post takes the English-language article as a starting point and doesn’t take a strong position on the source language) https://linch.substack.com/i/199316613/probably-not-a-translation-artifact
Literally 100% for me asking Claude to take text and translate it (including old Papal encyclicals), but that’s small n and maybe it is copying the style of publicly-available human-written translations. That said Pangram gives you a couple of free checks a day and I’d encourage people to play around with this!
(Ideally other people would check with LLMs other than Claude)
The converse question is more of a bother, but: any data on how human translations of AI-generated text show up on Pangram?
I’ve just tried this once, getting Claude to write a paragraph in Latin, translating it to English myself, and plugging it into Pangram. Surprisingly to me, Pangram flags it as 100% AI-generated, even though it doesn’t sound all that AI-ish to my ear (but admittedly I don’t have the most acute ear for such things). Also note that the translation was kind of a rush, and I could probably have made it sound more pleasing (e.g. “a big rock” could probably have been “a large rock”, and “I’m in the habit of walking” is not obviously the best way of rendering “ambulare soleo”).
None that I have.
I agree that the authors are probably not native to Rome. That said, my guess is most Vatican officials are either literally Italian (recall that John Paul II was the first non-Italian pope in a long time) or from a bunch of random countries such that the natural language to work in is that of the country that you’re in (which in this case is basically Italian).
I also agree these arguments aren’t air-tight, and would like to hear from people who know more (e.g. Vatican reporters).
ChatGPT5.5 estimates 50% Italian-first, 35% English-first, as prior.
Academia often has people “from a bunch of random countries” and often defaults to English.
Hmm, Daniel Gallagher, a strong Latinist who’s worked in the Curia, writes:
(h/t Linch Zhang for finding this article)
I think I’m now falling back to “if it wasn’t composed in Italian, I’m confused why the Italian version would show as more AI-generated”.
From the same piece, this paragraph makes it sound like Italian is the working language of encyclicals (albeit not definitively). Bolding mine:
The editor of Where Peter Is, a prominent Catholic analysis website, thinks that it was plausibly not composed in Italian.