We should not expect the Vatican to want or need to use AI to write documents meant for wide public consumption. Yes, the liturgical writing style they tend to use can be awkward, but reading and writing such documents is kinda what Catholic priests do. Perhaps it’s hard to believe that unassisted humans — in this age where people’s attention spans have been fried by phones and social media — can write something 2% as long as Zvi’s AI newsletter series, but an organization with the resources of the Catholic Church actually can. Also, the encyclical was presumably written and reviewed by several people working collaboratively; it’s not like it was just one guy who can secretly use ChatGPT on his own.
I just don’t see why you think we should be so confident about this. Indeed, the encyclical was presumably worked on collaboratively by multiple people. Lots of people use AI to help them write, it wouldn’t be so shocking to think that this includes some Vatican officials.
Re: whether it “reads as AI”, my understanding is that the whole reason Linch thought to put this into Pangram was that it read as AI. Most people don’t put everything they read into AI! You can also read someone on reddit saying the same thing here.
The rest of your post seems like it’s just batting away the evidence. Every time people have tested Pangram rigorously, it’s been shown to have a very low false negative [EDIT: I actually meant ‘positive’] rate. Furthermore, I think the evidence of old papal encyclicals not being flagged is also relevant here. It’s possible that that’s just because Pangram trained on those, but most people aren’t super interested in encyclicals, and Pangram Labs is a small enough company that I doubt they’ve trained on the whole internet. I think this would be legit if we had very very strong reasons to think there was no way that anyone at the Vatican would ever use AI in their writing, but I just don’t think you’ve offered those reasons.
I fed a couple of my very recent blog posts to Pangram, because I do love em-dashes and bulleted lists. And in an unscientific sample, Pangram scored them as 100% human.
This has updated my prior on Pangram: It does not appear to trivially fail on the kind of writing that I do, given a totally inadequate sample size.
I recently took a look at Jabarian & Imas 2025 and was disappointed to find that their human-authored texts came from public pre-LLM-era datasets. So I can’t trust the very low false positive rate without reasoning about Pangram’s training process. (To be fair, it would be expensive to commission 1,992 human-authored texts just to evaluate Pangram.)
There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.
Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we’d be able to conclude something about the true FPR.
Also as a model size/compressibility issue, it’s actually really hard to “cheat” by memorizing the entire internet! Claude Opus couldn’t do it and Claude Opus’s almost certainly bigger and had more training time than Pangram Labs’s classifier.
Also, if Pangram were trained on previous encyclicals, you’d think that it would have learned that encyclical-like style is a tell for being human-generated.
I just don’t see why you think we should be so confident about this. Indeed, the encyclical was presumably worked on collaboratively by multiple people. Lots of people use AI to help them write, it wouldn’t be so shocking to think that this includes some Vatican officials.
Re: whether it “reads as AI”, my understanding is that the whole reason Linch thought to put this into Pangram was that it read as AI. Most people don’t put everything they read into AI! You can also read someone on reddit saying the same thing here.
The rest of your post seems like it’s just batting away the evidence. Every time people have tested Pangram rigorously, it’s been shown to have a very low false negative [EDIT: I actually meant ‘positive’] rate. Furthermore, I think the evidence of old papal encyclicals not being flagged is also relevant here. It’s possible that that’s just because Pangram trained on those, but most people aren’t super interested in encyclicals, and Pangram Labs is a small enough company that I doubt they’ve trained on the whole internet. I think this would be legit if we had very very strong reasons to think there was no way that anyone at the Vatican would ever use AI in their writing, but I just don’t think you’ve offered those reasons.
Max Spero, CEO of Pangram Labs, claims that it was not trained on previous encyclicals.
I fed a couple of my very recent blog posts to Pangram, because I do love em-dashes and bulleted lists. And in an unscientific sample, Pangram scored them as 100% human.
This has updated my prior on Pangram: It does not appear to trivially fail on the kind of writing that I do, given a totally inadequate sample size.
I recently took a look at Jabarian & Imas 2025 and was disappointed to find that their human-authored texts came from public pre-LLM-era datasets. So I can’t trust the very low false positive rate without reasoning about Pangram’s training process. (To be fair, it would be expensive to commission 1,992 human-authored texts just to evaluate Pangram.)
There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.
Update: apparently it’s from this dataset which was published pre-2020, so also not from the post-LLM-era.
Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we’d be able to conclude something about the true FPR.
FWIW their human corpus is articles in places including Readers Digest, I wouldn’t actually be that shocked if that had some LLM content in late 2024.
Also as a model size/compressibility issue, it’s actually really hard to “cheat” by memorizing the entire internet! Claude Opus couldn’t do it and Claude Opus’s almost certainly bigger and had more training time than Pangram Labs’s classifier.
Also, if Pangram were trained on previous encyclicals, you’d think that it would have learned that encyclical-like style is a tell for being human-generated.