February 28th.
DanielFilan
Karma: 9,686
AXRP Episode 50 - Eli Lifland on AI 2027
Resistance training is mostly done to get stronger and to increase muscle mass, both of which have lots of downstream positive effects (including increased TDEE and better insulin sensitivity, not to mention benefits for attractiveness).
Also, perhaps crucially, you’re more able to move around in the world and carry + manipulate heavy objects!
Oops, yes. Have finally fixed this
FWIW their human corpus is articles in places including Readers Digest, I wouldn’t actually be that shocked if that had some LLM content in late 2024.
Doesn’t seem to be a translation artefact, partly because the English version also has some parts flagged as AI-written, and partly because as I say in the post:
Are you worried that this is somehow the fault of machine translation of human-composed text? I tried using Claude, a prominent LLM product, to translate some old encyclicals, and the results were flagged by Pangram as human-generated.
Re: the language of the original, these things get written by big teams, and most people on the teams are not native English speakers. See my thread with Anna Salmon for discussion on this point: my conclusion from the linked evidence is that Italian is indeed the most likely drafting language (as IIUC it is the working language of the Vatican) but Spanish is also possible.
Yep probably, oops
Try the paragraphs I list as being AI-generated? Also there should be a thing at the top that says “share” where you can get a shareable link.
Update: apparently it’s from this dataset which was published pre-2020, so also not from the post-LLM-era.
There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.
Yep, I could have been less abrasive in that interaction. That said, it just does seem pretty relevant for understanding how credible his denial is to know that he has other undisclosed AI writing!
Note that cardinals are probably not the ones doing the actual writing of these things—they’re too busy doing cardinal stuff. My guess is that it’s staff at the Dicastery for the Doctrine of the Faith and other departments of the Vatican.
Max Spero, CEO of Pangram Labs, claims that it was not trained on previous encyclicals.
Also, if Pangram were trained on previous encyclicals, you’d think that it would have learned that encyclical-like style is a tell for being human-generated.
We should not expect the Vatican to want or need to use AI to write documents meant for wide public consumption. Yes, the liturgical writing style they tend to use can be awkward, but reading and writing such documents is kinda what Catholic priests do. Perhaps it’s hard to believe that unassisted humans — in this age where people’s attention spans have been fried by phones and social media — can write something 2% as long as Zvi’s AI newsletter series, but an organization with the resources of the Catholic Church actually can. Also, the encyclical was presumably written and reviewed by several people working collaboratively; it’s not like it was just one guy who can secretly use ChatGPT on his own.
I just don’t see why you think we should be so confident about this. Indeed, the encyclical was presumably worked on collaboratively by multiple people. Lots of people use AI to help them write, it wouldn’t be so shocking to think that this includes some Vatican officials.
Re: whether it “reads as AI”, my understanding is that the whole reason Linch thought to put this into Pangram was that it read as AI. Most people don’t put everything they read into AI! You can also read someone on reddit saying the same thing here.
The rest of your post seems like it’s just batting away the evidence. Every time people have tested Pangram rigorously, it’s been shown to have a very low false negative [EDIT: I actually meant ‘positive’] rate. Furthermore, I think the evidence of old papal encyclicals not being flagged is also relevant here. It’s possible that that’s just because Pangram trained on those, but most people aren’t super interested in encyclicals, and Pangram Labs is a small enough company that I doubt they’ve trained on the whole internet. I think this would be legit if we had very very strong reasons to think there was no way that anyone at the Vatican would ever use AI in their writing, but I just don’t think you’ve offered those reasons.
The editor of Where Peter Is, a prominent Catholic analysis website, thinks that it was plausibly not composed in Italian.
Brief update: I looked at some of the footnotes in AI-generated paragraphs and didn’t find any issues with them.
This article made it to the Italian news!
(this is a longer turnaround than most of my podcasts, my defence is that both I and Eli were busy)