There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.
Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we’d be able to conclude something about the true FPR.
There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.
Update: apparently it’s from this dataset which was published pre-2020, so also not from the post-LLM-era.
Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we’d be able to conclude something about the true FPR.
FWIW their human corpus is articles in places including Readers Digest, I wouldn’t actually be that shocked if that had some LLM content in late 2024.