DanielFilan

Karma: 9,716

DanielFilan 10 Sep 2026 18:09 UTC
2 points
0
in reply to: DanielFilan’s comment on: Estimating GPT-6 Astra’s no-CoT Time Horizon
Another way of saying this: the [8m, 1h] CI basically comes from assuming that if you got more >2h benchmarks, Astra would get 0% on literally all of them. I don’t think that’s a reasonable assumption, especially just eye-balling your figure 1 (which would make me think that Astra would plausibly get around 50% on new benchmarks in the 2-4 hour bucket), and from looking at Neel’s post where it seems like Astra has an unusually lopsided no-CoT skill profile and is really great at some types of tasks.

DanielFilan 10 Sep 2026 17:54 UTC
3 points
0
in reply to: Francis Rhys Ward’s comment on: Estimating GPT-6 Astra’s no-CoT Time Horizon
Agree that the sigmoid model is clearly not actually right, and also that more benchmarks are needed, and that it’s reasonable to think there exist benchmarks that Astra gets 0% on without CoT. That said, I actually have no idea whether Astra would get literally 0% on benchmarks of 2 hours (which it sounds like is one of the things in your fit), or whether there exist possible benchmarks of >2 hours where Astra would not get 0% on, which would increase the TH estimate. Like, just based on the data you have, it really would not surprise me at all if Astra had a 60% success rate on tasks in the 2-4 hour bucket!

Basically overall, I feel pretty skeptical of the “add some synthetic 0% benchmarks” methodology, and would prefer a takeaway of “Astra doesn’t actually have a well-defined no-CoT TH because success rate is not sigmoidal in task length, but if it did, it would probably be somewhere north of 8 minutes”.

DanielFilan 9 Sep 2026 19:28 UTC
4 points
0
on: Estimating GPT-6 Astra’s no-CoT Time Horizon
To account for the fact that our task suite has an insufficient number of benchmarks with long time horizons, we assume that there is some number (N) of benchmarks with problems of length x (2-96hours in the main plot). To compute the 50% TH we assume astra gets 0% performance on these benchmarks.
Is this a reasonable assumption? Intuitively, it would be surprising if astra got non-zero no-CoT performance on very long benchmarks (e.g., a no-CoT TH of 32 hours seems a priori implausible).
I don’t get why this is a reasonable assumption. To the degree you believe the sigmoid model, it’s apparently not what that model predicts on your data (given that it changes your TH estimate), so I suppose it must be from other evidence. But what is that other evidence?

DanielFilan 3 Aug 2026 0:49 UTC
2 points
0
in reply to: DanielFilan’s comment on: AXRP Episode 50 - Eli Lifland on AI 2027
(this is a longer turnaround than most of my podcasts, my defence is that both I and Eli were busy)

DanielFilan 3 Aug 2026 0:48 UTC
4 points
0
in reply to: anaguma’s comment on: AXRP Episode 50 - Eli Lifland on AI 2027
February 28th.

AXRP Episode 50 - Eli Lifland on AI 2027

DanielFilan3 Aug 2026 0:40 UTC

20 points

4 comments80 min readLW link

DanielFilan 7 Jul 2026 5:14 UTC
7 points
0
on: Info dump: Some Important Models for Health and Fitness
Resistance training is mostly done to get stronger and to increase muscle mass, both of which have lots of downstream positive effects (including increased TDEE and better insulin sensitivity, not to mention benefits for attractiveness).
Also, perhaps crucially, you’re more able to move around in the world and carry + manipulate heavy objects!

DanielFilan 31 May 2026 8:27 UTC
3 points
0
in reply to: Justin Priebe’s comment on: Many portions of Magnifica Humanitas appear to be AI-written
Oops, yes. Have finally fixed this

DanielFilan 31 May 2026 8:18 UTC
6 points
2
in reply to: Nisan’s comment on: no, Magnifica Humanitas is not AI-written
FWIW their human corpus is articles in places including Readers Digest, I wouldn’t actually be that shocked if that had some LLM content in late 2024.

DanielFilan 29 May 2026 14:23 UTC
2 points
0
in reply to: Joshua Harris’s comment on: Many portions of Magnifica Humanitas appear to be AI-written
Doesn’t seem to be a translation artefact, partly because the English version also has some parts flagged as AI-written, and partly because as I say in the post:

Are you worried that this is somehow the fault of machine translation of human-composed text? I tried using Claude, a prominent LLM product, to translate some old encyclicals, and the results were flagged by Pangram as human-generated.

Re: the language of the original, these things get written by big teams, and most people on the teams are not native English speakers. See my thread with Anna Salmon for discussion on this point: my conclusion from the linked evidence is that Italian is indeed the most likely drafting language (as IIUC it is the working language of the Vatican) but Spanish is also possible.

DanielFilan 29 May 2026 14:17 UTC
2 points
0
in reply to: Billy M’s comment on: Many portions of Magnifica Humanitas appear to be AI-written
Yep probably, oops

DanielFilan 29 May 2026 14:17 UTC
2 points
0
in reply to: Joe Breslin’s comment on: Many portions of Magnifica Humanitas appear to be AI-written
Try the paragraphs I list as being AI-generated? Also there should be a thing at the top that says “share” where you can get a shareable link.

DanielFilan 28 May 2026 17:04 UTC
10 points
0
in reply to: DanielFilan’s comment on: no, Magnifica Humanitas is not AI-written
Update: apparently it’s from this dataset which was published pre-2020, so also not from the post-LLM-era.

DanielFilan 28 May 2026 15:09 UTC
4 points
0
in reply to: Nisan’s comment on: no, Magnifica Humanitas is not AI-written
There are a few other studies: one of them looks at magazine articles published in 2024 as the “human-written set” and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There’s another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I’ll ask the authors when those articles came out but I would assume late 2025.

DanielFilan 28 May 2026 15:05 UTC
4 points
4
in reply to: bhauth’s comment on: no, Magnifica Humanitas is not AI-written
Yep, I could have been less abrasive in that interaction. That said, it just does seem pretty relevant for understanding how credible his denial is to know that he has other undisclosed AI writing!

DanielFilan 27 May 2026 21:16 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: no, Magnifica Humanitas is not AI-written
Note that cardinals are probably not the ones doing the actual writing of these things—they’re too busy doing cardinal stuff. My guess is that it’s staff at the Dicastery for the Doctrine of the Faith and other departments of the Vatican.

DanielFilan 27 May 2026 20:57 UTC
15 points
0
in reply to: DanielFilan’s comment on: no, Magnifica Humanitas is not AI-written
Max Spero, CEO of Pangram Labs, claims that it was not trained on previous encyclicals.

DanielFilan 27 May 2026 20:14 UTC
6 points
0
in reply to: DanielFilan’s comment on: no, Magnifica Humanitas is not AI-written
Also, if Pangram were trained on previous encyclicals, you’d think that it would have learned that encyclical-like style is a tell for being human-generated.

DanielFilan 27 May 2026 20:13 UTC
19 points
14
on: no, Magnifica Humanitas is not AI-written

We should not expect the Vatican to want or need to use AI to write documents meant for wide public consumption. Yes, the liturgical writing style they tend to use can be awkward, but reading and writing such documents is kinda what Catholic priests do. Perhaps it’s hard to believe that unassisted humans — in this age where people’s attention spans have been fried by phones and social media — can write something 2% as long as Zvi’s AI newsletter series, but an organization with the resources of the Catholic Church actually can. Also, the encyclical was presumably written and reviewed by several people working collaboratively; it’s not like it was just one guy who can secretly use ChatGPT on his own.

I just don’t see why you think we should be so confident about this. Indeed, the encyclical was presumably worked on collaboratively by multiple people. Lots of people use AI to help them write, it wouldn’t be so shocking to think that this includes some Vatican officials.

Re: whether it “reads as AI”, my understanding is that the whole reason Linch thought to put this into Pangram was that it read as AI. Most people don’t put everything they read into AI! You can also read someone on reddit saying the same thing here.

The rest of your post seems like it’s just batting away the evidence. Every time people have tested Pangram rigorously, it’s been shown to have a very low false negative [EDIT: I actually meant ‘positive’] rate. Furthermore, I think the evidence of old papal encyclicals not being flagged is also relevant here. It’s possible that that’s just because Pangram trained on those, but most people aren’t super interested in encyclicals, and Pangram Labs is a small enough company that I doubt they’ve trained on the whole internet. I think this would be legit if we had very very strong reasons to think there was no way that anyone at the Vatican would ever use AI in their writing, but I just don’t think you’ve offered those reasons.

DanielFilan 27 May 2026 20:07 UTC
9 points
4
in reply to: Shankar Sivarajan’s comment on: no, Magnifica Humanitas is not AI-written
Pangram says 100% human-written

DanielFilan

AXRP Epi­sode 50 - Eli Lifland on AI 2027

AXRP Episode 50 - Eli Lifland on AI 2027