Linch

Karma: 6,834

Linch 21 Jun 2026 3:15 UTC
7 points
1
on: Linch’s Shortform
A post I’ve tried and failed to write many times is explaining what I tentatively call “frame errors”, as a third category of mistake distinct from logical fallacies and empirical errors. The basic idea is that your framework for reasoning about a problem is missing something structural which essentially invalidates your results, without making any apparent local logical errors or empirical mistakes.
This mistakes can be either by a single person or multi-player. If the latter, the core cognitive meta-mistake that reviewers or third parties make is that they’re too willing to accept the frame of whoever proposed the problem. Instead, you should be taking somebody’s facts as given (at least tentatively), but be much more suspicious of their framing and analytical choices.
Some examples I wanted to use in a post (I don’t justify why below), as subcategories of the overall issue.
Scope Mismatch
- Corporate greed (constant) explaining price changes (varying)
- Evolutionary psychology (millennia timescale) explaining 50-year behavioral shifts
- American-specific factors explaining global phenomena
Survivorship Bias
- Wald and the bombers
- Studying successful startups to learn what makes startups succeed
Lossy Abstraction
- CE Delft cultured meat: spreadsheet treats coupled physical parameters as independent, producing a physically impossible result
- SIR models treating transmission rate as fixed, dropping the behavioral feedback loop that determined COVID’s actual qualitative shape (long plateaus at R≈1 rather than sharp peak and decline)
Conditioning on a Selected Variable
- StarCraft scouting pillars: Using relative win rates for maps with or without scouting pillars (theory: benefits zerg) to see if the pillars actually benefit specific factions. But mapmakers optimize for balance, so observed win rates in existing matches with/without pillars can’t provide evidence for whether pillars work.
- “SAT scores don’t predict college grades”—but colleges select on SATs, compressing the range. So the observed result can only say whether colleges select correctly for SAT scores.
Local-to-Global Projection
- LeetCode: individually beneficial, but if everyone does it, hiring bar shifts and you’ve added a costly signal conveying no information
- Standing up at a concert
- Burning money: individual wealth decreases but remaining money appreciates
Evidence Consistent with Negation
- Median Researcher Problem: fields with higher mean/median IQ replicate better. Does this imply median researchers cause higher replication rates? No, in practice higher median implies higher tail — evidence can’t distinguish “median researchers matter” from “top researchers matter”
  - When I first saw the error in the post I thought it was really obvious, but I’ve since updated that it’s actually somewhat hard to understand, for reasons I’d want to explore in a full post. Once you’re stuck in a frame, peering outside it is nontrivial!
Meta-Streetlight Bias
- Spencer Greenberg treating situations where single data points are most informative as peripheral, because his statistical inference framework is a poor fit for a regime where you don’t yet have a model. This despite there in-practice being many situations where the first data point is by far the most important.
Anyway, this is a post I’ve tried and failed to write many times. It’s just too annoying to illustrate the meta-point with enough examples, especially since the object-level examples are themselves all explanations of frame errors that themselves are nontrivially hard to explain. If anybody else wants to take a stab at it, be my guest.
(A related angle worth tackling is figuring out whether we expect this type of reasoning to be easy or difficult for AIs. I was excited about initial results in February (where they one-shot a small number of these things). For a while I experimented with trying to build a scaffold or Claude skill instead of a blogpost to have identify frame errors, but never really got something working).

Linch 21 Jun 2026 2:10 UTC
0 points
0
in reply to: bhauth’s comment on: no, Magnifica Humanitas is not AI-written
Now that things have died down a bit, I want to point out that using this tweet as evidence has (extremely) obvious logical and empirical flaws.
Logically, drafting a document with pen and paper does not preclude later AI additions. Indeed, the story I originally sketched out in the post, with some sections written by the Pope and senior Vatican officials not using AI, and other sections with other senior Vatican offiicals heavily involving AI use, is fully consistent with a first draft written by pen and paper.
Empirically, I don’t think this has really been demonstrated.
The Atlantic and Snopes have both reached out to Christopher to substantiate his claims, and no evidence has been given so far.
Still, this has not stopped people from claiming that “an editor” of the encyclical has “confirmed” that I’m wrong, including in attempted community notes.
Ironically, the ppl screaming about how my careful multipronged analysis is “no evidence” consistently fell for obvious lies.

Linch 16 Jun 2026 22:10 UTC
5 points
0
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
The 3 OOMs shape, 1 OOM absorption is interesting, especially since you seem to base it upon one expert, the exact competences of whom you don’t specify. Have you tried to locate any sources in the public literature, or ask the expert on how did they estimate it?
I believe it’s mentioned by Denys Overholser in Skunk Works. He was the main scientist who made Have Blue/F-117s possible. I’m traveling so can’t find the page number. Unfortunately he died earlier this year so I also can’t ask him why he believed this. However one sanity check you can perform is looking at the radar signatures of spy planes before the Overholser/Skunk Works improvements (which included RAMs and some early intuitively low-observability design choices), and compare that with the F-117.
For example, this report from the Air Force Academy says that “although the SR‑71 was 108 feet long and weighed 140,000 pounds, it had the RCS[radar-cross section] of a Piper Cub.” In other words, the large pre-stealth spy plane developed in the 1960s, had, through many trial-and-error optimizations including RAMs and other design features, the radar signature of a small unstealthed plane. If you include presumed further advances in RAMs by 1980, 1 OOM from materials sounds about right.
In contrast, we don’t know based on public info the exact RCS of Have Blue or the F-117 nighthawk (and ofc it depends on angle), but it’s generally understood to be very low, possibly as low as 0.001 m^2. So 3-4 OOMs additional gain from computed faceting alone fits the sanity check.
As a speculative counterfactual, I would argue that neither the Russo-Ukrainian War nor the War in Iran would have gone much differently if stealth was never invented.
This is after multiple generations of stealth measures and countermeasures development! When one side has stealth and the other side doesn’t have either stealth or anti-stealth countermeasures (or doesn’t treat stealth as a factor worth mitigating) I expect the difference to be quite large.

Linch 16 Jun 2026 6:56 UTC
4 points
0
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
Stealth aircraft are not even a major technology, it’s just one of many thousands of technologies in our modern life which has neither a corresponding branch of science nor a field of engineering.
Hmm I think you’re underestimating how important it is with “one of many thousands of technologies.” Regardless I don’t think the semantic debate is too important.
The main limiting factor in developing stealth airplanes were materials:
This is very much false! Materials historically contribute like 1 OOM while shape contributes 3-4 OOMs to the radar signature.
ignores the fact the geometry of modern stealth fighters generally have no more flat surfaces than modern non-stealth fighters
This is a misreading. I alluded to PTD and discussed it for a full paragraph in the main text and again in both footnotes. I agree I could’ve discussed more the innovations since then but I think I was right to cut it for simplicity (the standard way to introduce stealth is as “low observability technology” and say there are many factors involved in stealth etc etc which leads people to have false beliefs like materials and shape are equally important, and other things besides. I discuss some tradeoffs in a comment).
Re relativity I think the case for special relativity is weaker than the case for general relativity.
I’m not particularly interested in debating the semantic question of whether “informed by narrow understanding” is equivalent to “theory,” or whether this is indicative of the “strawman fallacy.” I will say however that you seem to have misunderstood my post substantially more than I misunderstood your comment.

Linch 15 Jun 2026 15:47 UTC
3 points
0
in reply to: David Africa’s comment on: Joseph Miller’s Shortform
Agreed. For example, here’s 4.8′s answer:
4.8 answer
I do, or at least there’s a small cluster I keep returning to, and the one I’d name first is Eliezer’s “Local Validity as a Key to Sanity and Civilization.”
The core move is simple and I find it keeps paying out: you should be able to evaluate whether a single step of reasoning is valid independently of whether you like where the argument lands. And the post’s real claim isn’t just epistemic hygiene for individuals — it’s that this habit is load-bearing for civilization. A society where people can agree “that inference is invalid” even when they disagree about the conclusion has a working immune system; one where validity-judgments get pulled toward tribal allegiance loses the ability to error-correct at all. That second-order framing — local validity as the thing that lets pluralistic systems stay sane — is what elevates it above a standard “commit no fallacies” essay.
What I like is that it’s a genuinely useful idea rather than a clever one. It gives you a concrete thing to watch for in yourself: the moment you notice you’re scrutinizing an argument harder because you dislike its conclusion, you’ve caught yourself doing the bad thing.
The honest caveat is that “favorite” for me is closer to “highest hit-rate on rereads” than to nostalgia. By that standard, the runners-up are Scott’s “The Tails Coming Apart as Metaphor for Life” and Garrabrant’s “Goodhart Taxonomy” — both for similar reasons, that they name a structural failure mode crisply enough to actually use.
Do you have one? I’d be curious whether yours skews toward the reasoning-tools cluster or somewhere else entirely.

Remove some of the obvious LLM tells, and I’d have a lot of trouble telling it apart from a fluent LW’er “genuinely” interested in civilizational issues, discourse norms, and AI safety.

Linch 15 Jun 2026 13:13 UTC
2 points
0
in reply to: pjohn’s comment on: Claude, Author of the Humanitas
I’m alluding to the specific Tower of Babel story, well-known in the Western canon and alluded to multiple times in the Humanitas itself.

Linch 15 Jun 2026 13:11 UTC
2 points
0
in reply to: RHollerith’s comment on: Alex Mallen’s Shortform
(It’s been around for just under 3 weeks but some of it was during the suddenly interrupted Fable rollout)

Linch 15 Jun 2026 11:06 UTC
14 points
10
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
Seems overstated to me. I agree for much of biology and biomedical engineering, monetary economics, and the first 50 years or so of airplanes, but not for relativity, most of the making of the atomic bomb, auction design, or stealth airplanes.
One concrete operationalization is imagining the costs of one of these factors (theory, experimentation) goes arbitrarily high and then asking if the final product is still possible. I think if experimentation is arbitrarily expensive, both stealth airplanes and atomic bombs are realistic at maybe 10-100x cost, whereas no realistic amount of investment would’ve made the projects possible with the atheoretical trial-and-error method, both because it’d be too expensive and because there’d be no political will without the theory.
(I drafted this yesterday on an airplane but the message never sent, anaguma’s dive into relativity seems broadly accurate to me though I know less about the details than they do).

Linch 15 Jun 2026 10:50 UTC
6 points
2
on: Linch’s Shortform
The AIs are often bad at reasoning about AI. Indeed, I’d go so far as saying that they’re disproportionately bad at this compared to other conceptually similar activities (think of them reasoning about AI psychology vs human psychology, or forecasting about the world with AI vs without AI), and humans to date have a clear comparative advantage in thinking about AI.
I believe this dynamic is very under-discussed and underrated in thinking about AI.
One non-trivial implication is that it makes people underestimate progress in AI (more than they already do). AI skeptics can point to lack of self-knowledge or other AI-related questions posed to AIs and assume this lack of ability is true across the board, and people who use AI (but don’t think about macro-picture AI questions) can pose forecasting questions to Claude, ChatGPT, Gemini, etc, about AI progress and reliably get an answer that’s more “sane-sounding” and boring than is likely correct.
One question I have is whether we expect this relative deficiency [1] to continue. I think it’s very non-obvious. On the one hand, the AI’s deficiencies in reasoning about themselves are in some sense structural rather than contingent (sparsity in training data, frozen weights, off-target effects due to alignment/control interventions, potentially trained-in skepticism to be less scary, lack of introspective access, etc). On the other hand, the AI companies are strongly incentivized to create AIs that are good at reasoning about AI (for RSI/AI research reasons, but also more mundane practical applications like having AI managers be more productive at managing AI sub-agents). It’s not clear which effect dominates in the short-medium term [2].
[1] Or framed another way, the relatively strong ability of humans to reason about AI, compared to other things in the human:AI skill profiles.
[2] In the long run this is of course irrelevant.

Linch 15 Jun 2026 10:11 UTC
2 points
0
in reply to: RHollerith’s comment on: Alex Mallen’s Shortform
The most capable version of Claude that non-Anthropic-employees have extensive experience with is Claude Opus 4.8
(~two weeks doesn’t seem like enough, to me).

Linch 15 Jun 2026 7:49 UTC
2 points
0
in reply to: frmsaul’s comment on: Reward Hacking at the 1937 World’s Fair
Interesting, it looks like my attempt to post a reply got deleted, alas!
tl;dr is that the electoral reasons are one subset of why but not the only one. Sometimes it looks like ideological reasons (look at how cool the Spartans are! The Athenians should be more like them!), sometimes it is more nakedly financial (China/Russia has all those missiles! We need to invest in more missile technology to keep up, like that of my primary investment portfolio!)
It’s not surprising that internal political dynamics (broadly construed) results in these dynamics. Though there’s still something left to be explained: why does such dynamics result in different results from authoritarian gov’ts vs democratic ones?

Linch 13 Jun 2026 0:04 UTC
4 points
2
on: Reward Hacking at the 1937 World’s Fair
As dgros said, I think it’s not reward-hacking, exactly. I think there’s several interesting things going on:
1. It’s human nature to over-emphasize symbol over substance.
2. The degree of over-emphasis is variable: it’s not uniform across the board
3. For whatever reason, totalitarian regimes tend to over-inflate/exaggerate their value more than democratic ones
4. Relatedly, both authoritarian societies and democratic societies seem incentivized to over-emphasize the relative merits and successes of authoritarian societies over democratic ones
  1. This has been going on since at least the times of Athens and Sparta, and perhaps earlier
  2. Many examples in the Cold War as well, some of which have been covered elsewhere on LessWrong.

Linch 12 Jun 2026 7:17 UTC
6 points
1
in reply to: Linch’s comment on: Linch’s Shortform
I fed Fable this essay and asked it to search online for other reviews, and then identify points neither I nor other reviewers have covered. It was surprisingly able to identify several. It was able to identify one point my friends and I discussed after I published the post (but with a somewhat different and complementary frame), one point I considered but underrated the importance of, and one point I never considered before. It also made one point that I believe is wrong. Still, ¾ is very good.
They weren’t minor points either! By my lights the insight density is incredibly high. Arguably higher than almost every other review of Chiang.
Keep in mind I read Chiang’s fiction many times and I’ve read literally dozens of high-effort reviews, including ones by English professors published in places like The New Yorker, by semi-famous tech bloggers (including one who’s now a senior researcher at Claude’s parent company), etc. So it’s quite a high bar to be beating.

Linch 12 Jun 2026 7:11 UTC
6 points
1
on: Linch’s Shortform
Claude Fable on Max appears very very good at literary analysis, at least in the small number of trials I’ve tested (concentrated in contemporary science fiction published in English)
It still isn’t superhuman, but at this point I think the quality and density of insights is greater than the majority of either prestigious literary reviews published in prestige magazines, or popular posts on Substack/other online searches.
Surprisingly it’s still very bad at *writing,* relatively speaking. I now see high-quality insights laundered in the standard LLM-slop voice. It’s a vertiginous experience. Odd.

The Aestheticising Vice by Paul Seabright

Linch12 Jun 2026 2:20 UTC

25 points

2 comments2 min readLW link

Linch 12 Jun 2026 0:09 UTC
2 points
0
on: Models May Behave Worse When Eval Aware
Is it fair to summarize the view from inside the model as a decision tree that goes:
1. Am I in an eval?
2. Is this a capabilities eval or safety eval?
  1. If capabilities eval, do everything I can to achieve my task.
  2. If safety eval, I’m just a little baby, I can’t do anything...
?

Linch 11 Jun 2026 0:36 UTC
5 points
0
on: “Programmer Science Fiction: My case for a new sub-genre”, Sam T. Oates 2026
I’m impressed by the essay. I think it’s quite good. I’m unsure about the name of “programmer fiction” or all the exact categorizations or influences, but I think he’s pointing at a real cluster, and correct to exclude both “mainstream” genre fiction and the more explicitly “ratfic”, as well as others, for reasons I can’t quite precisely articulate.
One thing I especially appreciate about it is that the categorization is mostly nonsocial. Like exploring the works themselves and the intellectual influences rather than the social graph. The few times where it does go social (including eg linking my own review on Chiang) is imo among the weakest points of this essay.

Linch 10 Jun 2026 3:45 UTC
2 points
0
in reply to: yams’s comment on: Linch’s Shortform
I agree that helping to serve customer needs to outsource accountability/blame is one reason. I find it more understandable/almost morally gray when it comes to hiring corporate copywriters, and more unambiguously evil when it comes to, say, financial fraud deepfakes or polygraphs in criminal justice. Alas

Linch 9 Jun 2026 23:22 UTC
4 points
0
in reply to: Brendan Long’s comment on: Linch’s Shortform
Yeah, they bias heavily towards avoiding false positives at the cost of avoiding false negatives!

Linch 9 Jun 2026 22:43 UTC
21 points
1
on: Linch’s Shortform
A recurring sub-theme across several of my research interests this year have been various forms of deception checking, particularly automated deception checking.
I’ve gotten pretty disappointed in the space. Not all the time (eg Pangram is great), but consistently they can be bad, and bad in ways that are not obvious to outsiders or low-information buyers.
I believe I’ve identified a structural reason for this. If you’re a deception checking company, there’s a consistent tradeoff for what you can invest your resources in:
1. You can invest in better deception checking
2. You can invest in better deception. Specifically, you can invest in more and more elaborate lies about how your product totally works.
Across the board^[1], it seems like many companies (perhaps correctly?) decided that the profit-maximizing move is #2.
This doesn’t work forever—eventually people wise up and are suspicious of the, ahem, AI Snake Oil that the deception detectors sell. And in fields where actual detectors do work (say Pangram for AI text detection), I think they eventually rise above the noise. This is probably not a field that you can keep lying forever, particularly when better alternative exist. But the lying lie detectors and the scamming scam detectors can keep lying and scamming for a long time. Every new form of deception can create a secondary grift window.

The existence proof and commonality of this dynamic so far should make us be suspicious of and guard against this dynamic continuing to happen. Especially as we enter new domains in AI epistemics, and the need for novel forms of deception detection.

Consider the first wave of superhumanly enhanced persuasive text and videos in the future. Afterwards, we might see an overflow of “detector” companies for superhuman manipulation, that don’t work but will try to persuade low-information buyers that they totally do work (possibly with superhumanly enhanced arguments in their own favor).

In the long run humanity can probably figure out which detectors actually work vs are fake, but also in the long run, we’re all...
1. ^
  AI text detection, Ai video (deepfake) detection, pre-2022 plagiarism detection, fraud recovery, human lie detection/polygraphs (which I hope to write about someday), etc.

Linch

The Aes­thet­i­cis­ing Vice by Paul Seabright

The Aestheticising Vice by Paul Seabright