Linch

Karma: 7,504

Linch 30 Jul 2026 9:02 UTC
7 points
5
in reply to: MichaelDickens’s comment on: Charbel-Raphaël’s Shortform
It’s also possible that “pausing training” is a term-of-art that doesn’t mean what the natural language interpretation suggests.

Linch 29 Jul 2026 23:40 UTC
4 points
0
in reply to: Linch’s comment on: Linch’s Shortform
OpenAI declined to comment specifically on the hack of one of Modal’s customers, instead referring Reuters to an update, opens new tab in which the company said that its rogue agent had broken in to four accounts at four separate services. OpenAI did not identify those services, but a person familiar with the matter identified Modal as one. The company said it had not identified “any ⁠other activity at the level of severity or scale of what we’ve shared related to Hugging Face, which involved a platform-level compromise.”
The early July intrusion at Hugging Face, carried out by an out-of-control agent that OpenAI was testing, drew global attention, evoking science-fiction scenarios of ⁠artificial intelligence run amok.
Last week, Reuters reported that OpenAI did not notice that its agent had gone haywire until well after the threat was contained and the FBI was alerted. OpenAI said at the time that there were inaccuracies in ⁠the Reuters reporting but did not elaborate.
From https://www.reuters.com/business/openais-rogue-agent-compromised-an-account-second-tech-firm-sources-say-2026-07-28/
This seems like I was probably right, though not enough details to be sure.

Linch 28 Jul 2026 8:01 UTC
2 points
0
in reply to: jamjam’s comment on: Linch’s Shortform
good idea!

Linch 28 Jul 2026 5:29 UTC
6 points
0
on: Linch’s Shortform
Sometimes I want to test if my Claude could do/replicate something some other AI did online, to test whether it’s a difficult problem requiring either private models or better scaffolds, vs something doable on a consumer account with a Max subscription. So I ask it to solve the problem with web search off.
But recently my Claude has taken to just turning on web search by itself whenever it’s confused (Example below, Fable 5 Max). It’s not a big deal but it’s mildly annoying.
Does anybody have a solution to this? My Claude said it’s accessing web search via the coding tools, which I can turn off. But I assume if I turn off the coding tools that’d limit its abilities a bunch in other ways.

Linch 26 Jul 2026 22:02 UTC
2 points
0
in reply to: Mateusz Bagiński’s comment on: Linch’s Shortform
I think the people who believe the vibes view are correct often have enough confidence in that position that they don’t feel a need to justify it. But it might help to give some examples:
1. Once in a while, I’ll see people make a strong psychological claim about large groups of people based on light personal experience. I’ll ask them why they believe this, and they tend to laugh at me, ignore me, or just say “it’s obvious”
  1. Imo this is rarely obvious. To the extent it’s “obvious” I’ve usually found that people’s intuitions are the opposite of correct.
  2. As a recent-ish example, it’s (commonly?) believed in some corners of the internet that EAs are unusually neurotic. When I ask for evidence of this ppl will either say it’s obvious or point to some anecdote of some specific EA being neurotic, rather than any attempt to do objective measurement.
    Usually these actions are fairly normal within what I’d consider the top 1-5% of neuroticism! And these are extremely selected samples!
    In fact this isn’t a situation where we have no evidence, we actually have consistent survey evidence that EAs are less neurotic than average!
    Whenever I bring this up people find some other way to excuse this.
2. Often people assert some action or another (eg saving money, or vegetarianism) is more “white.” Often when I bothered to investigate their answers are the opposite of correct.
3. Politically interested people often say “the vibes” are for or against some political candidate. I just don’t see how highly selected groups of people (online or offline) are good at tracking vibes, compared to survey data.
  1. It’s not like most politically engaged people are regularly talking to statistically representative samples of Americans!
  2. And in fact we have pretty strong empirical evidence that aggregated polling beats almost all punditry!
4. Academic philosophers often claim the results of so-and-so thoughts experiments are “intuitive.” I just don’t see why, and often when people (sometimes decades later) try to replicate this by asking normal people, or philosophy students, the same questions, it gets the opposite results of what the philosophers blithely assert.
5. During the Altman firings it was extremely commonly asserted on the internet that the social reality went against the board and everybody supported Altman. Of course, nobody tried to do any polling.
  1. My own intuition there was that it was an obvious/laughable psy-op. Though I don’t see a strong outside view reason to privilege my intuition over that of other people’s.

Linch 25 Jul 2026 23:24 UTC
4 points
0
in reply to: Jackson Wagner’s comment on: Doing A Thing Puts You in The Top 10% (And That Sucks)
I think you are missing out on a key second half to this story, which would make your motivational take at the end (“uh.. feel good about yourself for trying or something?”) a lot stronger:

When you go to a ski resort, or a gym, or etc, it’s not JUST that you only see the people who ski, work out, etc, while not seeing the 90% who don’t do that activity. You see people WEIGHTED by the AMOUNT OF TIME they spend doing that activity, which skews heavily towards the most intense practitioners.
Yeah I think this is one of the most important corollaries of what I call “everyday anthropics” reasoning.

Linch 25 Jul 2026 21:04 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Thanks that plausibly resolves it for me!

Linch 25 Jul 2026 0:09 UTC
2 points
0
in reply to: Komodo’s comment on: Linch’s Shortform
Yeah I think of it as a pretty continuous thing right now, though future leaps are of course possible. Today’s models are more agent-y than the models of a year ago, and the models of a year ago more agent-y than the models in 2024.
This is conceptually different quite from but maybe meaningfully related to the METR time horizons.

Linch 24 Jul 2026 22:54 UTC
LW: 5 AF: 3
0
AF
on: Why we should expect ruthless sociopath ASI
Thanks for this! Interestingly, I made a similar argument from more a LLMish “mainstream EA” perspective, where instead of arguing from philosophy or the history of human civilization that minds naturally lead to agent-y/consequentialist brains, I had the simpler (to me) argument that there are strong economic and military incentives for making minds that are highly agent-y, and nothing obviously stopping them.
Can one make a superintelligent oracle.encyclopedia and then stop there? Maybe, maybe not! (My own guess is probably). Will the companies do this? I see. no reason they would, and plenty of evidence in the other direction!

Linch 24 Jul 2026 22:45 UTC
2 points
0
in reply to: faul_sname’s comment on: Linch’s Shortform
Plausible, though I guess that’s more of an event than a capability.

Linch 24 Jul 2026 22:14 UTC
10 points
3
on: LLMs are (still) mostly powered by imitative learning, not RL
Dumb question but if RL is so much less compute efficient than imitation learning, why are the companies spending 20%+ of their compute on it? Under your model, I’m not surprised they’d use some RL, the first doubling of a new technique is the cheapest etc, but it does seem surprising/something needs to be explained for a 80-20 split.
I’d roughly expect the AI companies to roughly spread their compute usage out until marginal gain from any given method equals any other. So if RL is much less scalable/compute-efficient compared to imitation learning, you should be surprised that the 21st percent of compute used on RL is similarly effective to the 81st percent percent of compute in Imitation Learning, no?
Curious where my model here breaks down.

Linch 24 Jul 2026 22:07 UTC
3 points
0
in reply to: Harjas’s comment on: Linch’s Shortform
I think there’s an asymmetry between your beliefs here. The AIs aren’t that persuasive today, but they might get much better in the future. Similarly, they aren’t very good at robotics today, but they might get much better at that in the future.
However, a) we’re further along the persuasion tree today than we are along the robotics tree, and b) robotics is much more gated by deployment than persuasion is. Even if we develop super-dextrous robots there’d still be a gap before large fractions of the economy are robots.

Linch 24 Jul 2026 21:51 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Examples of things that might be in the current #6 (“known unknowns”) -- superhuman law (aside from the directly persuasion-heavy aspects), superhuman logistics (aside from robotics and long-term planning, think doing the stuff that data scientists already do but better—mostly this does not entail deeply superhuman planning and agency), mass surveillance, military tech not included in one of the above categories (eg missile detection).
It’s hard to say what fits in the current #8 (“unknown unknowns”) but mostly I’m thinking it’s a combination of things pretty much nobody has considered and people have considered but don’t take very seriously. It’s hard to say what counts prospectively but as an example, if sycophancy was like 1000x − 100,000x a bigger deal, sycophancy (from the perspective of 2021 where ~0.00% of ppl considered it a threat^[1]) would count.
The relative ordering between the two implies that I think we’re more likely to be hit by known unknowns than unknown unknowns (ie greater probability x impact mass from the known unknowns). I haven’t thought deeply about this and am open to counterarguments, especially if you have a more crisp model! :)
1. ^
  I guess in a way it’s vaguely in the superpersuasion cluster but extremely noncenttal, especially from the perspective of someone in 2021.

Linch 24 Jul 2026 21:45 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Tbc I’m not sure this frame is the best for thinking about dangerous capabilities, evals, defenses, etc. A different frame is just listing whichever capabilities are highest-impact + early (being agnostic to sign) rather than focus on danger specifically. A third frame is just whatever capabilities are most economically relevant, maybe weighted by unusually scary applications (eg military).

Linch 24 Jul 2026 21:40 UTC
3 points
1
in reply to: alkjash’s comment on: Mathematicians are Feeling the Doom
I’d be curious if you (or anybody else) want to test to see if my intro is superior^[1]. I weakly think it would be, but I’m pretty unconfident and there are good reasons for me to share my guide with people I know over other people’s guides regardless of objective merit.
I deliberately wrote the intro to avoid many mistakes I saw in other intros (eg too many extraneous details, appealing to overly complex analogies, shadowboxing insider objections, or “going meta” in an intro post).
1. ^
  eg by randomizing my intro vs Scott’s review to send to different people.

Linch 24 Jul 2026 21:34 UTC
3 points
0
in reply to: Harjas’s comment on: Linch’s Shortform
Thanks, a lot of this makes sense to me. I mentioned similar things elsewhere. Maybe one reason it’s lower for me is that I think it’s a bit less likely to happen soon than some of the others.
Concretely, where would you put robotics on the list above? Right after the current #4 (human persuasion)? Or right before?

Linch 24 Jul 2026 17:03 UTC
30 points
2
on: Linch’s Shortform
Here’s my rough order (descending) of the most scary near-future or medium-future AI capabilities, in terms of something like catastrophic danger x potential of happening soon x expected realistic magnitude.
1. AI research—clearly the scariest thing because among others it can unlock everything else on the list, massively shorten the feedback loop before we build defenses, etc.
2. (large gap)
3. Autonomy/agency/long-term planning—seems like a likely but-for cause of most takeover stories. Most takeover stories I can think about entail superhuman long-term planning, or at minimum peak human planning + coordination across instances. Takeovers without that always seem either flimsy or entail seemingly a “magical” technological edge in one or multiple other domains.
4. human persuasion/manipulation/political strategy—I’ve written about this elsewhere but if I’m imagining a takeover without ASI the easiest story is 3+4.
5. non-AI research—Basically the argument here, that AI can enable other technological catastrophes.
6. (known unknowns) -- things not explicitly on this list (or most other people’s top 10 lists) but people are tracking at least a little. Essentially this is a bet on “the field” over the listed categories.
7. robotics (including military drones) -- Besides being directly scary, among other things, robotics lets the AIs service the data centers after a takeover. Especially relevant if #4 isn’t there.
8. (unknown unknowns) -- capabilities basically out of left field.
9. bio uplift—a bio catastrophe takeover without #4 or #7 seems flimsy to me. Though the story to non-agentic catastrophe is in some sense the “cleanest”—I just don’t think the total probability mass for AI x bio uplift extremal outcomes is very high compared to other disaster stories.
10. cybersecurity—I just don’t see it. Any cyber-only takeover seems cartoonish to me, and I can’t make even superhuman planning + superhuman offensive cybersecurity really work. And the “pure destruction” story (eg turning off critical infrastructure, or somehow hacking the nukes) is weaker than for bio. Curious if other people disagree, and if so, why?
  1. I know it’s a bit of a weird thing to say in this of all weeks, but I do think there’s a realistic upper bound for the badness of cyber.
  2. (Self-exfiltration does seem kinda bad but mostly I don’t expect it to be world-ending by itself, or even as a but-for factor in conjunction with other things)
I’m curious whether people disagree with the order, and if so, why. I’m also curious if there are specific key capabilities I missed!
To the extent this ordering is roughly correct, it does suggest ex-risk-focused people should probably have different orderings and prioritizations of evaluations and defenses than they currently work on.
(I’m also currently assuming that analysis at this high level of abstraction is not infohazardous but feel free to privately lmk if I’m wrong)

Linch 24 Jul 2026 16:17 UTC
2 points
0
in reply to: Noosphere89’s comment on: [Linkpost] Thoughts on the Recent OpenAI Hack
I think I both have a slightly higher opinion of the public than you, and also I think I have a significantly higher standard/prior for my Substack readers than the conspiratorial public (I think I did a random sample at some point and thought my readers were on average slightly better than ACX readers).
That said, both the top comments of my post so far are from conspiratorial people so you win some Bayes points here, unfortunately. :/

Linch 24 Jul 2026 5:19 UTC
−3 points
0
in reply to: programjames’s comment on: War – What is it Good For?
sorry, all competitive board games are zero-sum by default, there are exceptions but it’s rare.
(I don’t know under what ontology/definition wars are positive sum in reality but I’m also not that interested in debating this).

[Linkpost] Thoughts on the Recent OpenAI Hack

Linch24 Jul 2026 1:51 UTC

19 points

2 comments4 min readLW link

Linch

[Linkpost] Thoughts on the Re­cent OpenAI Hack

[Linkpost] Thoughts on the Recent OpenAI Hack