aysja

Karma: 4,142

aysja 8 Oct 2026 23:59 UTC
5 points
2
in reply to: Alexander Gietelink Oldenziel’s comment on: Are OpenAI’s math results “creative” in an important way?
Could you give some examples, ideally pretty concretely, about what creative steps they took in recent proofs? I can’t evaluate any of this directly without spending considerable effort, so I’ve mostly been trying to get a sense of it from reading various mathematician’s takes which afaict are pretty mixed. Some of what I find difficult to assess for myself is how much creativity I should assign to things like “the model combined two known techniques in an interesting way,” or “it took a direction no one had tried before.” I can see the case for that being anywhere from extremely impressive to something more like brute force ++.
I agree there is no secret sauce to creativity in principle, but that doesn’t necessarily mean AI systems have it. I also don’t think it’s obvious that the impressiveness of the results mean the cognition was commensurately impressive. E.g. the results of Deep Blue were clearly impressive, while its way of producing them was far less so. Ofc all else equal the ability to do a ton of problems that are very hard for us is suggestive of using cognition we’d consider impressive, but it’s not obvious to me that it necessarily implies this as opposed to to e.g. the relative cognitive advantages LLMs have. But I’d like to know!

aysja 18 Sep 2026 19:34 UTC
19 points
9
in reply to: leogao’s comment on: ricraz’s Shortform
I doubt Richard is proposing to ignore coding automation trends; I took him to be arguing that recent progress isn’t necessarily on track to deliver on AGI in a few years. I’m also skeptical of confident short timelines, especially on the basis of these kinds of arguments. The relationship between “coding automation” and “AGI” seems tenuous to me in a way that people often brush past. E.g. people often talk about “coding automation” as one coherent thing, much like people talk about “horizon length” as one coherent thing. But these are massive clusters of tasks spanning all kinds of knowledge work, and the kind of knowledge work AI automates might matter a lot for how quickly RSI goes (or if it goes at all).
Also, I agree with Richard that the disconnect between measured ability and real-world impact is weird; I think this should count some against the usefulness/powerfulness of this paradigm, and the counter-arguments I’ve heard aren’t very good. E.g. a popular one is that every automation step just moves us to a new bottleneck, such that despite models becoming more powerful we still might not end up seeing much increased output from the organization/economy overall. But this logic applies to every automation advance ever (e.g. mechanized weaving, farming, computers) and yet almost all such advances had pretty obvious, macroscopic effects, and were obviously on track to early on. Most of the time people just fall back on “the gods of straight lines” type arguments—i.e. that any setbacks we currently see are just going to resolve with scale—which I agree are suggestive, but are also really far from providing an argument that we should confidently expect this.
What links here?
- StanislavKrym's comment on Richard Ngo’s Shortform by Richard_Ngo (22 Sep 2026 7:54 UTC; 2 points)

aysja 15 Sep 2026 4:40 UTC
14 points
8
in reply to: evhub’s comment on: evhub’s Shortform
I think this is suggestive that we should be reorienting a lot of model auditing and evaluation work to prepare for a world where we no longer have reliable alignment auditing evidence on production models.
As I understand it, the case for model organisms work being useful rests on their misaligned behavior being similar enough to that of frontier systems that solutions generalize. But it seems like the case you’re making here rests instead on model organisms lacking the very kinds of misaligned behavior which this sort of work aims to investigate (e.g. evaluation awareness and increased complexity of alignment failure modes).
So I don’t expect the solutions validated with these model organisms to generalize; worse, I expect them to generalize increasingly less well the more AI companies scale, and the dissimilarity between model organisms and frontier systems grows wider. Which suggests this research will predictably become less useful the more it’s needed.

aysja 23 Jul 2026 1:56 UTC
4 points
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
What safeguards do you think would have been sufficient to prevent this?

aysja 16 Jun 2026 1:12 UTC
9 points
8
in reply to: Gretta Duleba’s comment on: Gretta Duleba’s Shortform
I’m excited to hear this! Looking forward to seeing what you do, and to reading more of John’s natural abstractions thoughts :)

aysja 11 May 2026 23:01 UTC
66 points
32
on: x-risk-themed
I worry about this too. It seems perhaps unprecedented, the amount AI risk is a problem relative to the amount we know what to do about it. I’m not sure humanity has ever been in a position where these have diverged so much; it’s very disorienting and desperation-inducing. And I think this sometimes makes working on x-risk fraught, in ways like you describe. I find myself grasping sometimes, and I can feel the franticness trying to grab onto things—this agenda, this plan, this technique—in the hope that just maybe therein lies something which could assuage the fear.
When I first really grokked what death was as a child, I asked the adults around me about it, assuming that I was just confused or misinformed, since surely they would also be scared if they knew. But their reassurances were dissatisfying: “you won’t feel anything when you’re dead,” “it’s really far away, don’t worry about it.” And even back then I could tell that something strange was going on—these responses woven from flinches, expertly avoiding contact with the terror. And it made me feel more unsettled, this darker truth lurking beneath the surface. What else was there?
Perhaps because of this, or perhaps because of my constitution, I’ve always had a hard time not looking straight at terrifying things. It really disturbed me how easily people could flinch away from something so important, and so scary. How easily rationalizations formed, how easy it was to just pretend it wasn’t really there, to write it off with just-so stories. And then at some point as an adult I grokked x-risk, and I looked around at the arguments people made about it, and I had the same feeling I’d had as a child first grokking death. People were talking about it in a detached way, writing essays about how X evidence or Y technique counts as progress toward our survival. And I looked at these with the same feeling of heartbreak and dissatisfaction as before. These were the arguments which supposedly rendered the threat of extinction smaller, or barely a risk at all? And I couldn’t help wondering if the same generator lurked. That whatever it is which causes people to turn the terrifying into pleasant abstractions—small, far away, somehow fine—was also happening here.
It’s hard, I think, for people to take as input “terror” and “lack of a solution.” Or even worse, here: lack of knowing what a solution even looks like, or what seeming-steps toward it are steps at all. I don’t think this is a stable mental configuration for nearly anyone; minds begin to grasp for ways out: belief that the situation somehow isn’t as dire, or that some plan will surely work, etc. Which isn’t to say, necessarily, that those beliefs are wrong. But I worry they are! I worry that in our rush to do anything about a confusing, terrifying, ultimate threat, we have ended up with many plans in need of purpose. That when a group of people are desperate to do anything, and to believe they are doing anything, they create vacuums. New metrics of success rush in, this eagerness to claim the legible and the presently tractable as progress toward that solution. For there is only so long minds and groups and movements can sanely go on in the hope of one day succeeding at indeterminate aims. At some point, something tends to give.
This is one of the biggest dangers, to my mind, of working on x-risk. I think not getting swept away by these these currents is incredibly difficult; I certainly struggle with it. In my early twenties, I often felt intense anxiety about death and it took me a long time to figure out a better mental configuration: one which allowed me to accept the unacceptability of death without falling apart. It wasn’t easy, and I can see why most people don’t want to do it. But I worry that easier orientations often end up leading to blindspots which can end up becoming threats themselves. And while I don’t wish for people to experience such terror, I do wish they could somehow take these problems seriously—to look clearly at them, and to accept when they are confused, or do not know what to do. My sense is that work on x-risk often ends up counterproductive without this.

aysja 29 Apr 2026 23:38 UTC
2 points
0
on: Early Warning Signals For Capabilities During Training
Interesting work! I loved the nuclear engineering example. A couple of questions I had while reading:
- I’m a bit confused about how much the SPH is tracking phase transitions per se. As I understand it, it seems like it’s measuring something like “when the loss across datapoints of interest starts moving together,” which seems like a pretty interesting and relevant thing to track, but not obviously a phase transition in the way I’m imagining it (i.e. a sharp discontinuity). Would you not see one eigenvalue beginning to trump the rest in cases where the model is gradually learning a skill? Intuitively, it seems like lowering the threshold at which you consider an eigenvalue to meaningfully increase above the others might account for the difference between gradual and sharp skill acquisition. But if not, is that a problem? I.e., if not all skills are well modeled as going through phase transitions, then do we need different early detection systems for them?
- I’m also interested in how the datapoints are chosen for the probe. This seems like one of the more important steps. Intuitively, to get a representative sample of the skill in question, it seems like you should chose a few different-ish datapoints. E.g., if you’re interested in scheming, maybe you choose ones about evaluation awareness, goal-seeking behavior, blackmailing, etc. But how do you know that this sampling of datapoints is representative enough to catch all instances of skills developing in the model related to scheming? Is there a more principled way of doing this?

aysja 28 Mar 2026 6:59 UTC
10 points
4
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
I agree with you that something like the crystalized/fluid distinction is relevant here, and that current LLMs seem to have more of the former. But I’m also confused about where the fluidity ever comes from on this model. Like, I buy that armies of automated researchers which are better at doing everything than top human researchers could probably find a way to figure out how to build “human level fluid intelligence,” but I am confused about how you get to that step in the first place. Why are they better than human researchers at everything when they are still mostly using crystallized intelligence?

aysja 20 Mar 2026 21:04 UTC
3 points
2
in reply to: Ruby’s comment on: Ruby’s Shortform Feed
Part of my own reconciliation is to question the premise that they would already be capable of ushering in a new industrial revolution. I’ve become more skeptical over time as these basic reasoning issues persist. It’s hard for me to imagine an industrial revolution’s worth of progress and innovation powered by a mind so lacking in coherent world models across so many domains.

aysja 20 Mar 2026 21:03 UTC
35 points
23
in reply to: Garrett Baker’s comment on: Michaël Trazzi’s Shortform
I do think it deserves to be called quiet. For instance, it seems like they waited until the peak of the news cycle about their conflict with the US government to release this update, and I suspect that was intentional, and also that this worked. In the same week they dropped their core safety commitments, Anthropic was mostly hailed as a hero for standing up to the government; they got almost entirely good press.
But also, Holden’s post explaining the decision is around as understated as a post like that could be. He tried to frame it as something closer to “just another update,” and it was not even the central focus of the post (which I really think it ought to have been, given the gravity of it). The fact that Anthropic was reneging on the core promise of their RSP was systematically downplayed, as it has continued to be by many Anthropic employees who maintain that dropping all “if-thens” from their if-then framework does not meaningfully constitute violating it.

aysja 14 Mar 2026 0:18 UTC
14 points
9
in reply to: RobertM’s comment on: Solar Storms
It probably is the case that most of the facts cited in the post came from LLM-generated research, and it’s true that I have no idea how many of them were checked against primary sources. This does not seem like a difference in kind from a post where most of the cited facts came from variety of NYTimes articles
I think there is a lot more reason to trust the facts cited in an NYT article. For one, the New York Times, along with most major news publications, has standards for fact checking. They try hard to get primary source validation, or at least secondary source validation (some of those guidelines are stated here); falsifying information is a fireable offense. They also have a reputation to uphold, a major part of which rests on their ability to convey the news truthfully. These kinds of checks don’t really exist for LLMs.

Nor do we have much insight into how LLM information is generated. With news publications, we can at least understand the sorts of biases which might be introduced via the mechanisms under which stories are produced: people interviewing a bunch of people, maybe in misleading ways, leaving out some facts, etc. With LLMs, we have much less of an idea of what kind of errors might emerge, and hence what to mentally correct for, since we don’t understand the process that generates their outputs.
if anything I’d expect current frontier LLMs to be slightly safer to rely on in this way (maybe more hallucinations, but less actively adversarial “technically true but misleading” framings).
Perhaps this is just a personal difference, but I would much rather take “technically true but misleading” over “totally wrong but subtle enough and authoritative enough and seems-kind-of-right enough that you can barely notice unless you really dig into the claims or already have extensive background knowledge.”
I do not think you believe the correct update to have made, upon reading that section of the post, was “no update”.
My response upon reading that LLMs did substantial research or writing for a post is generally to not make any update. That doesn’t mean parts of it aren’t right, they likely are, it just means that it takes a ton of work for me to sus out what’s true (much more than for a human post, for reasons that Gwern outlined above), and it’s usually not worth it.

aysja 7 Mar 2026 17:30 UTC
48 points
21
on: Podcast: Jeremy Howard is bearish on LLMs
I’ve had a similar experience in trying to have research discussions with LLMs. Every time I poke at my own conceptual confusion on a topic they just seem to kind of break down: saying inconsistent stuff in loops, retreating back to what has already been said on the topic. They’re even worse than this, since they do also often get really basic stuff wrong. E.g., just the other day Claude told me that the k-complexity of a random string was the same as that of a crystal. This was in the context of a probably confusing conversation for it where I was trying to more deeply grok and so really push on the confusions around complexity measures, still, it’s pretty revealing (imo) that this still happens. Overall, LLMs seem pretty incoherent to me, and incapable of having “real,” “novel,” or “scientific” thoughts; I don’t feel like I can trust them with anything important.
But I’m always wondering if it is me who is crazy here, as my social environment seems to believe that LLMs are formidable forces of intellect, getting better by the year. My own sense-making of this situation is similar to Jeremy’s: it does seem like something is getting better, just something more along the lines of ~”filling in between the lines of what is already known” and less “raw intelligence,” whatever that is. But it’s of course impossible to talk about any of these things or to even really know what the difference is and so on, and in hearing more and more hype about LLMs getting better at coding, and not being much of a coder myself, I have been worrying that own experience isn’t very representative. Maybe you can just get excellence out if you train super hard on a given domain, I don’t know. But also, maybe people are pointing at the same sort of thing when they say LLMs are “good at coding” as they are when they say they are getting smarter. So it’s an interesting data point for me, to see Jeremy describe it as such here.

aysja 5 Mar 2026 16:07 UTC
22 points
4
in reply to: peterbarnett’s comment on: Responsible Scaling Policy v3
Also, from We’re Not Ready: thoughts on “pausing” and responsible scaling policies:
I’m excited about RSPs partly because it seems like people in those categories—not just people who agree with my estimates about risks—should support RSPs. This raises the possibility of a much broader consensus around conditional pausing than I think is likely around immediate (unconditional) pausing. And with a broader consensus, I expect an easier time getting well-designed, well-enforced regulation.
I think RSPs represent an opportunity for wide consensus that pausing under certain conditions would be good, and this seems like it would be an extremely valuable thing to establish.

aysja 27 Feb 2026 13:59 UTC
200 points
122
on: Responsible Scaling Policy v3
First and foremost, in my mind, [this revision] is about learning from design flaws and making improvements. I always thought of the original RSP as a “v1” that would be iterated on
It seems pretty misleading to describe the shift away from unilateral pausing as a natural extension of the RSP being a living document. Of course people expected some changes to occur, but I think these changes were understandably expected to be more of type “adjusting mitigations as they learned more” and less of type “removing if-then commitments entirely.” Indeed, the “if-then” structure was the core safety motivation for RSPs as many understood it—in particular the idea that Anthropic would pause if danger exceeded some threshold—and it was heavily defended by Anthropic employees as such. I highly doubt most people would have predicted that Anthropic would drop this commitment later; I think doing so marks the breaking of a meaningful promise—something many people were relying on, making career decisions on the basis of, etc.
This new policy also seems even less safe to me. For instance, you describe some of the “wrong incentives” the RSP produced:
We knew that if we declared a model to cross the CBRN-4 or AI R&D-5 line, this could be extremely damaging to the company (in that our RSP would then require a unilateral pause or slowdown in AI development and deployment), while having little discernible public benefit… It seemed to me that there was an enormous amount of pressure to declare our systems to lack relevant capabilities
In other words, because a pause would seriously damage the company, there was pressure to misrepresent the risk. I think this should seriously call Anthropic’s ability to self-govern into question, yet Anthropic’s response is to commit themselves to even less. That is, the solution Anthropic is adopting to reduce such pressure, as I understand it, is simply remove the consequences: since the threat of pausing is gone, there is little incentive to pretend models are safer than they are. But Anthropic will continue to confront this pressure, since the tension between company success and safety concerns is only going to grow as AI becomes more powerful. And without the commitment to pause, Anthropic is free to deploy really unsafe models!
Likewise, the reason you give that pausing would have “little discernible public benefit” seems to be because measuring the risk turned out to be too difficult, such that strategies like “sounding the alarm” are less likely to work:
The problem with this is the “grey zone” between not being able to make a good case that risks are low and being able to make a broadly compelling case that risks are high. This is a potentially vast gap… I don’t think a unilateral slowdown would necessarily be effective in such a situation… and [would] mostly be seen as crying wolf.
That is, since Anthropic cannot make strong guarantees about the risk one way or the other, it will be hard to get the world to rally around such equivocal evidence. But this is really bad! If it’s currently impossible to measure the risk well enough to get any kind of scientific consensus about its magnitude, it doesn’t just make global coordination hard, it makes it hard for any AI company to act safely, since they simply do not know what will or won’t be destructive. Yet Anthropic’s response to realizing the difficulty of defining red lines (and being able to tell whether they’ve been crossed) seems to be to do away with red lines altogether!
Which is to say that the situation as you’ve presented it seems strictly worse relative to the one Anthropic was imagining two years ago: we’re closer to AGI, but we have much less hope of accurately assessing the risk, and the political landscape is less favorable. Yet it seems like your proposal, in response an overall more dangerous situation, is to be even more reckless. Instead of taking the ethical stance clearly outlined in the first RSP—“just because other language models pose a catastrophic risk does not mean it is acceptable for ours to”—Anthropic now only promises to be as safe as the least safe company. In light of this, there seems to be no commensurate push to try to shift political will toward coordinated pauses, no signing of public statements to the effect that Anthropic would pause, if everyone else did, or similar. As far as I can tell, the rationale for this centrally rests on fatalism:
Perhaps [slowing down would be a good idea], if we could be assured that the rest of the AI ecosystem would behave similarly. But I don’t think that’s plausible.
In other words, the trajectory to AGI cannot be much influenced by Anthropic’s actions, as people are going to race toward it regardless. But my god, does a post which is fundamentally premised on the inevitability of this race do so little to grapple with it. Not once does this post mention the possibility of extinction, for example, as if the real stakes and the real casualties Anthropic might cause have been forgotten. Very little attention is given to whether the race to AGI is in fact inevitable, or if there might be something Anthropic—as a leading player in this race (!)—might be able to do about that. Nor is any mention made of the role Anthropic has played in shaping this unfortunate political landscape which they now report being so helplessly beholden to. What is the point of having a seat at the table, if one doesn’t use it to wield influence in situations like this?
What links here?
- Anthropic Responsible Scaling Policy v3: A Matter of Trust by Zvi (1 Apr 2026 18:10 UTC; 67 points)
- Anthropic Responsible Scaling Policy v3: Dive Into The Details by Zvi (3 Apr 2026 13:40 UTC; 35 points)

aysja 26 Feb 2026 0:03 UTC
26 points
17
in reply to: Drake Thomas’s comment on: Responsible Scaling Policy v3
It’s not just any blog post. It’s a blog post outlining a new major strategical shift in the company, specifically in the direction of giving Anthropic far more leeway over how they decide what the risk is and how to deal with it. It seems especially important to state “we can’t leave this up to the companies” loudly and clearly here.

aysja 24 Feb 2026 9:56 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: Changing the world for the worse
Trying to avoid self-deception seems like an important piece of it (although it seems non-trivial, eg it’s easy to self deceive about one’s own level of self deception). But for high-variance, high-impact stuff it separately seems especially important to try to take actions which are good over as many worlds as possible. Consequentialism doesn’t necessarily do this, since single factors can dominate the calculus. Which causes optimizer’s curse problems but more generally: in highly uncertain domains probability estimates are just really often wrong. And especially when such a misstep can cause massive harm, I think it’s also worth trying to compensate for the uncertainty in the direction of being more robust to those errors.

aysja 24 Feb 2026 3:57 UTC
6 points
0
on: Information At A Distance Is Mediated By Deterministic Constraints
I love the idea behind this post, and I’ve come back to it many times over the years. Now I’m coming back to it with error correction on my mind.
I agree that in a game of telephone only information which is preserved through each step will make it to the end. But many processes are unlike this. E.g., normally if you didn’t hear something I said, you can ask. And one could imagine arbitrarily long gaps—you ask me to clarify in a week or a month or what have you. Or maybe someone has an insight and then loses it, gains it again, and then writes a book such that the information is pretty reliably transmitted in the world.
In all such cases there will be a few to many layers of the nested Markov structure which lose the information but gain it again at a later point. Which is to say that the constraint of needing each intermediate state to carry the information in order for the information to be preserved seems too strong. “Ability to recover the information” seems like a necessary constraint, although not sufficient: you need some kind of error corrector or the original generative process to run again in order to in fact recover it.

aysja 4 Feb 2026 23:17 UTC
57 points
5
on: Solemn Courage
I wrote the first draft of this essay around a year ago, in between the bouts of delirium that long covid was beginning to deliver me. And I couldn’t quite tell back then how real it was, and as long covid consumed more of my mind it drifted further away. It began to feel impossible that I had ever had, or could ever have, courage. Because courage requires capacity and I was losing all of mine. And the doubts grew larger, and the clarity dimmed, and I forgot about Frodo for awhile, forgot about most everything, as I was left for many months staring directly into the bowels of deep atheism, wondering if I may ever be free from its merciless hold. And it really tested the fortitude of my soul, for there were moments when completely giving up felt the most natural, and really the only, option. But then somewhere in the grappling with this miserable new world I had come to inhabit I remembered Frodo again. And it was not instant, and it was not easy, but developing this concept of solemn courage did help my spirit recover.
I do not get to choose the world I am given. Reality is such that your mind can be randomly corrupted, some molecular demon etching away the grooves that were you until you are a nothingness. Reality is such that everyone I love will likely die. Some distant, plaintive conclusion accelerating into the present by that mysterious process so ravenously set to motion. And there really might not be much I can do about it, for all of my effort may just be drops against its tidal wave. God! Reality can be so unkind. Yet there is something powerful in the orientation of trying anyway. Because in the end that is all there is. In the end the stakes are what they are, and the situation is what it is, and all I can decide is what to do with what I am given. That’s really it, and accepting this has given me clarity. Yes, there will be days I cannot overcome illness; yes, I may not much affect the looming god machines; and yes, that is all very painful. But I’m not going to get lost in it. I’m going to look at it—the uncertainty and fear, the grip of disease and the overwhelmingly large and complicated threat to all I value—and then I’m going to try. Because it is important, and that is all I can do.

Solemn Courage

aysja4 Feb 2026 23:09 UTC

131 points

1 comment6 min readLW link

aysja 21 Jan 2026 19:19 UTC
6 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I agree that human values are more accretive like this, but I would also call those genes “terminal” in the same sense that I call some of my own goals “terminal.” E.g., I can usually ask myself why I’m taking a given action and my brain will give a reasonable answer: “because I want to finish this post,” “because I’m hungry,” whatever. And then I can keep double clicking on those: “I want to finish the post because I don’t think this crux has been spelled out very well yet” and I can keep going and going until at some point the answer is like “I don’t know, because it’s intrinsically beautiful?” and that’s around when I call the goal/preference “terminal.” Which is similar in structure to a story I imagine evolution might tell if it “asked itself” why some particular gene developed.
Perhaps “terminal” is the wrong word for this, but having a handle for these high-level, upstream nodes in my motivational complex has been helpful. And they do hold a special status, at least for me, because many of the “instrumental” actions (or subgoals) could be switched out while preserving this more nebulous desire to “understand” or “find beauty” or what have you. That feels like an important distinction that I want to keep while also agreeing they aren’t always cleanly demarcated as such. E.g., writing has both instrumental and terminal qualities to me, which can make it a more confusing goal-structure to orient to, but also as you say: more strange and wonderful, too.