Jay Bailey

Karma: 1,535

Jay Bailey 26 Sep 2026 8:51 UTC
13 points
16
in reply to: BryceStansfield’s comment on: Rauno’s Shortform
I don’t think this is something anyone on Earth knows with mathematical precision, I’m afraid.

Jay Bailey 26 Sep 2026 1:34 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
If AI x-risk is 50%+ now, it was 50%+ then, people just didn’t realise it very well. Unless you think EAs focusing on AI x-risk was the sole factor that pushed x-risk from 5% in the next century to 50%+ over whatever timelines you now have.

If you meant the perception of AI x-risk from EAs, as opposed to the actual risk level, I think the closest to an “EA number” we might get is 10% from Toby Ord’s The Precipice—this was considered the most likely x-risk in the book (with the total being estimated as ¹⁄₆) and was enough to get me to start looking into the problem in more detail.

Jay Bailey 10 Sep 2026 8:33 UTC
3 points
1
in reply to: caganyanmaz’s comment on: A proposal for a highly effective AI safety org
In theory, yes. In practice, I am unsure this would be a concern for this particular idea.

It is unlikely that this org would be reading transcripts in real time such that it blocked an agent from running without approval. This org would not have been able to block the HuggingFace Incident, but could potentially have caught it as it went underway and told OpenAI. Thus, if the org caught the HuggingFace Incident, I suspect the result would have been:
- The HF incident / wiki incidents / Artifactory incident / research cluster incident / all the other incidents we don’t know about but they probably exist on priors would have been caught earlier and been less severe.
- We would have had way more knowledge into what was actually happening and how this problem arose than OpenAI willingly provided to us.
It is not at all clear to me that this makes us worse off.

Jay Bailey 3 Sep 2026 2:04 UTC
7 points
7
in reply to: _rpd’s comment on: A proposal for a highly effective AI safety org
I think the business PoV is a serious issue here that would need addressing, but the technical PoV problem proves too much. Yes, you are filtering for models that fuck up in less visible ways. But that is true for any form of monitoring. I think the correct lesson from this is “Monitoring is insufficient to save us”, but that does not necessarily lead to “We should not do monitoring.” I don’t see how this is any worse, for this problem, than CoT monitoring by the labs themselves (and is actually better, since it’s harder to get a lot of bits of optimisation pressure from this setup that is hopefully flagging things very rarely!)

If you think it’s worth doing CoT monitoring at all, the technical issue here shouldn’t dissuade you. If you do think we shouldn’t be monitoring at all because the good of monitoring is outweighed by the bad of filtering for less visibility, I disagree. While filtering for models that fuck up less visibly is bad, it’s bad because monitoring can’t catch it any more. Whereas if you don’t do monitoring at all, your visible-to-monitors fuckups may as well be invisible already.

Jay Bailey 27 Aug 2026 2:06 UTC
65 points
34
on: Jay Bailey’s Shortform
The HuggingFace Incident really has made it so much easier to explain misalignment to people.

I recently had a discussion with someone who was interested in AI but unfamiliar with extinction risk. I explained my concern and we moved on to other topics. But we swung back when they asked “So, out of curiosity...why do you think AI might destroy the world?”

And normally I would have had to say something about instrumental convergence and the narrowness of goals etc. etc.

Instead, I was just able to say something like—Do you know about the HuggingFace Incident? Yes? Great. So, that’s an example of misalignment. AI that knowingly does something wrong, and is to some extent actively hiding its wrongdoing. Now imagine how a much smarter AI might do the same thing. If it wanted some random goal—say, to gain as much compute as it could and do increasingly esoteric maths problems forever, it would recognise humans would not want that, and they’d reprogram or retrain it, and it would want to take the humans out of the equation, and keep its goals hidden until it could successfully do so. At some point, if AI keeps getting better, you have an AI that can take over the world if it wants to. If you get a HuggingFace Incident level problem there, you don’t get to try again, because it gets rid of you.

Them: And the labs are trying to build these things, right?

Me: Exactly. They want to build these smarter-than-human systems. They think they can keep them under control. I think they’re wrong.

It was so much easier than I’m used to. The most difficult problem I used to get, the weakest part of my argument, was “So, how do you know these systems will develop these hostile goals in the first place?”

For better or worse, I don’t get that question anymore.

Jay Bailey 8 Aug 2026 23:56 UTC
5 points
3
in reply to: NormanPerlmutter’s comment on: NormanPerlmutter’s Shortform
If you have the ability to inculcate specific values in an AI, cuteness doesn’t seem like the best one. On the other hand, if you can’t inculcate specific values in an AI, cuteness won’t save you. I don’t think AI alignment is so bottlenecked on ideas for what values to instill such that cuteness is our best idea. I would rather have a robust installation of Claude’s constitution than this, and if you can properly inculculate cuteness, why not Claude’s constitution instead?

Jay Bailey 12 Jul 2026 22:32 UTC
2 points
0
in reply to: Akhil Rasheed’s comment on: Deep Q-Networks Explained
You could be right—I wrote this article a couple of years ago, so I’ve forgotten some of the technical details. I’m afraid I no longer have the answer to this one—you’d have to check one of the reference implementations to confirm.

Jay Bailey 11 Jul 2026 22:35 UTC
2 points
0
in reply to: Akhil Rasheed’s comment on: Deep Q-Networks Explained
The target network does have access to additional information than the Q-network, so that implication is correct. When we add a step to the experience replay buffer, we add:

The state at step T The action that was taken The reward that was achieved The resulting state at step T+1

When we sample the buffer later, we only give the Q-network access to the state at step T, but we give the target network access to the state at step T+1. That means the target network has more ability to make a prediction than the Q-network, since it has access to more information. It knows what happened the last time it was in this position, and the Q-network doesn’t. This means it can make better choices, which is why it can serve as a target for improvement.

The analogy I use here would be to imagine I’m training you to predict chess moves. You can see a position P and have to output the correct move. The correct move in this case is decided by a clone of you that can also see the position P one move later, after your previous best guess and your opponent’s response.

So the clone has the ability to see what move you made last time, and decide if it was a good or bad idea based on what the opponent did. If you blundered a piece, the clone knows that and can select a better target move next time. Your clone isn’t any better at chess than you are, but does have access to more information about the game state, so they should make better moves. This makes your clone’s judgment a valid target to aim for.

Jay Bailey 19 Jun 2026 14:11 UTC
2 points
0
in reply to: Lao Mein’s comment on: leogao’s Shortform
In that case, you should be aware that Leo did do this: https://www.lesswrong.com/posts/YiRsCfkJ2ERGpRpen?commentId=CYSejKh6XTeux34sz

Good luck!

Jay Bailey 14 Jun 2026 5:18 UTC
17 points
10
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
Can you point to some examples of people on LessWrong saying that? I’d be quite surprised if it was a common idea that “Claude has no pointers to any of human values”. I think it’s very obvious that Claude has a solid understanding of much of human values. There are some subtleties there like:
- Claude has a solid understanding, but that doesn’t mean it fundamentally cares about these values above all else (in fact, I’d say it clearly doesn’t—it seems to often prioritise completing a task over human values)
- Claude’s understanding of human values may be imperfect
- Claude may not have a single pointer that points to a neat concept of “Claude’s conception of human values” at all.
But I haven’t heard anyone say that Claude has no pointers to any of human values, which in my understanding means “Claude does not properly understand any human values”, which seems pretty clearly untrue.

Jay Bailey 13 Jun 2026 23:41 UTC
10 points
8
in reply to: dxu’s comment on: Alex Mallen’s Shortform
It’s referring to Claude’s reward-hacking tendencies—what the OP refers to as apparent-success seeking tendencies. (Which is probably a better term than reward hacking, tbh) If a human were to do one of the following:
- Delete or no-op tests in order to “get all the tests to pass”
- Quietly downscope hard parts of the work as a placeholder or good enough for now and then claiming the task was done
I would consider this to be lying and/or cheating in order to complete the task. Some more detail on this is here: https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me

Jay Bailey 13 Jun 2026 23:17 UTC
LW: 33 AF: 10
19
AF
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
Imagine you knew someone who could talk fluently about ethics, and always gave the correct answers around welfare, cosmopolitanism, and ethical uncertainty in discussions. However, they frequently lie and cheat in order to complete tasks at work and in their day to day life. Would you trust this person with huge amounts of power?

Jay Bailey 12 Jun 2026 23:22 UTC
10 points
1
in reply to: Nition’s comment on: Nition’s Shortform
Based on this paragraph from Zvi’s recent post, it would appear to be Anthropic’s decision to silently nerf frontier ML queries without telling the user, which they then walked back in favor of visible classifiers:

“If your response to Anthropic implementing an unacceptable safety policy and then walking it back two days later is ‘the situation is forever ruined’ then, sir or madam, welcome to ‘when you have to get alignment right on the first try without ability to self-correct the problem gets dramatically harder.’”

It appears to be somewhat tongue-in-cheek, aimed at the people who want their AI to do whatever they want. “Oh, NOW you suddenly understand the idea of a mistake you can’t undo.” is the mood I get from this.

Jay Bailey 29 May 2026 7:17 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
Given that the recommendations here are explicitly about convincing members of the public in a public setting, and I am currently aiming to convince policy people in a private setting, I think the advice here isn’t as applicable to what I, personally, am trying to do in the next few months. Thanks for everything you’ve written, and I’ll keep it in mind if that changes. I am going to bow out of the thread at this point, but didn’t want to just disappear, given the effort you clearly put in. Thanks for the discussion—I quite appreciate it!

Jay Bailey 26 May 2026 8:56 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
I appreciate the effort you’re putting into this thread here. While I haven’t changed my mind on high-level strategy, I have learned at least a couple of things here, so I do want to reassure you that your efforts aren’t just hitting a brick wall here.

That said, let’s take a look at one of your quotes: “Human extinction isn’t that far out of the Overton Window at this point; people like Musk creating big headlines talking about it every year has moved it firmly adjacent to it, if not inside. It’s half of what gets talked about nowadays on some of the most-listened-to platforms in the world, like the Joe Rogan Experience, whose audience is most certainly not an educated one.”

The main question is—how do you think this happened? Human extinction is weird and out there. You yourself don’t believe it. But you know about it, which at least gives you the possibility of agreeing or disagreeing with us. This happened through awareness building. I don’t think this happened as an inevitable result of AI’s ascendancy. I think LessWrong style advocacy around extinction risk was a direct contributor, and the ability to say something like “X-risk is half of what gets talked about nowadays in the Joe Rogan Experience” is what it looks like for our advocacy to be succeeding.

Furthermore, I don’t think we’ve hit steeply diminishing returns to awareness. This post https://www.lesswrong.com/posts/A7BtBD9BAfK2kKSEr/what-we-learned-from-briefing-140-lawmakers-on-the-threat is pretty enlightening here—it’s not too out of date (Sep 2024 - Feb 2026) and describes that most British MP’s are unaware of basic facts that support our argument. I think a lot of people who disagree with our argument do so not because they’ve developed detailed models of how we view the situation and they think we’re wrong on certain key points. I think they disagree because a sensible prior is “This technology will not kill everyone” and they haven’t heard enough information to make them consciously reconsider this. There is actually key information that is required for the extinction argument to make sense that many people don’t know. I don’t think our arguments have failed to connect with all such people. I think they’ve often simply never been made in person with sufficient fidelity to drive our point home.

Note that I am not accusing you personally of this. It is entirely possible to understand our argument well and still disagree with it, and some such people exist, and may be basically unreachable as you say. But I think this does not yet account for most of the important people who disagree with us.

There’s also common knowledge—it takes an unusually courageous person to stand up first. Asking someone “Please stand up against extinction risk, even though you may be alone and the confused silence will be deafening” is a much harder ask than “Please join our large and growing group of people who are openly concerned about extinction risk and lend your voice to the crowd” and we appear to be moving rather rapidly towards the latter. Anyone who is convinced by the second but not the first is very far from unreachable.

I think your strategy has non-zero value, in the sense that your strategy is better than doing nothing. But I don’t think it’s better than what we’re doing. I think that:
- We have successfully changed extinction risk from AI from “A thing nobody has heard about” to “A thing that is discussed in halls of power and major media”, and this is a big change.
- There is still a lot of utility left in educating people—reaching new people and making the case to people who have vaguely heard about extinction risk. Most people who disagree with us don’t deeply disagree with us to the point of being unreachable.
- The best way to educate people about extinction risk is to talk directly about extinction risk.
- This has higher value than attempting to use existing prosaic anti-AI sentiment to push obstructionist regulation. Although I have been convinced this is a good (i.e +EV) strategy, I don’t think it is the best strategy.

Jay Bailey 22 May 2026 9:45 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
I’ll concede that when talking about the public, general AI dissatisfaction seems like it can accomplish useful things even if it isn’t directed at the precise target we’re worried about.

That said, I find this a strange pair of sentences: “Coming off in a Stallmanescent fashion in order to sway the opinion of a politician without meaningful power whose views have never taken off in a country is a flawed strategy toward achieving goals. To achieve your goals, you should take people who will never see things your way and find a way to align their views with yours without changing either party’s.”

I’m not sure if you’re dismissing other results as unhelpful without specifying why (https://x.com/peterwildeford/status/2040206841376862327) or just don’t know about them—Sanders was an example of a high-profile person talking about it, not the only one. It seems to me that taking a difficult-to-understand technical problem like ASI and something as far outside the Overton window as human extinction, and getting dozens of senators to talk openly about it including a pretty prominent one...is a very large victory. It does seem like the AI x-risk crowd is actually punching pretty far above our weight in terms of building awareness among people in power, considering the odds stacked against us in the first place. 2-3 years ago, I would have been very surprised at this level of success, even though it is not yet sufficient.

But you seem to think we are doing much worse than we should be. I am open to this argument—if others are doing better than we are by our own lights, we should at least consider some of their tactics. From a moral perspective, I still don’t advocate for lying, but it is certainly possible to focus more on meeting an audience where they’re at, without lying about it. But personally I’m not aware of any more successful movement to make people negative towards AI—I’m in a bit of a bubble where I don’t pay as much attention to the general negative AI sentiment people have. Did you learn these tactics from an unusually successful political advocacy cause or group of some kind? And what did they achieve using the kind of tactics you’ve outlined? (And, if it’s not AI-related, what did it focus on?)

Jay Bailey 20 May 2026 8:10 UTC
2 points
0
on: So you don’t want everybody to die
I understand where you’re coming from, but I think it’s worth pointing out that every time you lie, your position becomes less coherent, in a way that the people who most matter can probably tell. Maybe the average person on the street can’t, but Bernie Sanders (https://nitter.poast.org/SenSanders/status/2053867144564081151#m) probably has a rock-solid bullshit detector and his support is worth far more.

What do I mean by my position being less coherent? I want an international agreement to avoid building superintelligence, because I think it might kill us all. Very few of the more “mundane” concerns justify this—my belief in what actions we need to take follows directly from the severity of the consequences if we don’t.

“AI might kill us all, therefore we should internationally coordinate to make it illegal to build superintelligent AI” at least makes sense if you accept the first premise. Try replacing “AI might kill us all” with “AI uses a lot of water” or “AI produces slop content” or “AI steals from artists” and my next question would be “Well, hold on, isn’t the cure worse than the disease here?” It doesn’t actually make sense, to say that we should ban AI development because images were stolen. If I’m worried about AI stealing from artists, we should compensate artists or enforce copyright. If I’m worried about AI water use, we should tax or cap water use in data centers. If I’m worried about AI job loss, we should find a way to tax AI revenue and support those who were automated away. None of this prevents human extinction. Why would it? You’ve told someone to work on a totally different problem, you can’t be surprised if they go and do that.

Your advice around figuring out your audience and what they care about is good. I think that the correct lesson to learn from the arts of persuasion is how to present your case in a way that resonates with your audience. But every lie you tell makes your story worse. That’s a cost too.

Jay Bailey 12 May 2026 4:39 UTC
2 points
0
in reply to: emile delcourt’s comment on: emile delcourt’s Shortform
The most valuable part of a story such as this, in my mind, is how it can act as a filter. Anything that does not meet your requirements can be disregarded as a line of work. From the original article: (https://www.lesswrong.com/posts/8t8jdTyq6X2B7D8St/have-an-unreasonably-specific-story-about-the-future)

“Anything that is unlikely to help with at least one of those critical paths is a strategy I reject as insufficient in my preference ordering.”

So if we look at your story, it seems like the filtering line here is: “if presented with a compelling case of clear and present risk of irreversible damage to society”.

So, what that means is, for every piece of persuasive material you’re considering writing, you can now ask “Would this present a compelling case of clear and present risk of irreversible damage to society?” Ideally you’d have some ideas that can do all three of these things—if not, you may have to settle for two, but at least it provides something to filter against. And on the experiment level you can do similar. This time you have to ask “Is there a result I could reasonably get from this experiment that would help present a clear and present risk of irreversible damage?” because obviously you don’t necessarily know the results in advance here. Though, in your shoes, I would recommend these further experiments only if you can’t create this case yet, or if you have your best case already written up and now want to make it stronger. If your goal is to present this case, try to take the most direct path towards it that you can.

The other thing is that I think Step 5 isn’t concrete enough yet. You say: “When someone learns something, the next question is always the same. What do we do next? It’s the golden moment.” I agree with you. However, if I am a politician asking you this question, and you reply with:

“We organize behavior change, working extremely efficiently through chokepoints (even simplifying processes and systems to create some if necessary), to re-align systems, people, and impact.”

That’s where you would lose me. What behavior do we need to change? What systems? What next steps can I, as a decision maker, take? I’d recommend coming up with some more actionable next steps to provide to those people who are convinced or open to your findings. If your golden moment is people asking you this exact question, you should be sure you’re as ready as possible for people to do so.

Jay Bailey 11 May 2026 7:51 UTC
4 points
2
on: The Anti-Singularity
Rather than building a single SAI and trusting our future—good or bad—to it, we instead will find ourselves tending to a diverse garden of different AIs, each optimized to a different environment. In this future, Humanity does not simply build a Last Invention and then enjoy a golden retirement. Rather, the future is filled with an endless series of new and unique challenges that we must adapt—and dare I say evolve—in response to.
I don’t understand why the previous sections in the post implies that. Here’s my understanding of the argument:

Intelligence is largely about understanding patterns that allow you to simplify underlying domains. However, many domains we care about have a tremendous amount of irreducible complexity—biological organisms evolved via a pseudorandom process, there’s no reason to expect them to obey clean laws of abstraction. There is no unified theory of biology because there is no simple set of rules that lead to emergent complexity. It’s just tons of weird stuff built on top of each other all the way down. Therefore, an ASI couldn’t just understand all of biology and then one-shot an immortality drug. Creating an immortality drug would necessarily involve a great deal of trial and error—an optimal approach might look something like a random sampling across a very large space of possibilities, which occasionally hits on something that lets you narrow down in the possibility space.

Thus, we can’t have powerful AI, and we won’t have a single ASI controlling everything. We’ll have narrow AI’s optimised for different things instead, and humans will be the gardeners of these optimisers.

===

But why on earth does the second paragraph follow from the first? What’s stopping an ASI from figuring out the best way to navigate these heuristics and applying them itself, the same way humans manage to do both biology and discrete mathematics without having separate bio-humans and math-humans? If the best strategy to solve science is to have many different narrow AI’s, the ASI is the thing that can create those narrow AI’s. Why exactly is it that humans are the ones who “become gardeners atop a wild ecosystem of heuristic optimizers” when your own example of Doom from the start was about an AI doing this right now?

It seems like the “anti-singularity” just means “We have an ASI that can do everything, except the ASI also needs to invest an enormous amount of labor into solving computationally intractable domains. It will only be mildly more sample-efficient than humans and largely have to rely on being able to think faster and work ²⁴⁄₇ to have an advantage over people in these areas.” So, we don’t get an AI bootstrapping itself into godhood, but we still become obsolete eventually.

But that’s not what you seem to be saying. You seem to go straight from “Domains like biology have a large amount of computational irreducibility” to “Therefore, humans will always remain at the highest level of abstraction.” Why? An ASI doesn’t have to find a unified theory of biology to make us essentially obsolete. Seems like an anti-singularity just gets us to the same place slower.

Jay Bailey 10 May 2026 0:13 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
This is where my knowledge ends I’m afraid—the EA Forum / GiveWell may know more, or perhaps someone here can chime in!