Jay Bailey

Karma: 1,430

Jay Bailey 19 Jun 2026 14:11 UTC
2 points
0
in reply to: Lao Mein’s comment on: leogao’s Shortform
In that case, you should be aware that Leo did do this: https://www.lesswrong.com/posts/YiRsCfkJ2ERGpRpen?commentId=CYSejKh6XTeux34sz

Good luck!

Jay Bailey 14 Jun 2026 5:18 UTC
17 points
10
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
Can you point to some examples of people on LessWrong saying that? I’d be quite surprised if it was a common idea that “Claude has no pointers to any of human values”. I think it’s very obvious that Claude has a solid understanding of much of human values. There are some subtleties there like:
- Claude has a solid understanding, but that doesn’t mean it fundamentally cares about these values above all else (in fact, I’d say it clearly doesn’t—it seems to often prioritise completing a task over human values)
- Claude’s understanding of human values may be imperfect
- Claude may not have a single pointer that points to a neat concept of “Claude’s conception of human values” at all.
But I haven’t heard anyone say that Claude has no pointers to any of human values, which in my understanding means “Claude does not properly understand any human values”, which seems pretty clearly untrue.

Jay Bailey 13 Jun 2026 23:41 UTC
10 points
8
in reply to: dxu’s comment on: Alex Mallen’s Shortform
It’s referring to Claude’s reward-hacking tendencies—what the OP refers to as apparent-success seeking tendencies. (Which is probably a better term than reward hacking, tbh) If a human were to do one of the following:
- Delete or no-op tests in order to “get all the tests to pass”
- Quietly downscope hard parts of the work as a placeholder or good enough for now and then claiming the task was done
I would consider this to be lying and/or cheating in order to complete the task. Some more detail on this is here: https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me

Jay Bailey 13 Jun 2026 23:17 UTC
LW: 33 AF: 10
19
AF
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
Imagine you knew someone who could talk fluently about ethics, and always gave the correct answers around welfare, cosmopolitanism, and ethical uncertainty in discussions. However, they frequently lie and cheat in order to complete tasks at work and in their day to day life. Would you trust this person with huge amounts of power?

Jay Bailey 12 Jun 2026 23:22 UTC
10 points
1
in reply to: Nition’s comment on: Nition’s Shortform
Based on this paragraph from Zvi’s recent post, it would appear to be Anthropic’s decision to silently nerf frontier ML queries without telling the user, which they then walked back in favor of visible classifiers:

“If your response to Anthropic implementing an unacceptable safety policy and then walking it back two days later is ‘the situation is forever ruined’ then, sir or madam, welcome to ‘when you have to get alignment right on the first try without ability to self-correct the problem gets dramatically harder.’”

It appears to be somewhat tongue-in-cheek, aimed at the people who want their AI to do whatever they want. “Oh, NOW you suddenly understand the idea of a mistake you can’t undo.” is the mood I get from this.

Jay Bailey 29 May 2026 7:17 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
Given that the recommendations here are explicitly about convincing members of the public in a public setting, and I am currently aiming to convince policy people in a private setting, I think the advice here isn’t as applicable to what I, personally, am trying to do in the next few months. Thanks for everything you’ve written, and I’ll keep it in mind if that changes. I am going to bow out of the thread at this point, but didn’t want to just disappear, given the effort you clearly put in. Thanks for the discussion—I quite appreciate it!

Jay Bailey 26 May 2026 8:56 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
I appreciate the effort you’re putting into this thread here. While I haven’t changed my mind on high-level strategy, I have learned at least a couple of things here, so I do want to reassure you that your efforts aren’t just hitting a brick wall here.

That said, let’s take a look at one of your quotes: “Human extinction isn’t that far out of the Overton Window at this point; people like Musk creating big headlines talking about it every year has moved it firmly adjacent to it, if not inside. It’s half of what gets talked about nowadays on some of the most-listened-to platforms in the world, like the Joe Rogan Experience, whose audience is most certainly not an educated one.”

The main question is—how do you think this happened? Human extinction is weird and out there. You yourself don’t believe it. But you know about it, which at least gives you the possibility of agreeing or disagreeing with us. This happened through awareness building. I don’t think this happened as an inevitable result of AI’s ascendancy. I think LessWrong style advocacy around extinction risk was a direct contributor, and the ability to say something like “X-risk is half of what gets talked about nowadays in the Joe Rogan Experience” is what it looks like for our advocacy to be succeeding.

Furthermore, I don’t think we’ve hit steeply diminishing returns to awareness. This post https://www.lesswrong.com/posts/A7BtBD9BAfK2kKSEr/what-we-learned-from-briefing-140-lawmakers-on-the-threat is pretty enlightening here—it’s not too out of date (Sep 2024 - Feb 2026) and describes that most British MP’s are unaware of basic facts that support our argument. I think a lot of people who disagree with our argument do so not because they’ve developed detailed models of how we view the situation and they think we’re wrong on certain key points. I think they disagree because a sensible prior is “This technology will not kill everyone” and they haven’t heard enough information to make them consciously reconsider this. There is actually key information that is required for the extinction argument to make sense that many people don’t know. I don’t think our arguments have failed to connect with all such people. I think they’ve often simply never been made in person with sufficient fidelity to drive our point home.

Note that I am not accusing you personally of this. It is entirely possible to understand our argument well and still disagree with it, and some such people exist, and may be basically unreachable as you say. But I think this does not yet account for most of the important people who disagree with us.

There’s also common knowledge—it takes an unusually courageous person to stand up first. Asking someone “Please stand up against extinction risk, even though you may be alone and the confused silence will be deafening” is a much harder ask than “Please join our large and growing group of people who are openly concerned about extinction risk and lend your voice to the crowd” and we appear to be moving rather rapidly towards the latter. Anyone who is convinced by the second but not the first is very far from unreachable.

I think your strategy has non-zero value, in the sense that your strategy is better than doing nothing. But I don’t think it’s better than what we’re doing. I think that:
- We have successfully changed extinction risk from AI from “A thing nobody has heard about” to “A thing that is discussed in halls of power and major media”, and this is a big change.
- There is still a lot of utility left in educating people—reaching new people and making the case to people who have vaguely heard about extinction risk. Most people who disagree with us don’t deeply disagree with us to the point of being unreachable.
- The best way to educate people about extinction risk is to talk directly about extinction risk.
- This has higher value than attempting to use existing prosaic anti-AI sentiment to push obstructionist regulation. Although I have been convinced this is a good (i.e +EV) strategy, I don’t think it is the best strategy.

Jay Bailey 22 May 2026 9:45 UTC
2 points
0
in reply to: Rattengift’s comment on: So you don’t want everybody to die
I’ll concede that when talking about the public, general AI dissatisfaction seems like it can accomplish useful things even if it isn’t directed at the precise target we’re worried about.

That said, I find this a strange pair of sentences: “Coming off in a Stallmanescent fashion in order to sway the opinion of a politician without meaningful power whose views have never taken off in a country is a flawed strategy toward achieving goals. To achieve your goals, you should take people who will never see things your way and find a way to align their views with yours without changing either party’s.”

I’m not sure if you’re dismissing other results as unhelpful without specifying why (https://x.com/peterwildeford/status/2040206841376862327) or just don’t know about them—Sanders was an example of a high-profile person talking about it, not the only one. It seems to me that taking a difficult-to-understand technical problem like ASI and something as far outside the Overton window as human extinction, and getting dozens of senators to talk openly about it including a pretty prominent one...is a very large victory. It does seem like the AI x-risk crowd is actually punching pretty far above our weight in terms of building awareness among people in power, considering the odds stacked against us in the first place. 2-3 years ago, I would have been very surprised at this level of success, even though it is not yet sufficient.

But you seem to think we are doing much worse than we should be. I am open to this argument—if others are doing better than we are by our own lights, we should at least consider some of their tactics. From a moral perspective, I still don’t advocate for lying, but it is certainly possible to focus more on meeting an audience where they’re at, without lying about it. But personally I’m not aware of any more successful movement to make people negative towards AI—I’m in a bit of a bubble where I don’t pay as much attention to the general negative AI sentiment people have. Did you learn these tactics from an unusually successful political advocacy cause or group of some kind? And what did they achieve using the kind of tactics you’ve outlined? (And, if it’s not AI-related, what did it focus on?)

Jay Bailey 20 May 2026 8:10 UTC
2 points
0
on: So you don’t want everybody to die
I understand where you’re coming from, but I think it’s worth pointing out that every time you lie, your position becomes less coherent, in a way that the people who most matter can probably tell. Maybe the average person on the street can’t, but Bernie Sanders (https://nitter.poast.org/SenSanders/status/2053867144564081151#m) probably has a rock-solid bullshit detector and his support is worth far more.

What do I mean by my position being less coherent? I want an international agreement to avoid building superintelligence, because I think it might kill us all. Very few of the more “mundane” concerns justify this—my belief in what actions we need to take follows directly from the severity of the consequences if we don’t.

“AI might kill us all, therefore we should internationally coordinate to make it illegal to build superintelligent AI” at least makes sense if you accept the first premise. Try replacing “AI might kill us all” with “AI uses a lot of water” or “AI produces slop content” or “AI steals from artists” and my next question would be “Well, hold on, isn’t the cure worse than the disease here?” It doesn’t actually make sense, to say that we should ban AI development because images were stolen. If I’m worried about AI stealing from artists, we should compensate artists or enforce copyright. If I’m worried about AI water use, we should tax or cap water use in data centers. If I’m worried about AI job loss, we should find a way to tax AI revenue and support those who were automated away. None of this prevents human extinction. Why would it? You’ve told someone to work on a totally different problem, you can’t be surprised if they go and do that.

Your advice around figuring out your audience and what they care about is good. I think that the correct lesson to learn from the arts of persuasion is how to present your case in a way that resonates with your audience. But every lie you tell makes your story worse. That’s a cost too.

Jay Bailey 12 May 2026 4:39 UTC
2 points
0
in reply to: emile delcourt’s comment on: emile delcourt’s Shortform
The most valuable part of a story such as this, in my mind, is how it can act as a filter. Anything that does not meet your requirements can be disregarded as a line of work. From the original article: (https://www.lesswrong.com/posts/8t8jdTyq6X2B7D8St/have-an-unreasonably-specific-story-about-the-future)

“Anything that is unlikely to help with at least one of those critical paths is a strategy I reject as insufficient in my preference ordering.”

So if we look at your story, it seems like the filtering line here is: “if presented with a compelling case of clear and present risk of irreversible damage to society”.

So, what that means is, for every piece of persuasive material you’re considering writing, you can now ask “Would this present a compelling case of clear and present risk of irreversible damage to society?” Ideally you’d have some ideas that can do all three of these things—if not, you may have to settle for two, but at least it provides something to filter against. And on the experiment level you can do similar. This time you have to ask “Is there a result I could reasonably get from this experiment that would help present a clear and present risk of irreversible damage?” because obviously you don’t necessarily know the results in advance here. Though, in your shoes, I would recommend these further experiments only if you can’t create this case yet, or if you have your best case already written up and now want to make it stronger. If your goal is to present this case, try to take the most direct path towards it that you can.

The other thing is that I think Step 5 isn’t concrete enough yet. You say: “When someone learns something, the next question is always the same. What do we do next? It’s the golden moment.” I agree with you. However, if I am a politician asking you this question, and you reply with:

“We organize behavior change, working extremely efficiently through chokepoints (even simplifying processes and systems to create some if necessary), to re-align systems, people, and impact.”

That’s where you would lose me. What behavior do we need to change? What systems? What next steps can I, as a decision maker, take? I’d recommend coming up with some more actionable next steps to provide to those people who are convinced or open to your findings. If your golden moment is people asking you this exact question, you should be sure you’re as ready as possible for people to do so.

Jay Bailey 11 May 2026 7:51 UTC
4 points
2
on: The Anti-Singularity
Rather than building a single SAI and trusting our future—good or bad—to it, we instead will find ourselves tending to a diverse garden of different AIs, each optimized to a different environment. In this future, Humanity does not simply build a Last Invention and then enjoy a golden retirement. Rather, the future is filled with an endless series of new and unique challenges that we must adapt—and dare I say evolve—in response to.
I don’t understand why the previous sections in the post implies that. Here’s my understanding of the argument:

Intelligence is largely about understanding patterns that allow you to simplify underlying domains. However, many domains we care about have a tremendous amount of irreducible complexity—biological organisms evolved via a pseudorandom process, there’s no reason to expect them to obey clean laws of abstraction. There is no unified theory of biology because there is no simple set of rules that lead to emergent complexity. It’s just tons of weird stuff built on top of each other all the way down. Therefore, an ASI couldn’t just understand all of biology and then one-shot an immortality drug. Creating an immortality drug would necessarily involve a great deal of trial and error—an optimal approach might look something like a random sampling across a very large space of possibilities, which occasionally hits on something that lets you narrow down in the possibility space.

Thus, we can’t have powerful AI, and we won’t have a single ASI controlling everything. We’ll have narrow AI’s optimised for different things instead, and humans will be the gardeners of these optimisers.

===

But why on earth does the second paragraph follow from the first? What’s stopping an ASI from figuring out the best way to navigate these heuristics and applying them itself, the same way humans manage to do both biology and discrete mathematics without having separate bio-humans and math-humans? If the best strategy to solve science is to have many different narrow AI’s, the ASI is the thing that can create those narrow AI’s. Why exactly is it that humans are the ones who “become gardeners atop a wild ecosystem of heuristic optimizers” when your own example of Doom from the start was about an AI doing this right now?

It seems like the “anti-singularity” just means “We have an ASI that can do everything, except the ASI also needs to invest an enormous amount of labor into solving computationally intractable domains. It will only be mildly more sample-efficient than humans and largely have to rely on being able to think faster and work ²⁴⁄₇ to have an advantage over people in these areas.” So, we don’t get an AI bootstrapping itself into godhood, but we still become obsolete eventually.

But that’s not what you seem to be saying. You seem to go straight from “Domains like biology have a large amount of computational irreducibility” to “Therefore, humans will always remain at the highest level of abstraction.” Why? An ASI doesn’t have to find a unified theory of biology to make us essentially obsolete. Seems like an anti-singularity just gets us to the same place slower.

Jay Bailey 10 May 2026 0:13 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
This is where my knowledge ends I’m afraid—the EA Forum / GiveWell may know more, or perhaps someone here can chime in!

Jay Bailey 9 May 2026 22:31 UTC
8 points
1
in reply to: leogao’s comment on: leogao’s Shortform
I think part of it is what you said. Even in Western countries where people have way more education and slack, people often underinvest in preventative health like flu shots, so it makes sense that people in third-world countries would fall prey to this too, and many more probably don’t even know about the benefits of nets or how to get them.

Another pair is economies of scale and spillover effects. IIRC it’s cheaper per net to protect a whole village at once, and because malaria nets are treated with insecticide, nets kill mosquitos and so they protect other people as well. And people tend to underinvest in things with positive externalities like that—again we go back to the flu shot example.

There’s probably other reasons, but these are the ones I know of.

Jay Bailey 24 Apr 2026 10:33 UTC
10 points
12
in reply to: Richard_Kennaway’s comment on: leogao’s Shortform
I don’t know how many people explicitly believe it but there is a general worldview that inherently assumes it. There are common memes that use this to show the unfairness of pay disparity, such as this one. Inherently it assumes the only fair way for one person to be paid 351x more than another is if they work 351x harder—LTV. https://www.reddit.com/r/antiwork/comments/yrdbyg/ceos_are_not_worth_351_times_the_average_worker/

Jay Bailey 23 Apr 2026 2:43 UTC
3 points
0
in reply to: Bill Seitz’s comment on: There are only four skills: design, technical, management and physical
I can try this:

A human being should be able to: change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts , build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, and lastly die gallantly.

Change a diaper—Physical

Plan an invasion—Design

Butcher a hog—Physical

Conn a ship—Physical

Design a building—Design

Write a sonnet—Design

Balance accounts—Technical

Build a wall—Physical. Design if you also need to plan it.

Set a bone—Physical

Comfort the dying—Does not fit (DNF) or maaaaybe Management

Take orders—DNF

Give orders—Management

Cooperate—Management

Act alone—DNF

Solve equations—Technical

Analyze a new problem—Design

Program a computer—Technical

Cook a tasty meal—Physical

Fight efficiently—Physical

Die gallantly—DNF

The ones that don’t fit are generally quite broad cross-domain soft skills (Take orders, act alone) or social/philosophical ones (Comfort the dying, die gallantly) but for the most part, these items can fit pretty well.

Jay Bailey 21 Apr 2026 1:45 UTC
2 points
0
in reply to: bohaska’s comment on: There are only four skills: design, technical, management and physical
Drawing does seem like a physical skill to me. Habryka’s mentioned elsewhere in this comment section that playing an instrument is a physical skill and I think drawing is quite similar to this.

What makes this more complicated is that it seems feasible that a complex piece requires the design skill to configure it, then the physical skill to make it a reality.

Jay Bailey 20 Apr 2026 2:45 UTC
2 points
0
in reply to: davekasten’s comment on: There are only four skills: design, technical, management and physical
Not Habryka, but it seems like according to this article policy creation is pretty solidly a design skill, whereas the art of getting people to get on board with your proposed policy is a management skill. Sometimes policy creation also has technical prerequisites if you’re writing stuff in technical domains that need to engage with specific facts on the ground like FLOP counts and such.

I think producing a policy document is very close to producing a law document, and law is explicitly called out as a design skill.

Jay Bailey 20 Apr 2026 1:36 UTC
11 points
7
in reply to: dan.parshall’s comment on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026
The Orthogonality Thesis says you could have a mind of arbitrary intelligence pointed at any goal, not that any specific training process will wind up pointed at some random target. What you observe is entirely consistent with orthogonality being true.

The main argument this is meant to argue against is “Well, if you had a paperclip maximiser that got really smart, it would realise that maximising paperclips was stupid and decide to do something else instead.” If this seems obviously incorrect to you, then the Orthogonality Thesis has done its job.

Eliezer does make arguments that AI’s will not be pointed at the thing you try to point them at, in a similar way to how evolution didn’t evolve to make humans robustly care about inclusive genetic fitness, but that isn’t quite the same as the Orthogonality Thesis. Orthogonality is a necessary but not sufficient condition for Eliezer’s arguments on this applying to the kind of AI systems we actually train.

Jay Bailey 19 Apr 2026 6:26 UTC
14 points
0
in reply to: habryka’s comment on: There are only four skills: design, technical, management and physical
For the benefit of later readers, this was Scott Young’s MIT challenge, doing a full CS degree in 12 months. I saw this as rather incredible the first time I saw it, but maybe the more incredible thing was having both the circumstances and the agency to attempt it in the first place. https://www.scotthyoung.com/blog/myprojects/mit-challenge-2/

Jay Bailey 19 Apr 2026 6:20 UTC
11 points
1
in reply to: habryka’s comment on: There are only four skills: design, technical, management and physical
Okay, I think I understand this now. For the benefit of other people with Jay-shaped problems in this area, I’ll write down what I’ve learned:

If I were to design a CRUD app, I would immediately begin decomposing the problem into the basic language of the specific domain of web applications. We need a front-end, a back-end, and a database. We need to connect them up in some way. We need to host the application somewhere. Okay, I know what parts are needed.

Now, what is the specific problem I want to solve with this CRUD app? What differentiates it from any generic CRUD app? What information do I need to show to the user and how in order to ensure they can easily do whatever they will want to do with this app? Now I am in “core design skills” land. However, I had to go through that first paragraph to do so, I couldn’t just start here.

So, it feels like I know nothing about interior design, not because I am bad at every step of interior design but because I am bad at the very first step—I cannot yet break down room design into the various components like lighting, furniture, color schemes, and so on that are required. (Honestly, even being able to write that previous sentence with specific examples indicates that I am less clueless than I originally thought of myself as) Since that’s the first step to solving the problem, all the important skills I have that would actually help with this problem are hidden behind the relatively simple ones I lack.

If I’ve got that right, that is a very powerful and new way of looking at things for me! I wouldn’t be surprised if this turns out to be one of the most valuable posts I see this year. It reminds me of when I first picked up an economics textbook—a bunch of things I didn’t understand but were scary black boxes transformed into things where I knew how I could grow to understand them if I wanted.