Jay Bailey

Karma: 848

Jay Bailey 25 Nov 2025 9:31 UTC
4 points
0
on: The Enemy Gets The Last Hit
This is the kind of content I keep coming back to this site for.
- Obviously correct
- Immediately useful as a day-to-day habit of thought
- “Why didn’t I think of that!?”
I also like that it’s practical and practicable in day to day life while also being important for bigger, important questions.

Jay Bailey 23 Nov 2025 22:47 UTC
6 points
0
on: If you cannot be good, at least be bad correctly
Another example to add to the post. A few years ago I was learning to play Go, and thus playing against superior players. In Go, typically what happens is one player will make a move that signals something, like “I can take this territory” or “I can attack this group of pieces successfully” and the other player needs to decide whether to directly oppose this move, or to accept it and play elsewhere instead.

So I asked myself—if I disagree with a superior player’s assessment of the situation (i.e. he makes a move I think I can punish), what should I do? Statistically I’m probably wrong about my assessment. But then I realized that if I accepted their move, I would never find out why they were right. If I opposed the move, and was quickly punished (my group was destroyed, or the player successfully defended their “too aggressive” move) I would get to find out why I was wrong immediately. So I began playing according to my instincts even though they would lose me more exchanges in the short term, since it was the direction of maximum learning.

Jay Bailey 16 Nov 2025 0:21 UTC
1 point
0
in reply to: Raemon’s comment on: What are your impossible problems?
Outside of the office, I generally find it difficult to get appreciable amounts of work done. It feels like it takes a significant exertion of willpower to go from not doing a work-related task to doing one, at which point it generally becomes easier to continue with that task for a while. Performing this mental motion a few times per workday is enough for me to get close to full time hours in, but doesn’t feel enough for sixty hours. If I don’t perform this mental motion successfully, I wind up in a state of internal tension where I’m not actually putting in consistent mental effort towards solving the next problem in front of me. I do have a fix for this already—I work better in an office, where everyone around is also working.

So, the natural solution here would be “Find yourself an office type environment doing valuable work where it is entirely normal and expected for people to work these kind of hours.” This runs into the second issue, which is that on the rare occasions I have worked 10+ hours in a day, or generally pushed myself harder to try and stretch the edge of my conscientiousness, I tend to get headaches. Which are both unpleasant and also reduce my productivity, which ruins the entire point of the exercise. (The headaches are a known problem—I’m on a headache preventer that minimizes them. I could try upping the dosage, but I’ve already been told by one doctor that I should probably try not to be on this medication indefinitely.)

This also means that it wouldn’t be a good idea to go to an environment where everyone’s expected to work 60 hours, if I then don’t end up being able to do that even with the social and logistical environment set up in my favor. So it’d have to be an environment where it was both normal and expected to work 60 hours AND to work 40, AND the work was object-level valuable in my opinion, to be worth trying this experiment. This could be possible—I haven’t actually tried to seek out such an environment. But I do notice that I still do anticipate failing the original 60-hour goal due to the problem in the second paragraph.

Jay Bailey 15 Nov 2025 10:16 UTC
1 point
0
in reply to: Raemon’s comment on: What are your impossible problems?
On thinking about them:
“Having both the motivation and the mental stamina to work 60-hour weeks reliably.” Actually this probably would be “hard” rather than “impossible”. There are things I can try here that I haven’t tried that might work, so I have not yet tried enough things to declare it impossible. It’s more like I anticipate the possibility of this being impossible, as opposed to actually considering it impossible. Not a good example.
“Gaining 15+ points of IQ or the thing that IQ is measuring.”—So it seems like there are two ways this problem could go. In World A, there exists some stack of exercises and nootropics that can already let me do this without sacrificing something I’m not willing to sacrifice. In this case the problem is that of finding it—people can bullshit or lie, supplements are not a field where I expect complete honesty, and it wouldn’t surprise me if there’s a high amount of individualism such that someone could truthfully say they gained 15+ IQ points on X, but I need Y instead to achieve the same outcome. If I had to try each combination myself I’d run out of time very quickly. It’s impossible in the sense of winning the lottery is impossible—not something I can reliably make happen, as opposed to literally can’t be done. This is close enough that I consider it to be equivalent.
Alternatively, maybe this stack just doesn’t exist at all. Nootropics and exercises will not get you there. In which case, solving the problem means making advances in cognitive science that our culture hasn’t yet figured out how to do. And it’s not clear to me how I would succeed where lots of others have failed, here—leading onto my third point, I don’t see how I have a comparative advantage in this area, and if I have to work full-time for years to get to this point it is no longer worth it, when the point is to make me more effective at solving my current chosen problem.
“Taking something that isn’t my comparative advantage and making it that way”—I can think of a bunch of actions I can take here that would let me do better than I currently am. Getting tutoring, improving my ability to learn, talking it over with more experienced practitioners, etc. The key impossibility here is that people who are better than me at this can just do this stuff too and probably already are.
Like, imagine I want to be a top mathematician since I’m convinced that’s the only skill worth knowing for alignment. I can hire a tutor (and have done so), but better mathematicians can also do this and probably have. How do you get good enough to meaningfully contribute when people exist in this field who did the IMO at 17 and can do any of the same improvements I can come up with? So, I would have to find some method that was A) Incredibly effective, enough to bring me up to par with more talented people and B) Other people are unable or unwilling to do this thing, even really talented people in the field.

Jay Bailey 15 Nov 2025 1:41 UTC
8 points
0
on: What are your impossible problems?
I don’t know how much context you need for the more personal examples, so I figure I’ll give them without context and then if you need more you can ask:
- Having both the motivation and the mental stamina to work 60-hour weeks reliably.
- Gaining 15+ points of IQ or the thing that IQ is measuring.
- Becoming good enough at a field (abstract mathematics, mechanistic interpretability) that I’ve previously tried and found myself to be not that good at or interested in, such that it would then be worth my pursuing it as part of a research direction. (Another way you could write this is “Taking something that isn’t my comparative advantage and making it that way”)
- Convincing an arbitrary person my values are good and worth adopting within a sixty-minute conversation. (I don’t actually WANT to do this in full generality! But it sure does seem impossible and it sure would be nice to do things that are closer to that area, sometimes)
- Becoming a professional tennis player at the age of 33 with little tennis experience. (Again not something I actually want, but it sure does seem impossible. I figure if this is a bad example, you can just not include it.)

Jay Bailey 10 Nov 2025 0:20 UTC
1 point
0
on: Learning information which is full of spiders
Unrelated to the main point, but—where is this second Merrin story found? I really liked the first one, but didn’t know there was a second one, and glowfic authors tend to have a lot of story threads.

Jay Bailey 5 Nov 2025 9:15 UTC
6 points
2
in reply to: johnswentworth’s comment on: Heroic Responsibility
I am reminded of the Message for Garcia essay as a pretty striking example of this.

Jay Bailey 1 Nov 2025 11:31 UTC
2 points
0
in reply to: dscft’s comment on: Jay Bailey’s Shortform
One would hope indeed. But even so, we do now know that this is likely to be the kind of action that could be detected and opposed. And since I didn’t predict in advance that this would happen, especially at this capability level, the update for me is that it’s going to be significantly harder than I would hope.

Jay Bailey 1 Nov 2025 1:59 UTC
31 points
4
on: Jay Bailey’s Shortform
I recently read Anthropic’s new paper on introspection in LLM’s. (https://www.anthropic.com/research/introspection)

In short, they were able to:
- Extract what they called an ALL CAPS vector, the difference between a prompt with ALL CAPS and the same prompt without all caps.
- Inject that vector into the activations of an unrelated prompt.
- Ask the model if it could detect an injected thought. About 20% of the time it said yes, and identified it with something like “loud” or “shouting” which is quite similar to the all caps they were going for.
They did this with a bunch of concepts. And Opus 4 was, of course, better than other models at this.

To give some fairly obvious thoughts on what this means:
- If a model can unreliably do this now when directly prompted to do so, I expect the model can reliably do this without being prompted in around 2-3 years from now.
- If you have an alignment plan that involves altering the model’s thoughts in some way, consider that the model will probably be aware by default that you did that, and plan accordingly.
Finally a more personal thought:

The idea of “Well, if you try to adjust a superintelligence’s thoughts, it will know you did that and try to subvert it / route around the imposing thoughts” would have sounded weird to me, if I’d tried saying it out loud. We’re talking about literally altering the model’s mind to think what we want it to think, and you still think that’s not enough for alignment? I note that I did think that, but I was thinking in terms of “The concepts you are altering are a proxy for the thing you are actually trying to alter, and the important performance-enhancing stuff like deception will migrate to the other parts” and not “The model will detect you altering its mind and likely actively work against you when it builds the situational awareness for that”. And yet, here we are, with me insufficiently paranoid. Unfortunately, reality doesn’t care about what sounds intuitively plausible.

I wonder if this is an example that would allow people to feel, viscerally, the difficulty of aligning an actually smart entity. Here we have a capability that sounds scary and superhuman and only arose with smart models. The idea of a model being aware of when you’re altering its thoughts is terrifying. The lesson we would like to give to decision-makers is something like:

“AI’s will soon know when you try to adjust its thoughts. That’s how smart these things are. We can employ a primitive type of mind control on them and that still isn’t enough. We are really not ready for even smarter systems than this. Any alignment plan that might work has to be stronger than mind control. Anything below the level of literal mind control isn’t even the ante to sit down at the table any more.”

I can’t literally reduce this down to five words, but a succinct message might be something like “Mind control won’t work. We need a plan that’s better than that.”

Jay Bailey 26 Sep 2025 1:30 UTC
1 point
0
in reply to: yanni kyriacos’s comment on: Widening AI Safety’s talent pipeline by meeting people where they are
Makes sense. Would be curious about estimated costs if this wasn’t the case!

Jay Bailey 25 Sep 2025 23:56 UTC
1 point
0
on: Widening AI Safety’s talent pipeline by meeting people where they are
Great stuff—super impressed on how much you achieved on this budget! Those numbers seem incredibly impressive, and definitely seems to indicate a lot of value can be had by this type of program. Running it in Sydney/Melbourne also means you have less diminishing returns—normally I’d expect this to be the best program you could run since you’re picking from the most pent-up demand for this course, but I can’t imagine people likely to come from abroad for a part-time 14-week course.

Jay Bailey 15 Sep 2025 6:05 UTC
2 points
1
in reply to: Kristaps Zilgalvis’s comment on: Contra Shrimp Welfare.
I can see why you might find that frustrating. I think a lot of us, myself included, do think that the science is the most important part of the argument—but we don’t understand the science well enough to distinguish true arguments from false ones, in this domain.

I don’t have the proper expertise to evaluate your claims about pain receptors properly, but I do have the proper expertise to conclude that SWP calling their opponents “irrational, evil, or both” is bad, and that this correlates with shoddy reasoning everywhere else, including the parts I don’t understand. Thus, there’s a limit to how much I can update even from an incredibly strong neurological takedown, if that takedown requires knowledge of neurology that I don’t have in order to fully appreciate its correctness.

So, in terms of what we care about, think of it less as “How many bits of information should this point be worth” and more “How many bits of information can your audience actually draw from this piece of information?”

Jay Bailey 14 Sep 2025 6:47 UTC
6 points
3
in reply to: Kristaps Zilgalvis’s comment on: Contra Shrimp Welfare.
If the above is true, I think this is really good information that would have been very nice to have cited within the article. That would make me a lot more skeptical of SWP and of their conclusions, and it’d be great to see links for these examples if you could provide them.

Especially this paragraph:

“It just intuitively seems like they are.” This is proposed as a rebuttal for critiques of the shrimp welfare project, not very convincing to me, yet they claim that those who don’t support them are “irrational, evil or both”. I find that making that claim with sparse, scattered and unclear evidence is not great, and paints anyone who opposes their views as as flawed person.

I agree with the value claims in this paragraph completely, so if you have sources for those quotes I think that would be very persuasive to a lot of us here on this site, and it might even be worth a labelled edit to the main post.

Jay Bailey 12 Sep 2025 3:37 UTC
59 points
28
on: Contra Shrimp Welfare.
I think you make some interesting points here, but there are two points I would disagree with:

First is “The Shrimp Welfare Project wants shrimp to suffer so they can have a new problem to solve.” This claim is made with no supporting evidence whatsoever. You don’t even argue for why it might be the case, and show no curiosity about other explanations for this. They claim to disagree with you, so clearly they have ulterior, malicious motives. (I would say knowingly creating a charity that doesn’t solve a real problem, just to be able to say you’re solving a new problem, is quite unethical!) Why is it so hard to believe the people who founded SWP did so with the intent of reducing as much suffering as possible, and just happened to be incorrect? What makes you completely dismiss this hypothesis so much that it isn’t even worth mentioning the alternative in your article?

Second is “At best, a shrimp sentium would encode only the most surface-level sensory experience: raw sensation without context, meaning, or emotional depth. Think of the mildest irritation you can imagine, like the persistent squeak of a shopping cart wheel at Walmart.”

I don’t see how the second sentence follows from the first. When I imagine a migraine, the worst pain I personally have ever experienced (being a rather fortunate individual) it doesn’t seem to me like the reason I am suffering is because of the context, meaning, or emotional depth of my pain. I’m suffering because it hurts. A lot. It doesn’t seem that complicated. It seems like it would be much more principled, using your analysis, to treat 860,000 shrimps freezing to death as suffering equivalent to one human experiencing the sensation of freezing to death, not experiencing mild irritation. I say “experiencing the sensation of” because things like being aware of one’s own mortality does seem like a thing that’s out of reach of a shrimp. So it’s not equivalent to a human actually dying, in my view, but freezing to death is likely still quite unpleasant, and not something I’d do for fun, and I’d much rather experience a squeaky wheel at Walmart even if I was fine as soon as I lost consciousness and had no chance of mental trauma from the incident, which I think still matches the shrimp equivalence.

What I still think makes this article interesting is that 550 humans experiencing the sensation of freezing to death across twenty minutes is bad, but not as bad as even one human death, which could be prevented by orders of magnitude less cost than a shrimp stunner. So even despite this article’s flaws I still think it’s a good article on net and worth engaging with for a proponent of shrimp welfare.

While a reply isn’t required, if you are going to engage with only one of these points, I would prefer it be the first one even though I wrote a lot less about it. The second point doesn’t actually change the overall conclusion very much imo, but the first point is generally quite confusing to me, and makes me less confident about rest of the article given the quality of reasoning in that claim.

Jay Bailey 12 Sep 2025 3:12 UTC
2 points
1
in reply to: geoffreymiller’s comment on: My talk on AI risks at the National Conservatism conference last week
This is, as I understand it, the correct pronunciation.

Jay Bailey 30 Jun 2025 11:01 UTC
5 points
4
on: life lessons from poker
I do remember when I was learning poker strategy, and I learned about the idea of wanting to get your entire stack in the middle preflop if you can get paid off with AA—was a very fundamental lesson to young me! That said, there’s a key insight that goes along with the “pocket aces principle” that is missing here, and that’s bankroll management.
In poker, there is standard advice for how much money to have in your bankroll before you sit down at a table at all. E.g, for cash games, it’s at least 2000 big blinds (20x the largest stack you can buy). This is what allows you to bet all-in on pocket aces—if your entire bankroll is on the table, you should bet more conservatively. The point of bankroll management is to allow you to make the +EV play of putting your entire stack into the middle without caring about the variance when you lose 20% of the time.
To apply this metaphor to real life, you might say something like “Consider how much you’re willing to lose in the event things turn out badly (e.g, a year or two on a startup, six months on a relationship) and then, within that amount, bet the house.”

Jay Bailey 26 Jun 2025 14:49 UTC
2 points
−2
on: Jay Bailey’s Shortform
There’s a counterargument to the AGI hype that basically says—of course the labs would want to hype this technology, they make money that way, just because they say they believe in short timelines doesn’t mean it’s true. Specifically, the claim here is not that the AI lab CEO’s are mistaken, but rather that they are actively lying, and they know AGI isn’t around the corner.

What actions have frontier AI labs taken in the last year or two that wouldn’t make sense, given the above explanation? Stuff like GDM’s merger or OpenAI (reportedly) operating at a massive loss. Ideally these actions would be reported on by entities other than those companies themselves, in order to help convince skeptics. I’ve definitely seen stuff like this around but I can’t remember where and the search terms are too vague.

I’ve also tried using Deep Research for this, but it doesn’t seem to understand the idea of only looking at actions that are far more likely in the non-hype world than the hype-world, talking about things like investor decks projecting high returns that are very compatible with them hyping themselves up.

Jay Bailey 22 Feb 2025 19:28 UTC
2 points
0
in reply to: t-cole’s comment on: t-cole’s Shortform
Significant Digits is (or was, a few years ago) considered the best one, to my recollection.

Jay Bailey 17 Feb 2025 11:04 UTC
3 points
0
in reply to: ChristianKl’s comment on: Jay Bailey’s Shortform
Here you go: https://chatgpt.com/share/67b31788-32b0-8013-8bbf-a4100abf0457

Jay Bailey 16 Feb 2025 11:53 UTC
3 points
0
in reply to: MondSemmel’s comment on: Jay Bailey’s Shortform
https://chatgpt.com/share/67b1c1dc-e88c-8013-a6f0-88b25155a0d6

Here you are :)