Towards_Keeperhood

• Towards_Keeperhood 7 Sep 2026 16:24 UTC

  2 points

  0

  in reply to: orthonormal’s comment on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  “commercial progress would’ve been properly halted” still seems like a strong overstatement, and I wouldn’t expect brand failures to have been that permanent if there are commercial incentives to fix those problems.

  But whatever, I don’t think we disagree on recommendations much, I just think you’re overstating the case.

• Towards_Keeperhood 7 Sep 2026 6:56 UTC

  2 points

  0

  in reply to: orthonormal’s comment on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  xAI not having many users isn’t surprising to me, given that they were none of first-to-market, best performance, or best alignment. Perhaps they were somewhere on the pareto frontier of cost-effectiveness (i don’t know), but probably not cheaper than deepseek, so only a small fractions of users would have incentives to switch to grok.

  I think they might have much more users if they were first or had the most capable model or maybe even if all the other models were even less aligned.

  Also, even if fewer normal people would use AIs, I think the majority of commercial incentives comes from coding and prospective work automation, and I don’t think adoption would’ve stopped there.

  (Though I do think safety work may now be significantly more important for commercial progress than it was in the past.)

• Towards_Keeperhood 6 Sep 2026 11:51 UTC

  9 points

  0

  in reply to: Eliezer Yudkowsky’s comment on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  What would you want employees to use their leverage for?

  Just leaving now already seems quite useful because it may slow companies down and increase warning signs, and indeed in 2-8 months this might start to become less useful as AIs become better at making AIs not look misaligned. (Though slowdown and warning signs might be even a little more useful the later they happen (but not too late or else we may directly get AI takeover^[1]), so I’d guess my median for the best time to leave is maybe 3 months from now, but very uncertain. And a few people should probably stay behind to be able to whistleblow later.) (EDIT: I’m actually less confident than I sound here about leaving being a good idea.)

  But yeah, AI lab safety researchers coordinating and threatening to leave if the company doesn’t take some very strong actions towards making Plan A/​S more likely seems sensible to try to do first.^[2]

  1. ^

     or get to the point where AIs only do bad stuff if they are pretty sure they won’t get caught and can evaluate this very well, so we probably wouldn’t see more warning signs

  2. ^

     ADDED: Perhaps do it as a work strike. And btw, should a company at some point start to act like a very responsible entity that actually understood the safety problem well would act, i.e. it does basically anything it can do to get a treaty, it seems perhaps good to (re)join that company for safety.

• Towards_Keeperhood 6 Sep 2026 11:08 UTC

  2 points

  0

  in reply to: WillPetillo’s comment on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  Warning shots are something we should focus on preparing a response to, if needed, not try to make more likely (directly or through intentional inaction).

  Only responding to the “or through intentional inaction” part:

  I think it’s important to clearly separate worldstate-question-part from the moral-principle-question-part of the “should we not take action to prevent warning shots” question here:

  1. the worldstate part: “Do we think that if a warning shot happens, the expected good it causes is greater than the the harm it causes?” (“good” according to common human values.)

  2. the moral principle part: “Assuming we think the expected good is greater (and not by any Pascal’s mugging), should we prevent the warning shot anyway?”

  I think for a very reflective and rational person, the answer to (2) is “no don’t prevent it”. Even if you disagree here, I think you should at least agree in the hypothetical where the expected good is much much greater (while still without Pascal’s mugging).

  I don’t think this is very ambiguous if you think about concrete scenarios that aren’t xrisk. (I think xrisk is too abstract or too scary for many people to grasp properly.) E.g. if a serial killer is dangling off a cliff, and you could try to save him, but then he will run away and you think there’s a 70% probability that he will kill at least 10 more people, should you try to save him?^[1]

  1. ^

     I guess someone will write “yes because you might be psychotic” or so, but I think that argument should be ignored because I don’t think “AI will kill everyone” is that much more speculative than “there will be a large warning shot where AIs kill people”.

• Towards_Keeperhood 6 Sep 2026 10:29 UTC

  4 points

  2

  in reply to: orthonormal’s comment on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  For instance, I expect that the prosaic alignment improvements from Sydney to the final GPT-4 were surface-level only (as evidenced by later failures like GPT-4o), and if the alignment researchers had refused to play whack-a-mole at the time, then commercial progress would have been properly halted.

  Don’t you think AI companies would’ve found other even less responsible people to do “safety” work that papers over problems?

  I’d expect there’s an equilibrium here which doesn’t get shifted that significantly depending on whether people who care about doing the right thing do or don’t do “safety” work at AI companies. Sure we might’ve seen some warning signs earlier, but then companies would’ve used that as signal to invest more into making their models look safe and not that much would’ve changed. Though open to considerations why this may be wrong.

• Towards_Keeperhood 5 Sep 2026 17:03 UTC

  2 points

  0

  in reply to: Steven Byrnes’s comment on: Cat-Bel­ling Problems

  I did consider that but didn’t think it mattered much because you can easily imagine a lottery where the bottleneck isn’t that you didn’t observe enough information but that you’re not computationally omniscient enough to interpret the evidence, and a fool might come up with reasons why they can totally predict it. But plausible there is some stretched way anyway to frame it as monovariant or so, idk.

  I basically generally disagree with cousin_it though because it’s usually the case that if you have a solution to a difficult problem you must’ve understood the problem well enough to know what some hard parts are and be able to state them, so I think it’s fine to ask for clearly explaining the core of the problem and how the solution attacks it.

• Towards_Keeperhood 5 Sep 2026 16:35 UTC

  2 points

  0

  in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform

  I think the main motivation of why warning shots seem good isn’t that labs take safety more seriously but that the government does. Like if a model had exfiltrated itself and hacked more datacenters and drained cryptocurrencies and we had to scramble to shut it off, the USG might start to consider things like Plan A.

  I think companies continue racing capabilities without doing safety could cause something like that. Final tradeoffs are of course complicated, although there are equilibrium dynamics that show a lot of effects for counterfactual impact roughly equal each other out.

• Towards_Keeperhood 5 Sep 2026 16:21 UTC

  2 points

  0

  in reply to: Caleb Biddulph’s comment on: Daniel Tan’s Shortform

  I was also thinking about this as part of evaluating what work seems useful to do. I wrote up my model here: https://​​www.lesswrong.com/​​posts/​​kxHiSsNh4MH82nhXD/​​assessing-the-impact-of-safety-work-needs-equilibrium

  (My “commercial incentives equilibrium” maps to your CSI->TSI and my “risk awareness equilibrium” to TSI->ISI I think.)

  But you can just improve safety techniques and increase TSI from the outside, to help people working on safety inside the lab without substituting for them.

  Helping people working on safety inside the lab still contributes to reducing warning shots. To a lesser extent, it can also reduce TSI because the company thinks it can copy work from the outside and doesn’t need to do all the safety work itself.

  (I still think some research is good, but it depends on what kind of research and what timing.)

• Towards_Keeperhood 5 Sep 2026 16:05 UTC

  5 points

  0

  on: Should safety re­searchers quit fron­tier labs re. warn­ing shots?

  I think there are good reasons for working at AI labs, but your reasons aren’t it (see below for why). Good reasons include:

  1. It’s good if people doing the safety work actually care about xrisk instead of optimizing for legible safety metrics, because this reduces the chance that AI labs just paper over problems in a way that produces deceptive AIs.

  2. You have the option to whistleblow later when it becomes important.

  3. You might have a small marginal affect on making people at the company a bit saner. (Plausibly not a consideration relevant for OpenAI; idk.)

  4. You earn a high salary which you can mostly donate to work that helps reduce xrisk.

  (Of course, 2-4 shouldn’t just be used as rationalizations that are then never actually acted upon.)

  I think the safety work output itself is approximately zero net value because it now exists inside an equilibrium where safety work there doesn’t help shift the overall equilibrium. See the explanation I just posted here. (Actually I think it’s very slightly harmful (see the post) but the factors above can compensate that well (which of course doesn’t imply that working at an AI lab is optimal for people to do even if it’s net positive).)

  Why I think your points aren’t strong:

  • Alignment MVPs are probably still useful: In spite of the OpenAI x Hugging Face incident, I expect that most AI safety researchers will use frontier models to help with research, including agent foundations researchers. Alignment MVPs, AIs that are sufficiently aligned/​controlled to aid research, may remain a crucial part of AI safety research during an AI pause or slowdown. Currently, the best AIs would arguably be unusable for safety research without the hard work of frontier AI company post-training and safety teams. If AI safety researchers quit frontier AI companies en masse, we might be stuck using current generation AIs for the duration of an AI pause; maybe this is unideal?

  I don’t think anything AI company safety teams did helped elicit any agent foundations research capability. (Though feel free to mention evidence if you have some.) I think it’s mostly about speeding up work safety teams at the work they themselves do, and whether that is good just comes down to the “is safety or warning shots more important” question this is all about. I.e. it’s only valid if you assume the conclusion that safety is more important.

  “Safety-stragglers” will cause warning shots anyways

  Recent events rather seem to point in the opposite direction, and the time gap might matter a lot.

  The purpose of an AI pause is do more safety and resilience work: [...] If we curtail the development of new AI safety researchers by discouraging them from working with research mentors at frontier AI companies now, we might have a smaller talent pool to capitalize on an AI pause.

  I think this is backwards and that people don’t learn very relevant stuff for more adequate alignment scenarios at AI labs and could better do so elsewhere. (Though I concede that hiring spots at other orgs is limited.)

  • The next warning shot might be lethal: It’s possible that the next OAIxHF-style incident causes loss of life, via bioweapons or cyber attacks on critical systems. [...]

  • Allowing warning shots “for the greater good” seems morally fraught

  I think there’s a difference between actively causing deaths for the greater good and letting deaths happen because otherwise much much much more deaths would probably happen later. I think there should be an ethical injunction against the former but not the latter. We both could be working in global health and development and prevent deaths there on a shorter timescale than in the case of AI, and yet I think it is not morally fraught to decide that AI seems more important.^[1]

  (You writing this makes me think you don’t comprehend the situation we’re in or the scale of the moral horror of extinction on a very deep level. If the math was clearly saying that in expectation much much more people would die if you do a thing that reduces that chance of some smaller sooner harm, I think it would be actively unethical to do the thing anyway.^[2])

  1. ^

     If it was a different situation where you alone could save people who would otherwise die for certain immediately, and the larger harm is uncertain, I can see why some people would consider it fraught, although I wouldn’t. It’s very much not the case here though. Xrisk isn’t much more speculative than that you’re preventing warning shots.

  2. ^

     To be celar, I’m not saying the math is saying that, I’m just invalidating this argument. If all other objections had been thoroughly found invalid and it was clear, I think your “morally fraught” argument is actually recommending something unethical.

Assess­ing the im­pact of safety work needs equil­ibrium anal­y­sis (now more than ever)

• Towards_Keeperhood 4 Sep 2026 19:04 UTC

  2 points

  2

  on: Cat-Bel­ling Problems

  Great post! Ironic though that you mostly sorta gave up on attacking the hard part of the problem of writing this particular post well. xD

  (I.e. the post is missing the part of the argument that is hard to write which on the surface somewhat funnily resembles the flawed perpetual motion machine prototypes.)

• Towards_Keeperhood 4 Sep 2026 18:10 UTC

  8 points

  5

  in reply to: cousin_it’s comment on: Cat-Bel­ling Problems

  How would you state the invariant in the case where someone tries to forecast lottery numbers?

  Invariant framing doesn’t seem natural to me.

[Macroa­gents] 2. De­sign lenses for op­ti­miz­ing macroagents

[Macroa­gents] 1. The Macroa­gent Ontology

Distil­la­tion of the AI2040 Align­ment Roadmap

• Towards_Keeperhood 23 Aug 2026 12:00 UTC

  2 points

  0

  on: Futarchy’s fun­da­men­tal flaw

  Finally, the flaw can be fixed. In statistics, there’s a whole category of techniques to get causal estimates out of data. Many of these methods have analogies as alternative prediction market designs. I’ll talk about those next time. But here’s a preview: None are free.

  AFAICT you didn’t write a follow up post about fixes, right? Do you plan to still do that sometime?

  (Or if you have a only half-finished draft lying around since a while, could you PM me to share it?)

• Towards_Keeperhood 23 Aug 2026 11:57 UTC

  2 points

  0

  on: Futarchy’s fun­da­men­tal flaw

  Very interesting post!

  Below is an excerpt of a post I am currently drafting, and I would be curious of whether you have disagreements with something I wrote here. (Mostly point 2 is relevant to you.):

  If you want to incentivize truthfulness, you can ask agents for predictions, wait until you can see the result, and score them according to logloss. You can also have them bet in prediction markets. However, there are problems that need to be addressed, most importantly:

  1. Agents betting in prediction markets have incentives to influence the outcome (e.g. by assassinating people) so they can make profit on their insider knowledge of what they will do.

  2. When the prediction is used to make decisions, you run into difficulties. As explained here, in a simple futarchy with conditional prediction markets, the thing that is being bet on is p(outcome | action), whereas what we actually want is p(outcome | do(action)). In particular, agents need to take into account the fact that if an action is chosen, the market probably predicted high p(outcome | action), which leads to confusing self-prediction dynamics. [Footnote: I think (but not sure) in many cases p(outcome | best_predicted_action) may converge to roughly p(outcome | do(best_predicted_action)) very shortly before the action is actually determined. But EDT has some serious flaws like XOR blackmailability that can cause different outcomes and serious problems.] Until shortly before the decision is made (where markets on counterfactual branches get cancelled), the market prices can be very inaccurate[Footnote: This is because action branches, where there’s more information still coming in for predicting how good they would turn out, will be overvalued, since that market-branch is less likely to get cancelled if the information suggests the branch might be favorable. There was an experiment testing one of dynomight’s thought experiments, and it did show this phenomenon. (Note that in the experiment, the final decision of the market was the correct decision (given that it was known that coin B always lands heads). (The decision point is the point where the non-top markets get cancelled.))], and the prices on the counterfactuals may stay inaccurate. [Footnote: It could be that there are cases where (even in theory) futarchy would result in decisions that are insane even from an EDT perspective, although I would very tentatively think this is not the case (but didn’t think much about it).] [Footnote: Here’s one proposed fix to the “futarchy implements EDT” problem, although it has weak-betting-incentives problems. (I’m not sure whether those weak-betting-incentives problems could be addressed well, e.g. by subsidizing the market. Didn’t think about it much.) The proposal works by only predicting cases where you use a fixed randomized intervention protocol for choosing the action, i.e. by having the market not affect decisions in the worlds where it pays out. But there might be other problems where the prediction affects the outcome which cannot be easily sidestepped and a deeper solution here would be desirable. Also, ideally we’d want to implement LDT somehow, not just CDT.]

• Towards_Keeperhood 12 Aug 2026 10:56 UTC

  2 points

  2

  in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform

  I think it’s important that we distinguish between slopiness from AIs not having been taught to try hard with sensible approaches (elicitation) and slopiness because the AI capability profile is shaped in a way that AIs are just not very good at some tasks. Though I agree it’s a spectrum.

  I think it’s probably not the case that the profile of potential AI capabilities (aka the profile if all capabilities were well elicited) is very similar to that of humans. In particular, I expect AIs to be e.g. differentially quite bad at e.g. MIRI-like conceptual alignment research where you need to grapple very deeply with a phenomenon or puzzle and form good new concepts/​ontologies/​frameworks to get a handle on that. Or like deriving general relativity from as little evidence as Einstein did. I.e. even if we tried hard and had more useful data here we might have a hard time to make AIs good at that. (E.g. maybe current AIs may lack some cognitive machinery part that humans have for forming natural ontologies. Or for noticing confusions /​ seeing what evidence is suprising given the world model and using that to find where beliefs may be wrong.)

  So I guess actually like 3 levels on a spectrum:

  1. actual elicited capabilities

  2. capability profile (what could be elicited from an AI model with good post-training (aka assuming you have good datasets/​tests))

  3. architecture potential profile

  (I realize this is only tangential to your comment, but I’ve been reading a bunch of stuff from Ryan lately and I felt this distinction wasn’t made properly but there’s no clear location where to best comment it and I just felt an impulse now.)

  (Or like, Ryan mentions both 1 and 2, but not sure if he thinks there’s a meaningful difference on 3 to humans. Personally 3 does make me more pressimistic about relative speedups of alignment research vs capability research. But tbc it could be that 3 is still not that important if we find kinds of safety research that don’t need that much of forming deep novel ontologies or so. But it could also be important.)

• Towards_Keeperhood 5 Aug 2026 16:08 UTC

  2 points

  0

  on: How do we (more) safely defer to AIs?

  I think epistemics (which you discuss at the end of your “objectives with broad alignment” section) should fall into the capability profile category, not broad alignment.

• Towards_Keeperhood 5 Aug 2026 16:05 UTC

  2 points

  0

  on: How do we (more) safely defer to AIs?

  You contradict yourself in the way you define the ontology. You write (emphasis mine):

  I’ll use the term “deference-goodness” to refer to differential improvements to AIs being effective and aligned at key tasks (as in, differential relative to just increasing general capabilities).^[16](I’m not including work which is just useful for reducing the chance that AIs are seriously misaligned in deference-goodness.)

  And then later you write:

  We can break down deference-goodness into an alignment/​elicitation component and a capability profile component:

  • Broad alignment: [...] The AIs can’t be seriously misaligned.^[18]

  I think having this distinction is needlessly complicated:

  • Aren’t seriously misaligned: They remain corrigible to the intended group/​structure and don’t try to seize power or kill humans.

  • Are effective and aligned at key tasks: The AIs do a good job—which includes being sufficiently aligned and sufficiently capable—of autonomously: [...]

  (The way I am reading this is that on key tasks, the second criterion implies the first (so the first only is relevant on non-key tasks because the second doesn’t apply there). If you meant sth different I don’t know what you meant.)

  If AIs are sufficiently aligned at the key tasks, they are likely also generally not seriously misaligned. I think you agree with this given footnote 16.

  I think it would be less confusing if you had defined the ontology like:

  AIs need to be in the BGD, which means they must be sufficiently aligned, and effective at key tasks: [...]

  And then defining deference-goodness via that. [ADDED: Ofc in a way that deference-goodness is meaningful even before AIs would be capable enough for handoff.] (The fact that alignment on non-key tasks doesn’t necessarily need to be quite as strong is still expressed through that, but now only implicitly, because it’s not that important.) This would make the way you define broad alignment above correct again, although in other parts of the post you are also mentioning that preventing egregious misalignment isn’t part of broad alignment.

  (I guess why you thought separating might make sense because the “aren’t seriously misaligned” maps to “the property that makes alignment through behavioral training possible”, but this doesn’t seem like a strong reason and the cost of having a confusing fuzzy ontology between broad alignment and preventing egregious/​serious misalignment seems much larger. (Tbc, it’s totally fine to have a concept for “make sure tests are representative” or “preventing alignment faking”, but I think it would be better if that just counts as part of broad alignment.)) (Or maybe you think the methods for preventing serious misalignment are very different from achieving broad alignment? That would surprise me though.) (Or maybe you were first thinking more about elicitation and then realizing that post-handoff most of elicitation is captured by alignment? That would explain it but not be a good reason.)

  (If you decide to keep the split, then probably rename “broad alignment” to “broad alignment at key tasks”.)

  (Tbc, I’m not suggesting editing the whole post ofc, just saying what I think is the better ontology going forward. Whether you add an edit note is up to you.)