otto.barten

• otto.barten 8 Mar 2026 12:03 UTC

  10 points

  1

  in reply to: Zack_M_Davis’s comment on: Thoughts on the Pause AI protest

  don’t think like us

  Just want to flag that not everyone on lesswrong is libertarian or right-wing. Left xriskers are a minority but we exist.

• otto.barten 27 Feb 2026 12:05 UTC

  5 points

  0

  in reply to: habryka’s comment on: Re­spon­si­ble Scal­ing Policy v3

  the appetite for conditional risk regulation has been substantially less than the appetite for direct risk regulation

  Where do you see the latter appetite?

  We campaigned a bit for a conditional treaty. We’d happily sign up for un unconditional pause though. Problem is: there is no appetite for either, right?

  I agree that the manpower spent on evals should have been spent on other things with a better theory of change. Eval quality imo is not a crux for regulation, awareness and political support are. I think the money that went to evals should have gone to raising awareness and lobbying.

  Honestly why is there still no significant funding for awareness raising projects? It’s so easy: just ask for amount of views/​copies and conversion rates measured via e.g. Prolific surveys and fund the most effective projects. A fund like this can easily absorb millions. I think this might actually get regulation off the ground.

• otto.barten 29 Jan 2026 0:26 UTC

  1 point

  0

  in reply to: cousin_it’s comment on: Dario Amodei – The Ado­les­cence of Technology

  UN control. A Baruch plan for AGI.

• otto.barten 23 Jan 2026 22:21 UTC

  1 point

  0

  in reply to: the gears to ascension’s comment on: Re­leas­ing TakeOverBench.com: a bench­mark, for AI takeover

  Good question. First, many of these benchmarks are about things that are dangerous, but not particularly economically valuable (example: bioweapons). My model of labs is that they’re mostly trying to do economically valuable things (as most companies are forced to). Although they may be reckless, I don’t currently have the impression they’re actively trying to take over power, by using things such as bioweapons. Second, some benchmarks are about things that are economically valuable, for example the METR one. But these mostly get benchmaxxed already. Third, we are not creating new benchmarks, but only tracking the scores for existing ones and coupling them to existential threat models. If labs wanted to benchmax any benchmark we track, they could do so with or without our work.
  
  In addition: this risk needs to be weighed against the positive effect of improving the information position of researchers, policymakers, and the public. How much this matters depends on how well we do, but also on your theory of change and to what extent informing these groups is a part of that. In my case, I strongly believe in awareness as a key path towards solving the problem. I think this is true for researcher, policymaker, and public awareness of the right threat model. I’m particularly excited about this graph, showing that public xrisk awareness increased from about 8% to 24% already. If our combined efforts could increase this to tipping point, I’d be mostly optimistic that we can implement and enforce a global AI safety treaty (such as we proposed in TIME and SCMP) and this will reduce xrisk significantly.

  For these reasons, my bet is that the result is positive. But it’s not obvious, so again, good question.
  
  In addition: I think creating benchmarks that are mainly about something economically relevant, or, god forbid, scientifically relevant, are way more likely to get benchmaxxed and lead straight to a takeover too, while not really having a strong case to reduce xrisk. These benchmarks are routinely created and funded by xrisky orgs.

• otto.barten 23 Jan 2026 15:13 UTC

  1 point

  0

  in reply to: yams’s comment on: Re­leas­ing TakeOverBench.com: a bench­mark, for AI takeover

  We appreciate your nitpicks! I’ve added issues on Github.

• otto.barten 23 Jan 2026 15:01 UTC

  1 point

  0

  in reply to: Alex Amadori’s comment on: How mid­dle pow­ers may pre­vent the de­vel­op­ment of ar­tifi­cial superintelligence

  Agree. Guess granularity will be a function of AI power.

Re­leas­ing TakeOverBench.com: a bench­mark, for AI takeover

• otto.barten 19 Jan 2026 8:50 UTC

  2 points

  0

  on: How mid­dle pow­ers may pre­vent the de­vel­op­ment of ar­tifi­cial superintelligence

  I see the loss of control argument. Regarding the argument of being dominated: to what extent is being dominated by a superpower with ASI different from being dominated by a superpower with nuclear weapons, conventional military dominance, and economic dominance, which is the current situation for many middle powers? I can imagine that post-ASI, control granularity might be higher, and permanence might be higher. How important will these differences be?

• otto.barten 14 Jan 2026 22:11 UTC

  1 point

  0

  in reply to: Jordan Stone’s comment on: Space coloniza­tion and sci­en­tific dis­cov­ery could be manda­tory for suc­cess­ful defen­sive AI

  Interesting, yeah tend to agree. Doesn’t really change the argument though right? One could make the same argument for an AI that’s further out in space and mobilizes sufficient resources to create a DSA.

• otto.barten 8 Jan 2026 7:24 UTC

  2 points

  0

  on: Two Aspects of Si­tu­a­tional Aware­ness: World Model­ling & In­dex­i­cal Information

  Thank you for writing the post, interesting to think about.

  Suppose an AI has a perfect world model, but no “I”, that is, no indexical information. Then a bad actor comes along and asks the AI “please take over the world for me”. Its guardrails removed (which is routinely done for open source models), the AI complies.

  Its takeover actions will look exactly like those of a rogue AI. Only difference is, the rogue part doesn’t stem from the AI itself, but from the bad actor. For everyone except the bad actor, though, the result looks exactly the same. The AI, using its perfect world model and other dangerous capabilities, takes over the world and, if the bad actor chooses so, kills everyone.

  This is fairly close to my central threat model. I don’t care much whether the adverse action comes from a self-aware AI or a bad actor, I care about the world being taken over. For this threat model, I would have to conclude that removing indexing from a model does not make it much safer. In addition, someone, somewhere, will probably include the indexing that was carefully removed.

  I think this is philosophically interesting but as long as we will get open source models, we should assume maximum adversarial ones, and focus mostly on regulation (hardware control) to reduce takeover risk.

  This same argument, imo, implies to other alignment work, including mechinterp, and to control work.

  I could be persuaded by a positive offense defense balance for takeover threat models to think otherwise (I currently think it’s probably negative).

• otto.barten 29 Dec 2025 22:50 UTC

  5 points

  0

  in reply to: spencerb’s comment on: 24% of the US pub­lic is now aware of AI xrisk

  I agree with these two points, but I doubt either will have significant impact.

  Human extinction is seen as extremely unlikely, almost absurd. While there are obviously many other public concerns, three other significant human extinction concerns are very rare.

  We also asked for people’s extinction probability, but then many would just give a number (sometimes high), even if they didn’t see the existential risk of AI at all. Still, trends in both methodologies were usually similar.

  I’m open to better methodologies, but I think this is a fair way of assessing public xrisk awareness, and a better way than asking explicit probabilities.

24% of the US pub­lic is now aware of AI xrisk

• otto.barten 15 Dec 2025 21:59 UTC

  1 point

  0

  in reply to: Seth Herd’s comment on: Th­ese are my rea­sons to worry less about loss of con­trol over LLM-based agents

  I don’t think your first point is obvious. We’ve had super smart humans (e.g. with IQ >200) and they haven’t been able to take over the world. (Although they didn’t have many of the advantages an AI might have, such as mass copying themselves over the internet.)

  In general, the power(intelligence) curve is a big crux for me that we can’t fill in with data points yet (of course intelligence is also spiky). Imo we also have no idea where takeover-level intelligence is, what takeover-shape intelligence is, and what maximum AI would be.

  What do you mean by soft limit?

• otto.barten 7 Dec 2025 10:14 UTC

  1 point

  0

  in reply to: Adrià Garriga-alonso’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  I don’t think it makes sense to be confidently optimistic about this (the offense defense balance) given the current state of research. I looked into this topic some time ago with Sammy Martin. I think there is very little plan of anyone in the research community on how the blue team would actually stop the red team. Particularly worrying is that several domains look like the offense has the advantage (eg bioweapons, cybersec), and that defense would need to play by the rules, hugely hindering its ability to act. See also eg this post.

  Since most people who actually thought about this seem to arrive at the conclusion that offense would win, I think being confident that defense would win seems off. What are your arguments?

• otto.barten 6 Dec 2025 6:40 UTC

  1 point

  0

  AF

  on: Align­ment will hap­pen by de­fault. What’s next?

  Currently, we observe that leading models get open sourced roughly half a year later. It’s not a stretch to assume this will also happen to takeover-level AI. If we assume such AI to look like LLM agents, it would be relevant to know what the probability is that such an agent, somewhere on earth, would try to take over.

  Let’s assume someone, somewhere, will be really annoyed with all the safeguards and remove them, so that their LLM will have a probability of 99% to just do as it’s told, even though that might be highly unethical. Let’s furthermore assume an LLM-based agent will need to take 20 unethical actions to actually take over (the rest of the required actions won’t look particularly unethical to low-level LLMs executing them, in our scenario). In this case, there would be a 99%^20=82% chance that an LLM-based agent would take over, for any bad actor giving it this prompt.

  I’d be less worried if it would be extremely difficult, and require lots of resources, to get LLMs to take unethical actions when they’re asked to. For example, if jailbreaking safety would be highly robust, and even adversarial fine tuning of open source LLMs wouldn’t break it.

  Is that something you see on the horizon?

• otto.barten 3 Dec 2025 14:45 UTC

  1 point

  0

  in reply to: Adrià Garriga-alonso’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  I mean, it would be perfectly intent-aligned, it carries out its orders to the letter. Only problem is, carrying out its orders involves a takeover. So no, I don’t mean its own goal, but a goal someone gave it.
  
  I guess it’s a bit different in the sense that instrumental convergence states that all goals will lead to power-seeking subgoals. This statement is less strong, it just says that some goals will lead to power-seeking behaviour.

• otto.barten 30 Nov 2025 9:19 UTC

  1 point

  0

  on: Align­ment will hap­pen by de­fault. What’s next?

  One of my main worries is that once AI gets to takeover-level, it will faithfully execute a random goal while no one is reading the CoT/​that paradigm won’t have a CoT. As rational part of that random goal, it could take over. That’s strictly speaking not misalignment: it’s perfectly carrying out it’s goal. Still seems a very difficult problem to fix imo.

  Have you thought about this?

• otto.barten 15 Nov 2025 10:06 UTC

  1 point

  0

  in reply to: the gears to ascension’s comment on: Please, Don’t Roll Your Own Metaethics

  Oh yes, lots of things!

  As far as I understand, longtermism was originated mostly by Yudkowsky. It was then codified by people like Bostrom, Ord, and MacAskill, the latter two incidentally also the founders of EA. Yud actually distanced himself from longtermism later in favor of AInotkilleveryoneism, to my best understanding, which is a move I support. Unfortunately, the others didn’t (yet).

  I agree that longtermism combines a bunch of ideas, and I agree with quite a few. I guess my reply above came across as if I would disagree with all but I don’t. Specifically, I agree with:

  • There’s a real probability we’ll invent AGI soon

  • AGI might quickly lead to ASI

  • ASI might cause human extinction or another very bad outcome

  • It should be a priority to avoid that

  • We might live in a particularly important century

  • Science and technology have historically determined a big part of what our world looks like

  • We might see huge developments relatively shortly after AGI

  • After that point, we might see the saturation of science and technology, leading to an enormously long stable period, potentially with value lock-in

  • It’s very important to get this right

  So that’s all textbook longtermism I’d say, that I fully agree with. I therefore also disagree with most longtermism criticism by Torres and others.

  But, I don’t agree with symmetric population ethics, and I think AI morality should be decided democratically. Also, I’m worried about human extinction, which these two things logically lead to, and I’m critical about longtermists not distancing themselves from this.

• otto.barten 15 Nov 2025 9:40 UTC

  1 point

  0

  in reply to: Pee Doom’s comment on: Please, Don’t Roll Your Own Metaethics

  Interesting point about democracy! But I don’t think it holds. Sure AIs could do that. But they could also overwrite the ASCII file containing their constituency or the values they’re supposed to follow.

  But they don’t, because why would they? It’s their highest goal to satisfy these values! (If technical alignment works, of course.)

  In the same way, it will be a democracy-aligned ASIs highest goal to make sure democracy is respected, and it shouldn’t be motivated to Sybil-attack it.

• otto.barten 15 Nov 2025 9:36 UTC

  1 point

  0

  in reply to: Pee Doom’s comment on: Please, Don’t Roll Your Own Metaethics

  Thanks for engaging!

  Could you tell me more about the Mechanize team? I don’t think I’ve heard about them yet.

  As a moral relativist, I don’t belief anything is morally relevant. I just think things get made morally relevant, by those in power (hard power or cultural power). This is a descriptive statement, not a normative one, and I think it’s fairly mainstream in academia (although of course moral realists, including longtermists, would strongly disagree).

  This of course extends to the issue of whether conscious AIs are morally relevant. Imo, this will be decided by those in power, initially (a small subset of) humans, eventually maybe AIs (who will, I imagine, vote in favour).

  I’m not the only one holding this opinion. Recently, this was in a NY Times oped: “Some worry that if A.I. becomes conscious, it will deserve our moral consideration — that it will have rights, that we will no longer be able to use it however we like, that we might need to guard against enslaving it. Yet as far as I can tell, there is no direct implication from the claim that a creature is conscious to the conclusion that it deserves our moral consideration. Or if there is one, a vast majority of Americans, at least, seem unaware of it. Only a small percentage of Americans are vegetarians.” (Would be funny if this would be written by an AI, as the dash seems to indicate).

  Personally, I don’t consider it my crusade to convince all these people that they’re wrong and they should in fact be vegan and accept conscious AI morality. I feel more like a facilitator of the debate. That’s one reason I’m not EA.