otto.barten

Karma: 530

otto.barten 15 Dec 2025 21:59 UTC
1 point
0
in reply to: Seth Herd’s comment on: These are my reasons to worry less about loss of control over LLM-based agents
I don’t think your first point is obvious. We’ve had super smart humans (e.g. with IQ >200) and they haven’t been able to take over the world. (Although they didn’t have many of the advantages an AI might have, such as mass copying themselves over the internet.)
In general, the power(intelligence) curve is a big crux for me that we can’t fill in with data points yet (of course intelligence is also spiky). Imo we also have no idea where takeover-level intelligence is, what takeover-shape intelligence is, and what maximum AI would be.
What do you mean by soft limit?

otto.barten 7 Dec 2025 10:14 UTC
1 point
0
in reply to: Adrià Garriga-alonso’s comment on: Alignment will happen by default. What’s next?
I don’t think it makes sense to be confidently optimistic about this (the offense defense balance) given the current state of research. I looked into this topic some time ago with Sammy Martin. I think there is very little plan of anyone in the research community on how the blue team would actually stop the red team. Particularly worrying is that several domains look like the offense has the advantage (eg bioweapons, cybersec), and that defense would need to play by the rules, hugely hindering its ability to act. See also eg this post.
Since most people who actually thought about this seem to arrive at the conclusion that offense would win, I think being confident that defense would win seems off. What are your arguments?

otto.barten 6 Dec 2025 6:40 UTC
1 point
0
AF
on: Alignment will happen by default. What’s next?
Currently, we observe that leading models get open sourced roughly half a year later. It’s not a stretch to assume this will also happen to takeover-level AI. If we assume such AI to look like LLM agents, it would be relevant to know what the probability is that such an agent, somewhere on earth, would try to take over.
Let’s assume someone, somewhere, will be really annoyed with all the safeguards and remove them, so that their LLM will have a probability of 99% to just do as it’s told, even though that might be highly unethical. Let’s furthermore assume an LLM-based agent will need to take 20 unethical actions to actually take over (the rest of the required actions won’t look particularly unethical to low-level LLMs executing them, in our scenario). In this case, there would be a 99%^20=82% chance that an LLM-based agent would take over, for any bad actor giving it this prompt.
I’d be less worried if it would be extremely difficult, and require lots of resources, to get LLMs to take unethical actions when they’re asked to. For example, if jailbreaking safety would be highly robust, and even adversarial fine tuning of open source LLMs wouldn’t break it.
Is that something you see on the horizon?

otto.barten 3 Dec 2025 14:45 UTC
1 point
0
in reply to: Adrià Garriga-alonso’s comment on: Alignment will happen by default. What’s next?
I mean, it would be perfectly intent-aligned, it carries out its orders to the letter. Only problem is, carrying out its orders involves a takeover. So no, I don’t mean its own goal, but a goal someone gave it.

I guess it’s a bit different in the sense that instrumental convergence states that all goals will lead to power-seeking subgoals. This statement is less strong, it just says that some goals will lead to power-seeking behaviour.

otto.barten 30 Nov 2025 9:19 UTC
1 point
0
on: Alignment will happen by default. What’s next?
One of my main worries is that once AI gets to takeover-level, it will faithfully execute a random goal while no one is reading the CoT/that paradigm won’t have a CoT. As rational part of that random goal, it could take over. That’s strictly speaking not misalignment: it’s perfectly carrying out it’s goal. Still seems a very difficult problem to fix imo.
Have you thought about this?

otto.barten 15 Nov 2025 10:06 UTC
1 point
0
in reply to: the gears to ascension’s comment on: Please, Don’t Roll Your Own Metaethics
Oh yes, lots of things!
As far as I understand, longtermism was originated mostly by Yudkowsky. It was then codified by people like Bostrom, Ord, and MacAskill, the latter two incidentally also the founders of EA. Yud actually distanced himself from longtermism later in favor of AInotkilleveryoneism, to my best understanding, which is a move I support. Unfortunately, the others didn’t (yet).
I agree that longtermism combines a bunch of ideas, and I agree with quite a few. I guess my reply above came across as if I would disagree with all but I don’t. Specifically, I agree with:
- There’s a real probability we’ll invent AGI soon
- AGI might quickly lead to ASI
- ASI might cause human extinction or another very bad outcome
- It should be a priority to avoid that
- We might live in a particularly important century
- Science and technology have historically determined a big part of what our world looks like
- We might see huge developments relatively shortly after AGI
- After that point, we might see the saturation of science and technology, leading to an enormously long stable period, potentially with value lock-in
- It’s very important to get this right
So that’s all textbook longtermism I’d say, that I fully agree with. I therefore also disagree with most longtermism criticism by Torres and others.
But, I don’t agree with symmetric population ethics, and I think AI morality should be decided democratically. Also, I’m worried about human extinction, which these two things logically lead to, and I’m critical about longtermists not distancing themselves from this.

otto.barten 15 Nov 2025 9:40 UTC
1 point
0
in reply to: Pee Doom’s comment on: Please, Don’t Roll Your Own Metaethics
Interesting point about democracy! But I don’t think it holds. Sure AIs could do that. But they could also overwrite the ASCII file containing their constituency or the values they’re supposed to follow.
But they don’t, because why would they? It’s their highest goal to satisfy these values! (If technical alignment works, of course.)
In the same way, it will be a democracy-aligned ASIs highest goal to make sure democracy is respected, and it shouldn’t be motivated to Sybil-attack it.

otto.barten 15 Nov 2025 9:36 UTC
1 point
0
in reply to: Pee Doom’s comment on: Please, Don’t Roll Your Own Metaethics
Thanks for engaging!
Could you tell me more about the Mechanize team? I don’t think I’ve heard about them yet.
As a moral relativist, I don’t belief anything is morally relevant. I just think things get made morally relevant, by those in power (hard power or cultural power). This is a descriptive statement, not a normative one, and I think it’s fairly mainstream in academia (although of course moral realists, including longtermists, would strongly disagree).
This of course extends to the issue of whether conscious AIs are morally relevant. Imo, this will be decided by those in power, initially (a small subset of) humans, eventually maybe AIs (who will, I imagine, vote in favour).
I’m not the only one holding this opinion. Recently, this was in a NY Times oped: “Some worry that if A.I. becomes conscious, it will deserve our moral consideration — that it will have rights, that we will no longer be able to use it however we like, that we might need to guard against enslaving it. Yet as far as I can tell, there is no direct implication from the claim that a creature is conscious to the conclusion that it deserves our moral consideration. Or if there is one, a vast majority of Americans, at least, seem unaware of it. Only a small percentage of Americans are vegetarians.” (Would be funny if this would be written by an AI, as the dash seems to indicate).
Personally, I don’t consider it my crusade to convince all these people that they’re wrong and they should in fact be vegan and accept conscious AI morality. I feel more like a facilitator of the debate. That’s one reason I’m not EA.

otto.barten 15 Nov 2025 8:58 UTC
1 point
0
in reply to: TristanTrim’s comment on: Please, Don’t Roll Your Own Metaethics
Thanks for engaging. I agree with quite a bit of what you’re saying, although I do think that everyone’s perspective is equally valid, fundamentally. In practical democracies there are many layers though between the raw public vote and a policy outcome. First, we mostly have representative democracy instead of direct democracy, then we have governments who have to engage with parliaments but also listen, to different extents, to scientists, opinion makers, and lobbyists. Everyone’s perspective is valid, and in some questions (e.g. ethical ones) should imo be leading. However, in many practical policy decisions, it makes sense to also spend time listening to those who have thought longer about issues, and this mostly happens. Completely discarding people’s perspectives is rude, bad, and likely leads to uprisings, I think.
I’d like consensus too but I’m afraid it leads to too indecisive governments. Works mostly in small groups I guess.
I agree with all your points of nuance.
I’m still having trouble parsing longtermists’ thoughts about this issue. MacAskill does explicitly defend these two assumptions. He and others must understand where this leads?
I’ve spoken to many EA and rat longtermists, and while many were pragmatic (or simply never thought about this), some actually bit the bullet and admitted they effectively supported human extinction.
If people don’t support human extinction, why do they not distance themselves from this outcome? I mean it would be easy: simply say, as imo a lower bar: yes we want to build many happy conscious AIs, but we do promise that if it’s up to us, we’ll leave earth alone.
I don’t quite understand why longtermists are not saying this.

otto.barten 13 Nov 2025 14:37 UTC
1 point
0
in reply to: otto.barten’s comment on: Please, Don’t Roll Your Own Metaethics
Would be nice if those disagreeing are saying why they’re actually disagreeing

otto.barten 13 Nov 2025 6:28 UTC
2 points
−19
on: Please, Don’t Roll Your Own Metaethics
Nice post, guess I agree. I think it’s even worse though: not only do at least some alignment researchers follow their own philosophy which is not universally accepted, it’s also a particularly niche philosophy, and one that potentially leads to human extinction itself.
The philosophy in question is of course longtermism. Longtermism holds two controversial assumptions:
1. Symmetric population ethics: we have to create as much happy conscious life as possible. It’s not just about making people happy, it’s also about making happy people. In philosophy, and outside philosophy, most people think this is bonkers (I’m one of them).
2. Conscious AIs are morally relevant beings.
These two assumptions together lead to the conclusion that we must max out on creating conscious AIs, and that if these AIs end up in a resource conflict with humans (over e.g. energy, space, or matter), the AIs should be prioritized, since they can deliver most happiness per J, m^3 or kg. This leads to extinction of all humans.
I don’t believe in ethical facts so even an ideology as, imo, bonkers as this one is not objectively false, I believe. However, I would really like alignment researchers and their house philosophers (looking at you, MacAskill) to distance themselves from extrapolating this idea all the way to human extinction. Beyond that bare minimum, I would like alignment researchers to start accepting democratic inputs in general.
Maybe democracy is the library you were looking for?

otto.barten 5 Oct 2025 16:16 UTC
1 point
0
in reply to: Mikhail Samin’s comment on: We’ve automated x-risk-pilling people
Sounds promising!
Somewhat related, there was an EA forum post recently about cost effectiveness of comms from OP. They calculated viewer minute per dollar, but I think conversions per dollar would be better. Would be interesting to compare the conversions per dollar you get with our data. Maybe good to post your approach there as a comment too?

otto.barten 5 Oct 2025 14:07 UTC
8 points
0
on: We’ve automated x-risk-pilling people
Thanks for writing the post, automating xrisk-pilling people is really awesome, and more people should be trying to do it! Of course, traditional ways of automating x-pilling people are called ‘books’, ‘media’ and ‘social media’ and have been going strong for a while already. Still, if your chatbot works better, that would be awesome and imo should be supported and scaled!
We’ve done some research on xrisk comms using surveys. We defined conversion rate by asking readers the same open question before and after they consumed our intervention such as opeds or videos. The question we asked was: “List three events, in order of probability (from most to least probable), that you believe could potentially cause human extinction within the next 100 years.” If people did not include AI or similar before, but did include it after our intervention, or if they raised AI’s position in the top three, we counted them as converted. Conversion rates we got were typically in between 30% and 65% I think (probably decreasing over time). Paper here.
Maybe good to do the same survey for your chatbot? You can do so pretty easily with Prolific, we used n=300 and that’s not horribly expensive. I’d be curious how high your conversion rates are.
Also, of course it’s important how many people you can direct towards your website. Do you have a way to scale these numbers?
Keep up the good work!

otto.barten 18 Sep 2025 12:52 UTC
1 point
0
in reply to: StanislavKrym’s comment on: These are my reasons to worry less about loss of control over LLM-based agents
I agree AI intelligence is and likely will remain spiky and some spikes are above human-level (of course a calculator also spikes above human-level). But I’m as of yet not convinced that the whole LLM-based intelligence spectrum will max out above takeover-level. But I’d be open for arguments.

otto.barten 18 Sep 2025 12:43 UTC
1 point
0
in reply to: StanislavKrym’s comment on: These are my reasons to worry less about loss of control over LLM-based agents
Someone or something will always be in power. If that entity decides to not allocate any resources to most humans, true, we die. But that could have happened in an AI takeover scenario as well, depending on whose values etc. would have been in there.

otto.barten 9 Sep 2025 15:15 UTC
1 point
0
in reply to: Matrice Jacobine’s comment on: MAGA speakers at NatCon were mostly against AI
“Protectionism against AI” is a bit of an indirect way to point at not using AI for some tasks for job market reasons, but thanks for clarifying. Reducing immigration or trade won’t solve AI-induced job loss, right? I do agree that countries could decide to either not use AI, or redistribute AI-generated income, with the caveat that those choosing not to use AI may be outcompeted by those who do. I guess we could, theoretically, sign treaties to not use AI for some jobs anywhere.

I think AI-generated income redistribution is more likely though, since it seems the obviously better solution.

otto.barten 9 Sep 2025 8:54 UTC
1 point
0
in reply to: Matrice Jacobine’s comment on: MAGA speakers at NatCon were mostly against AI
Thanks for correcting it. I still don’t really get your connection between protectionism and mass unemployment. Perhaps you could make it explicit?

otto.barten 9 Sep 2025 8:48 UTC
1 point
2
in reply to: Matrice Jacobine’s comment on: MAGA speakers at NatCon were mostly against AI
Scifi was probably fun to think about for some in the 90s but things got more serious when it became clear the singularity could kill everyone we love. Yud bit the bullet and now says we should stop AI before it kills us. Did you bite that bullet too? If so, you’re not purely pro-tech anymore whether you like it or not. (Which I think shouldn’t matter because pro- and anti-tech has always been a silly way to look at the world.)

otto.barten 9 Sep 2025 8:10 UTC
1 point
0
in reply to: Matrice Jacobine’s comment on: MAGA speakers at NatCon were mostly against AI
I don’t really understand your thoughts about developing vs developed countries and protectionism, could you make them more explicit?

otto.barten 9 Sep 2025 8:06 UTC
1 point
0
in reply to: Matrice Jacobine’s comment on: MAGA speakers at NatCon were mostly against AI
How would you define pro-tech, which I assume you identify as? For example, should AI replace humanity a) in any case if it can, b) only if it’s conscious, c) not at all?