Roko

Karma: 6,336

Roko 26 Aug 2025 13:05 UTC
2 points
0
in reply to: Matthew Barnett’s comment on: The Problem

I’m not sure Roko is arguing that it’s impossible for capitalist structures and reforms to make a lot of people worse off

Exactly. It’s possible and indeed happens frequently.

Roko 17 Aug 2025 21:46 UTC
3 points
0
in reply to: Eli Tyre’s comment on: The Problem
I think you can have various arrangements that are either of those or a combination of the two.
Even if the Guardian Angels hate their principal and want to harm them, it may be the case that multiple such Guardian Angels could all monitor each other and the one that makes the first move against the principal is reported (with proof) to the principal by at least some of the others, who are then rewarded for that and those who provably didn’t report are punished, and then the offender is deleted.
The misaligned agents can just be stuck in their own version of Bostrom’s self-reinforcing hell.
As long as their coordination cost is high, you are safe.
Also it can be a combination of many things that cause agents to in fact act aligned with their principals.

Roko 17 Aug 2025 12:16 UTC
3 points
0
in reply to: Eli Tyre’s comment on: The Problem

It sure seems to me that there is a clear demarcation between AIs and humans, such that the AIs would be able to successfully collude against humans

I think this just misunderstands how coordination works.

The game theory of who is allowed to coordinate with who against whom is not simple.

White Germans fought against white Englishmen who are barely different, but each tried to ally with distantly related foreigners.

Ultimately what we are starting to see is that AI risk isn’t about math or chips or interpretability, it’s actually just politics.

Roko 14 Aug 2025 16:24 UTC
2 points
0
in reply to: CronoDAS’s comment on: The Problem
yes, but yet again, it was because of how Africans were not considered part of the system of property rights. They were owned, not owners.

Roko 14 Aug 2025 9:13 UTC
4 points
0
in reply to: Knight Lee’s comment on: The Problem
It’s mostly not because of altruism, it’s because we have a property rights system, rule of law, etc.

And you can have degrees of cooperation between heterogenous agents. Full atomization and Borg are not the only two options.

Roko 14 Aug 2025 0:30 UTC
3 points
0
in reply to: cousin_it’s comment on: The Problem

should’ve said “most”.

That’s just run-of-the-mill history though.

Roko 14 Aug 2025 0:23 UTC
2 points
0
in reply to: Knight Lee’s comment on: The Problem

But I’m obviously talking about a very different kind of system which is more Borg-like and less market-like.

but then you have to justify why a borg-like monoculture will actually be competitive, as opposed to an ecosystem of many different kinds of entity and many different game-theoretic alliances/teams that these diverse entities belong to.

Roko 14 Aug 2025 0:20 UTC
2 points
0
in reply to: CronoDAS’s comment on: The Problem

As a matter of historical fact, there are lots of examples of certain groups of people being systematically excluded from having property rights

yes. And so what matters is whether or not you, I or any given entity is or is not excluded from property rights.

It doesn’t really matter how wizzy and flashy and super AI is. All of the variance in outcomes, at least to the downside, is determined by property rights.

Roko 13 Aug 2025 23:57 UTC
3 points
1
in reply to: Buck’s comment on: The Problem

suppose there’s some information about the world that the AIs are able to glean through vast amounts of experience and reflection, and that they can’t justify except through reference to that experience and reflection. Suppose there are two AIs that make conflicting claims about that information, while agreeing on everything that humans can check.

Well the AIs will develop track records and reputations.

This is already happening with LLM-based AIs.

And the vast majority of claims will actually be somewhat checkable, at some cost, after some time.

I don’t think this is a particularly bad problem.

Roko 13 Aug 2025 23:56 UTC
2 points
0
in reply to: Buck’s comment on: The Problem
Well you have to consider relative coordination strength, not absolute.
In a human-only world, power is a battle for coordination between various factions.
In a human + AI world, power will still be a battle for coordination between factions, but now those factions will be some mix of humans and AIs.
It’s not clear to me which of these is better or worse.

Roko 13 Aug 2025 19:17 UTC
3 points
−1
in reply to: Buck’s comment on: The Problem
Well it depends on the details of how the AI market evolves and how capabilities evolve over time, whether there’s a fast, localized takeoff or a slower period of widely distributed economic growth.

This in turn depends to some extent on how seriously you take the idea of a single powerful AI undergoing recursive self-improvement, versus AI companies mostly just selling any innovations to the broader market, and whether returns to further intelligence diminish quickly or not.

In a world with slow takeoff, no recursive self-improvement and diminishing returns, AI looks a lot like any other technology and trying to artificially centralize it just enables tyranny and likely massively reduces the upside, potentially permanently locking us into an AI-driven police state run by some 21st Century Stalin who promised to keep us safe from the bad AIs.

Roko 13 Aug 2025 1:02 UTC
2 points
0
in reply to: CronoDAS’s comment on: The Problem
but this wasn’t a self-enriching conspiracy of lawyers

Roko 13 Aug 2025 1:01 UTC
1 point
−1
in reply to: Noosphere89’s comment on: The Problem

once our labor is useless and the AI civilization is completely independent of us, the incentives to keep us into a system of property rights don’t exist anymore

the same is true of e.g. pensioners or disabled people or even just rich people who don’t do any work and just live off capital gains.

Why does the property rights system not just completely dispossess anyone who is not in fact going to work?

Roko 13 Aug 2025 0:30 UTC
3 points
−4
in reply to: Buck’s comment on: The Problem
More generally, trying to ban or restrict AI (especially via the government) seems highly counterproductive as a strategy if you think AI risk looks a lot like Human Risk, because we have extensive evidence from the human world showing that highly centralized systems that put a lot of power into few hands are very, very bad.

You want to decentralize, open source, and strongly limit government power.

Current AI Safety discourse is the exact opposite of this because people think that AI society will be “totally different” from how human society works. But I think that since the problems of human society are all emergent effects not strongly tied to human biology in particular, real AI Safety will just look like Human Safety, i.e. openness, freedom, good institutions, decentralization, etc.

Roko 13 Aug 2025 0:23 UTC
3 points
0
in reply to: Knight Lee’s comment on: The Problem
I don’t think a lack of clones or immortality is an obstacle here.

If one powerful human could create many clones, so could the others. Then again the question arises of whether those clones would become part of society or not, and if so they would share our system of property rights.

Roko 13 Aug 2025 0:20 UTC
2 points
−2
in reply to: Noosphere89’s comment on: The Problem

since non-humans can’t really resist modern human civilization, there’s no reason for humans not to steal non-human property

but it’s not because they can’t resist, it’s because they are not included in our system of property rights. There are lots of humans who couldn’t resist me if I just went and stole from them or harmed them physically. But if I did that, the police would counterattack me.

Police do not protect farm animals from being slaughtered because they don’t have legal ownership of their own bodies.

Roko 13 Aug 2025 0:07 UTC
4 points
0
in reply to: Buck’s comment on: The Problem

I think your arguments here stop working when the AIs are wildly superintelligent. If humans can’t really understand what actions AIs are taking or what the consequences of those actions are, even given arbitrary amounts of assistance from other AIs who we don’t necessarily trust, it seems basically hopeless to incentivize them to behave in any particular way.

But before we get to wildly superintelligent AI I think we will be able to build Guardian Angel AIs to represent our individual and collective interests, and they will take over as decisionmakers, like people today have lawyers to act as their advocates in the legal system and financial advisors for finance. In fact AI is already making legal advice more accessible, not less. So I think this counterargument fails.

As far as ELK goes I think if you have a marketplace of advisors (agents) where principles have an imperfect and delayed information channel to knowing whether the agents are faithful or deceptive, faithful agents will probably still be chosen more as long as there is choice.

Roko 12 Aug 2025 23:58 UTC
2 points
0
in reply to: Buck’s comment on: The Problem

I think it’s plausible that these AIs will be instances of a few different scheming models. Scheming models are highly mutually aligned

Sure, that’s possible. But Eliezer/MIRI isn’t making that argument.

Humans have this kind of effect as well and it’s very politically incorrect to talk about but people have claimed that humans of a certain “model subset” get into hiring positions in a tech company and then only hire other humans of that same “model subset” and take that company over, often simply value extracting and destroying it.

Since this kind of thing actually happens for real among humans, it seems very plausible that AIs will also do it. And the solution is likely the same—tag all of those scheming/correlated models and exclude them all from your economy/company. The actual tagging is not very difficult because moderately coordinated schemers will typically scheme early and often.

But again, Eliezer isn’t making that argument. And if he did, then banning AI doesn’t solve the problem because humans also engage in mutually-aligned correlated scheming. Both are bad, it is not clear why one or the other is worse.

Roko 12 Aug 2025 23:43 UTC
4 points
2
in reply to: cousin_it’s comment on: The Problem

The elite, enabled by AI, can take these things from you if they find it profitable. Roko says it’s impossible,

No, I think that is quite plausible.

But note that we have moved a very long way from “AIs versus humans, like in terminator” to “Existing human elites using AI to harm plebians”. That’s not even remotely the same thing.

Roko 11 Aug 2025 14:39 UTC
2 points
0
in reply to: CronoDAS’s comment on: The Problem
But in all of these cases and basically all other empires, a coalition of people was required to take those resources AND in addition they violated a lot of property rights too.

Strengthening the institution of property rights and nonviolence seems much more the thing that you want over “alignment”.

It is true that you can use alignment to strengthen property rights, but you can also use alignment to align an army to wage war and go violate other people’s property rights.

Obedience itself doesn’t seem to correlate strongly (and may even anti-correlate) with what we want.