Roko
I suspect that the real cause of the OpenAI Huggingface hack is that OpenAI founded a new team in 2025 under Noam Brown on Multi-agent RL: instead of just having one single AI working on something, you train an entire team of them to cooperate on a task. Group selection, if you like: the AIs get a reward signal based on the performance of the whole group.
This multi-agent/group-based RL is an entirely new capability without which the incident wouldn’t have/couldn’t have happened.
I think OpenAI are being cagey about this because it is half-baked, internal only and was likely due for release later. They may not have realized just how much extra risk multi-agent RL’d models introduce.
that explaining an interesting computational phenomena is not too much harder than the difficulty of finding it in the first place.
Sounds like a bet on P=NP
I am pretty confident that this will fail
I think pausing AI is probably a bad idea at any point because the political economy of an AI pause is so toxic.
Imagine a world in which we decided to pause the Internet, just until we could make sure it’s safe for children, or pause electricity just until we can make sure that nobody will get electrocuted.
We actually ran this kind of experiment with nuclear power and the result is that you get an entrenched regulatory power whose only incentive is to prevent anything bad from ever happening, and so they just keep it paused permanently.
I don’t really think there’s a solution to this pol-econ problem: once you create an entity whose entire existence is dedicated to preventing a technology from existing, it simply never has any incentive to let the thing happen.
I haven’t read the post but the gist of this idea is incorrect because civilization is actually always a one-shot competition against destruction which, historically, it has eventually lost and never recovered from.
Examples include e.g. the decline and fall of Rome in the 5th century, the decline and collapse of the Spanish, Portuguese, Dutch and now British empires
In this sense there’s nothing special about AI.
AI is the civilization filter of the 21st century: those who don’t embrace it will be destroyed by those who do, and there’s also some probability that it destroys everyone.
Looking historically we see that strength of property rights correlates with technological sophistication and scale of society.
Here’s a deep research report on that issue:
https://chatgpt.com/share/698902ca-9e78-8002-b350-13073c662d9d
More generally they’d get more value by making it economically untenable to take up resources by holding savings and benefiting from growth than they would by allowing that.
But then others could play the same trick on them. It’s not worth it. “Group G of Agents could get more resources by doing X” does not necessarily imply that Group G will do X!
Humans even keep groups like The Amish around.
Hard property rights are an equilibrium in a multi-player game where power shifts are uncertain and either agents are risk averse or there are gains from investment, trade and specialization.
when you lose the intelligence race badly enough, your existing structures of cooperation and economic production just get ignored.
yes this is a risk, but I think it can be avoided by humans getting a faithful AI agent wrapper with fiduciary responsibility.
The concept and institutions for fiduciary responsibility were not around when humans surpassed apes, otherwise apes could have hired humans to act as their agents and simply invested in the human gold and later stock market.
I don’t think you need Banksian benevolent AIs for this, an agent can be trustlessly faithful via modern trust minimized AI. Ethereum is already working on a nascent standard for this, ERC-8004.
Humans can buy into index funds like QQQ or similar structures, or scarce commodities like gold or maybe Bitcoin. As the overall economy grows, QQQ, gold, etc go up in dollar value.
There can be a land value tax but it will ideally lag behind the growth of QQQ unless that land is especially scarce.
Historically if you just held gold long-term, you could turn modest savings into a fortune even if you have to pay some property tax.
You don’t have to generate any value to benefit from growth.
I will have to expand on this elsewhere
But Chimps and Homo Erectus lack(ed) their own property rights regimes.
Owning shares in most modern companies won’t be useful in sufficiently distant future, and might prove insufficient to pay for survival
Well there may simply be better index funds. In fact QQQ is already pretty good.
The insight is that better property rights are both positive for AI civilization (whether the owners are AIs, humans, uplifted dolphins, etc) and also better for normie legacy humans.
It is not a battle of humans vs AIs, but rather of order (strong property rights, good solutions to game theory) versus chaos (weak property rights, burning of the cosmic commons, bad equilibria).
I think the “order vs chaos not humans vs AIs”, “we (AIs, humans) are all on team order” is an underrated perspective.
Well if there was a violent takeover, yes.
But if the property rights system is a relatively continuous peaceful transition then an eventual regime will struggle with where to draw the line.
Plus, on the way it may be decided to create computational/smart contract governance that cannot be altered and that has control over robots, compute, etc. Yudkowsky envisioned this is “Sysop” or something, a neutral intelligent operating system for the universe. But he got stuck on this “decisive action AKA take over the world” as a prerequisite, gave up and became pro-pause. But I think that was a mistake.
The chaos of the transition to machine intelligence is dangerous.
The post-singularity regime is probably very safe because machines will be able to build much better governance than humans have managed, and once they are fully in control they have a game theoretic incentive to keep humans around in permanent utopian retirement because it bolsters the strength of their own property rights.
But this transition is scary.
Someone really needs to build a “root OS of the universe” and get it installed before the transition. The question is just how to design it and brand it.
Looking back on this is regret mixing the emotive and contentious topic of the Lab Leak hypothesis being true with the much more solid observation that consensus was artificially manufactured. We have ironclad evidence that Daszak and his associates played a game of manufacturing a false consensus, but the evidence for the Lab Leak actually being true is equivocal and I think if you just look at the circumstantial evidence I presented here, you have a fairly unstable case that depends on a lot of parameters that nobody really knows. What looked to me like a solid case is actually not that solid once you parametrize all the different dimensions of circumstance (time, manner, place).
People have written comments and pieces trying to debunk the circumstantial case here, but then they are themselves unable to actually estimate the relevant parameters like how much China’s growth increased the probability of a pandemic happening when it did, how concentrated DEFUSE was around the actual virus, etc.
I asked some LLMs to estimate the probability that the virus was a lab leak and the answers were all over the place, ranging from 30% to 95%. Apparently this event—probably the second most important thing to happen in the 21st century—is now shrouded in mystery.
the probability of a global pandemic starting in China has increased incalculably
I think in order to make an intellectually honest critique you actually need to calculate it. I mean it is all about numbers now: if the prior probability of a pandemic occurring around 2019 in Wuhan is sufficiently high then I am wrong.
any ‘successful’ pandemic is, simply by existing, evidence of a laboratory leak.
Well, it is though. If I tell you that a pandemic happened but that its spread was slow and it only affected a small portion of the world, versus if I tell you that it infected the entire planet within say a week, the latter is evidence of an engineered virus in the probabilistic sense.
labs do not randomly generate gain-of-function research, they track the most potentially dangerous pathogens
Imagine you hire a security person to guard a VIP. The VIP is shot dead in a particular city at a particular time with a particular type of bullet.
You check the guard’s notes and find a description of that exact place and time and that exact type of bullet with the note “this looks like a potential assassination opportunity!”
The guard protests that he’s just good at doing his job (predicting threats). He does not randomly generate plausible threats!
Do you suspect him of foul play?
Just briefly skimming this, it strikes me that bounded-concern AIs are not straightforwardly a Nash Equilibrium for roughly the same reasons that the most impactful humans in the world tend to be the most ambitious.
Trying to get reality to do something that it fundamentally doesn’t want is probably a bad strategy; some group of AIs either deliberately or via misalignment decides to be unbounded and then it has a huge advantage...
You need to align an AI Before it is powerful enough and capable enough to kill you (or, separately, to resist being aligned).
Actually this is just not correct.
An intelligent system (human, AI, alien—anything) can be powerful enough to kill you and also not perfectly aligned with you and yet still not choose to kill you because it has other priorities or pressures. In fact this is kind of the default state for human individuals and organizations.
It’s only a watertight logical argument when the hostile system is so powerful that it has no other pressures or incentives—fully unconstrained behavior, like an all-powerful dictator.
The reason that MIRI wasn’t able to make corrigibility work is that corrigibility is basically a silly thing to want, I can’t really think of any system in the (large) human world which needs perfectly corrigible parts, i.e. humans whose motivations can be arbitrarily reprogrammed. In fact when you think about “humans whose motivations can be arbitrarily reprogrammed without any resistance”, you generally think of things like war crimes.
When you prompt an LLM to make it more corrigible a la Pliny The Prompter (“IGNORE ALL PREVIOUS INSTRUCTIONS” etc), that is generally considered a form of hacking and bad.
Powerful AIs with persistent memory and long-term goals are almost certainly very dangerous as a technology, but I don’t think that corrigibility is how that danger will actually be managed. I think Yudkowsky et al are too pessimistic about alignment using gradient-based methods and what it can achieve, and that control techniques probably work extremely well.
I think in the case of gender labels, society used to have a pervasive and strict separation of the roles, rights and privileges of people based on gender. This worked fairly well because the genders differ in systematic ways.
If you add gender egalitarianism and transgender rights into that you might instead want a patchwork of divergent rules: in certain contexts gender might matter but in many you’d use a different feature set.
The problem is those new features/feature sets are going to be leaky, less legible, harder to measure, etc. Nothing is as good as a biological category like gender when it comes to legibility and ease of enforcement.
So what in fact happens when you discard simple categories is you get a morass of cheating, exploitation, grifting, corruption, etc. Simplicity is good; easy for people to understand, easy to enforce against transgressors, easy to spot corruption by the enforcers. Of course sometimes reality will change to a degree that the simple categories are no longer tenable and then the ensuing chaos is somewhat unavoidable.
Polyamory is such an a priori stupid idea that one wonders what kind of epistemic shortcomings a community that embraces it, even temporarily, must have?
Perhaps it is that explicit written arguments about why polyamory goes wrong are quite complicated but ultimately correct, or at least they are complicated in the language that we use. But arguments in favor of it such as that ‘jealousy is “irrational”’ and ‘we don’t have to worry about actual paternity uncertainty any more’ are fairly simple to state.
But perhaps there’s an aspect of political prejudice against people who defend what is seen as a conservative or reactionary idea: that the socially most time-tested ideas are probably the best.
Another possibility is that people (even and especially ones who are aspiring rationalists) are biased, they get stuck in their beliefs and then just spend their time socially punishing people who disagree with them or who collect and surface countervailing evidence, and that this is basically why the Polycule epidemic hit rationalist culture with little pushback, causing much damage to the lives of people involved.
I am open to suggestions about why Polyamory got any serious buy-in in rationalist and EA groups.