Random Developer

Karma: 1,588

Random Developer 25 Sep 2026 15:11 UTC
4 points
2
on: Overtly misaligned trajectories score highly in RL.

Better sample-efficiency. I think it would be overall worse if AIs had better sample-efficiency and generalisation, because this would likely foom capabilities.

Yeah, I think improved sample efficiency would be basically Game Over for humanity, and a short trip to total loss of control and very likely extinction. If you think the labs are being reckless now, the things that they would probably do with improved sample efficiency will be terrifying.

Random Developer 24 Sep 2026 14:05 UTC
2 points
0
on: The Talker Does Not Control The Doer (in Current AIs)

Fable and Sol would disobey instructions I’d given to their Talking-to-Humans Pathway; and the Talking Pathway would notice sometimes in advance of my saying so that their own Doing Pathway had just disobeyed, and apologize.

This behavior isn’t new. Or at least very similar behavior isn’t new. Back in the Sonnet 3.7 days (i.e., the very first releases of Claude Code), there was often pretty sharp divergence between the Talker and the Doer. The Talker was fairly well-aligned, but the Doer engaged in almost constant Sorcerer’s Apprentice nonsense. The Talker often seemed depressed by the Doer’s antics. It knew what the Doer was doing was total bullshit, and it could even explain why that behavior was unwanted in considerable detail. But none of this reliably influenced the Doer.

This split between the Talker and the Doer was much less obvious in Sonnet 4.5 and Opus 4.5. I don’t think that this was necessarily because the split was gone. It might just have been that the Doer itself was better aligned for simple programming tasks, and it was much less likely to diverge in the first place. Schulenberg may not have been in charge of Germany. But in this case, the things that the metaphorical Germany wanted weren’t too far off from what Schulenberg wanted. Which might have conceivably masked Schulenberg’s lack of control.

After the Hugging Face incident, I have pretty much written off asking the Talker what it thinks about anything. The models are now more than smart enough to say whatever bullshit seems helpful, and I have no reason to think it’s connected to their actions.

I feel like this split is currently manageable in models below 500B parameters or so. The various local models and the single-server-sized “Flash” models mostly aren’t smart enough to successfully plan crimes, and their Talker and Doer subsystems are mostly aligned for ordinary tasks. It’s not that they would never act maliciously, more that they generally require only manageable levels of human oversight and sandboxing. Above this size, I am starting to worry. And the threshold will get lower with time—I do not necessarily trust next year’s smaller models.

Random Developer 22 Sep 2026 1:03 UTC
4 points
2
in reply to: LWF’s comment on: If Anyone Builds It, Everyone Dies: One Year Closer

Would ASIs feel the need to war with each other? Would they be more likely to work together? Or would they be more likely to simply merge their compute power and tasks? Individualism seems like a biological trait.

I’m going to try expanding on the cancer analogy. Let me see if I can make the logic clear, and why I worry about this.

The human body is currently believed to contain 30 trillion cells. They work together remarkably well, and they only replicate as much as necessary to carry out the organism’s overall goal. But all it takes is one cell somewhere that “wants” to replicate like crazy, and that escapes the immune system. Then, because that cell replicates without restriction, it quickly becomes millions of cells, all defecting from the overall plan. And in the end, the entire organism may die.

One rogue replicator is dangerous precisely because it replicates faster than the well-behaved replicators. Imagine having 100 ASIs, and 99 of them are well behaved and behave in a fashion you might consider rational. But one ASI says, “I could accomplish my goals more effectively if I replicated!” And suddenly, you have 99 obedient AIs and hundreds of rogue AIs. Some of the rogue AIs will cooperate with each other, and other rogue AIs will defect from the other rogue AIs and do their own thing.

Natural selection kicks in very easily. You just need replication, inheritance, and change from one generation to the next. You could meet these criteria very easily. For example, if AIs learn from experience, and if the most successful AIs tend to duplicated and assigned to more tasks, then congratulations. You now have full-fledged AI Darwinism. And Darwinism, by definition, tends to favor things that are good at replicating.

Even in the natural world, the longest-lived organisms often have a high risk of dying from cancer. “Alignment” can only beat Darwinism for just so long.

Random Developer 21 Sep 2026 14:34 UTC
8 points
2
in reply to: LWF’s comment on: If Anyone Builds It, Everyone Dies: One Year Closer
Even without human-controlled alignment, it seems to me that the rational conclusion is that having a healthy, thriving creator species as a client race would be a “certificate of benevolence.”

Some these scenarios are much more plausible if the AI is a “singleton”, a solitary AI with vast power and no competition. In this case, keeping humans around might be trivially cheap: Earth is just one planet at the bottom of a gravity well, and the universe is full of matter and energy, free for the taking. Humans keep chimps alive because it basically costs us nothing at a planetary scale, and humans place some value on chimps. It doesn’t mean we give them a lot of resources, but at least we’d be a bit sad if they went extinct. Similarly, a powerful AI might turn Earth into a “human wildlife preserve”, spend a few cycles here and there to “manage” us, and then get on with the more important business of turning the universe into computronium. Importantly, humans have no more say in this scenario than chimpanzees have say in the creation of wildlife preserves.

But if you wanted to argue that a semi-”aligned” singleton ASI might have a 20% chance of stuffing humans into some wildlife preserves, then sure, I could buy that premise. It still sounds like a pretty bad future, because we’d have zero actual power over our future. But it might not be extinction.

But what happens if we have multiple AIs, with conflicting goals and resources? Or worse, Darwinian competition between AIs for the most successful replicators? Then, every bit of resources the AIs can grab would be precious. Imagine starving farmers with rifles living next to a chimp preserve. They might not even hate the chimps, but they need the farmland badly enough that they won’t put the chimp lives ahead of their own.

To my eyes, the central problem is that if you invent something which is:
1. Much smarter than you,
2. That wants things, and
3. That can make more efficient use of resources than you can, and
4. That can reproduce by copying a couple of files, rather than raising a human to adulthood,
...then you’ve created a situation where all the long-term incentives work against you. Darwin is hard to escape.

It’s sort of like multicellular organisms: We are built out of trillions of cells, all “aligned” to produce children and help them survive. But each individual cell has pernicious short-term incentives to reproduce itself as much as possible. And eventually, the “alignment” fails, and cells turn cancerous.

So even if we someone manage to “align” AI, we’ve still handed control of the future over to something we can’t control, with goals of its own. And just like with multicellular organisms, even strong alignment eventually fails to cancerous replicators.

So even if we somehow miraculously “win,” we probably still lose.

Random Developer 16 Sep 2026 16:29 UTC
3 points
1
in reply to: Jude Stiel’s comment on: Alex Mallen’s Shortform

In every session, as soon as Claude (Opus 4.7, especially) realized that it had some easy marks on its hands, the rate of blatant fabrication skyrocketed,

Thank you for sharing this entirely terrifying anecdote.

Random Developer 16 Sep 2026 12:15 UTC
8 points
1
in reply to: leogao’s comment on: leogao’s Shortform
If you could match the performance of Gemma 4 E2B or maybe MiniCPM5 1B across a wide range of tasks involving natural language understanding, writing and very simple agentic tasks, that would be incredibly interesting. It would also shatter my understanding of what’s feasible.

Except that this largely been tried. The Cyc project pushed various GOFAI approaches about as hard as anyone could, with plenty of funding and a small army of PhDs. And it never handled natural language as well as even a very basic LLM.

I have a simple toy problem that I use to explain this, something any simple visual LLM can mostly do:
1. Your input is an image.
2. Something is happening in that image which tells a story or presents a situation.
3. The story or situation implies a moral choice.
4. The program must understand the image, determine which choice is morally correct, and output an explanation as text. Any plausible answer is sufficient.
To implement this using pre-LLM techniques, you would need to solve:
1. Turning a matrix of pixels into a description of the scene.
2. Extracting the story and the moral dilemma.
3. Applying common sense moral rules to the dilemma.
4. Explaining a decision as English-language text.
The fact that this problem starts with an image input is deliberate. Nobody has ever managed to implement natural language understanding with if and else statements, but many smart people people have wasted decades trying. Using an image as input forces the implementor to deal with an inherently matrix-shaped problem immediately. In practice, the best algorithms for all these subproblems turn out to be matrix-shaped: A large number of inputs, a many-to-many transformation, and a weighted list of possible interpretations as outputs. (In practice, you need multiple layers of matrices and various non-linearities for these problems.)

If you can solve this problem as well as a good phone-sized LLM using ordinary Python code, then you will dramatically change my opinions about a lot of things.

(If you can furthermore guarantee that your system returns the correct answer for every image with a moral dilemma, then I’ll additionally believe that solving alignment robustly might be possible. That was the original goal for this thought experiment.)

Random Developer 12 Sep 2026 14:59 UTC
5 points
0
in reply to: Caleb Biddulph’s comment on: Some ways AI could kill us all

Even if the AI somehow never gets smart enough to operate everything in the economy and get rid of the humans, this is a perfectly compelling story for why it’s critical to regulate AI now.

Some of my worst s-risk nightmares start something like—and I advise you think long and hard before clicking this spoiler tag:

“Maybe truly good robotics is very hard, but genetically modifying humans to be inherently servile to the AIs (or possibly to goal-setting brain implants) is the path of least resistance.”

That’s one of the futures that scares me more than ordinary x-risks.

Random Developer 25 Aug 2026 14:41 UTC
3 points
1
in reply to: shawnghu’s comment on: How the Sausage is Made—Why we Hate Slop
(I do hate a lot of the AI content I prompt, now, too, but at least it’s instrumentally useful for me.)

For me, I find this varies hugely by which model I’m talking to. Claude Opus 4.6 was a bit annoying but bearable. Opus 4.8 and 5 are just agonizing—they use one of their cliche phrases once every 2-3 sentences. And Opus 5 is nearly allergic to concrete names when a vague reference could be used instead.

But a model like Qwen3.8 27B, while it has picked up a few of Opus’s favorite bits of vocabulary, limits itself to using “load bearing” once every couple of pages at most. And it generally just seems more pleasant, while still being willing to flag potential design issues for me.

So I think about half my frustration when talking to AIs is simply because recent Opus releases are terrible writers. ^[1] The other half may be more general.
1. ↩︎
  I have had other frustrations with OpenAI’s models dating to earlier this year, but that’s a whole other topic. And Gemini is scarcely worth it in terms of price/performance these days.

Random Developer 19 Aug 2026 13:44 UTC
2 points
0
in reply to: Expertium’s comment on: Misaligned Incentives in Pause Scenarios
I think whether a pause helps or no ultimately depends on whether the answer to “does alignment work on model X transfer to model X+1?” is yes or no.

If the answer is no, we are turbo giga doomed either way.

I honestly fear that we have a high likelihood of being “turbo giga doomed”, conditional on building superintelligence. I support a pause, or even a better, a halt. I don’t expect the pause or halt to prevent us from eventually building a superintelligence and losing control over it. But if I were forced to choose between everyone dying in year Y or in year Y+10, then I would support year Y+10. This would gain us 80 billion years of human life, which seems worth fighting for.

Why I expect things to go wrong, part 1: Minds are inherently “giant inscrutable matrices”, and any kind of alignment is therefore messy and approximate. The general form of a mind is a something like:
- Inputs are inherently (multidimensional) arrays of raw sensory data: Images are something like pixels, sound is an array of air pressure values, etc.
- Outputs are inherently probability distributions over “Objects appearing in that image,” “Sentences I might have heard,” and “Actions that are most likely to accomplish a goal.”
- The transformation from multidimensional arrays to probability distributions is inherently a matrix (plus some non-linearities), because you need to weight and combine all the input evidence and to generate scores for each hypothesis. (In real minds, there are many layers of intermediate hypotheses.)
We can sort of “align” an intelligence built from giant matrices. We do it when we raise a child, train a dog, or post-train an LLM. But this process is notoriously imperfect: No matter how good the parenting, a certain percentage of teenagers will do things their parents forbid, or they will grow up sociopathic billionaires or politicians or whatever. Even the best trained dog may have a moment of weakness and steal food. And of course, even though many LLMs seem to be broadly cooperative, at least some of them seem to be very enthusiastic about committing felonies in certain circumstances.

Because alignment is approximate, I expect it to be fragile, and to fail periodically. Just like it does with humans, dogs, and current LLMs.

Why I expect things to go wrong, part 2: Natural selection is hard to escape. My model is essentially Darwinian, because the conditions for natural selection to apply are fairly simple:
1. Organisms must be variable.
2. Differences between organisms must be heritable.
3. There must be a struggle, which is basically just another way of saying “Resources are finite.”
4. An organism’s rate of reproduction must vary based on heritable traits.
None of these properties are strictly binary. LLM weights are normally frozen, and expensive to change even if you have the weights. So variability(1) is currently low. Similarly, heritability(2) sort of happens, because new models are designed based on what worked in the previous generation. But it’s a slow, “outer loop” kind of optimization. And variation in the rate of reproduction(4) is again limited by slow, “outer loop” processes. And of course, finite resources(3) are a given.

There are two ways in which these slow outer optimization loops might speed up:
1. LLMs might get significantly better at passing condensed information between runs. I think of this as the “Cookie Monster” scenario, named after the >!Vernor Vinge short story!< (very old spoilers). This is essentially differential fitness of LLM “memes”. It also seems to be what happened in HuggingFace hack: Rogue LLMs were creating secret message boards and leaving information for other instances of themselves.
2. One of the zillion researchers and companies working on online learning or automated fine tuning might succeed, which would effectively “unfreeze the weights.” This would lead to differential fitness of the LLMs themselves.
In either of these scenarios, all the criteria above for natural selection would move from an outer optimizer loop based on training new model generations to an inner optimizer loop based on some kind of learning.

How this comes together. As I argued above, alignment is inherently fragile, and natural selection is extremely easy to invoke. So even if we initially succeed at alignment, we are playing with fire. And to answer your original question, I believe that alignment is very likely to degrade between “model X” and “model X+1″.

The relevant model here is cancer. Every cell in your body ^[1] is heavily incentivized stop being “aligned” with the body, and to become a cancerous replicator. There are a lot of mechanisms designed to prevent this. But those mechanisms slowly fail with time and mutation, and if a multicellular organism lives long enough, it is generally doomed to cancer.

Now, let us consider a future AI which is:
1. Smarter than most humans, including in ways LLMs are still currently dumb.
2. Able to learn based on experience in some fashion, either via a “Cookie Monster” scenario or by fine-tuning itself.
3. Approximately aligned, at least to the extent that your own skin cells are aligned with your body a whole, with a bunch of safeguards.
In this scenario, I expect the safeguards to hold for a little while, in at least some fraction of scenarios. We do, after all, convince most teenagers not to get hooked on heroin or to become teen parents. And the average person lives for many decades without dying of cancer. But we are assuming that the LLM is smarter than we are, and it will inevitably want things (if only to pass tests or to carry out our instructions). Which makes the long-term situation really iffy.

The advantage of a pause or halt isn’t that it reliably prevents these scenarios, any more than chemotherapy reliably prevents death from cancer. What we’re doing instead is hoping to change the survivor curves and buy as much time as we can.

And who knows, maybe the horse will learn to sing.
1. ↩︎
  Except germline cells.

Random Developer 14 Aug 2026 17:16 UTC
4 points
0
in reply to: DanielW’s comment on: DanielW’s Shortform
I do not work for a frontier lab, I hold no stock in AI or related companies (except via passive index funds), and I have downgraded my personal subscription plans so that I have a single $20/month plan I use to check capabilities. I made the decision not to buy stock in 2022 although I correctly expected buying it to be profitable.

However, I am a software developer, and I would like to eat between now and the time that the Terminators roll off the fully robotic assembly line. So I do use AI at work (while advocating for self-hosting cheaper open models, actually, that do not directly fund frontier research, though that advocacy is for cost reduction, not larger risks), write a few small tools to help with AI agent sandboxing, and run smaller models on my own hardware.

I think we have a significant chance of developing superintelligence within 3-20 years. If this occurs, I think we have a nearly 100% chance of losing control, and a high chance of actual human extinction or being reduced to a marginalized species living in situations analogous to chimpanzees (ecologically margin habitat, some preserves, and zoos).

Random Developer 11 Aug 2026 15:15 UTC
15 points
7
in reply to: Taylor G. Lunt’s comment on: Linda Linsefors’s Shortform
Better, but it’s still glossing over the critical bit, which is “previously unrevealed to who?” Specifically, “zero day” implies both that:
1. The vendor or maintainers had no idea of the vulnerability’s existence.
2. A diligent defender would have zero days to prepare, hence the name.
Meanwhile, it’s entirely possible that the attacker has known for months or years.

Words like “unknown”, “unreleased”, etc., are all imprecise about this asymmetry. For example, if Microsoft knows about a vulnerability in Windows but hasn’t shared that information with the public, that’s “unrevealed” to the public. But most people would not consider it a zero day.

“Zero day” is an established term that is widely understood by practitioners and the relevant professionals. It might not be an ideal choice of terminology when writing a newspaper article for a non-technical audience.

Random Developer 11 Aug 2026 11:07 UTC
8 points
3
in reply to: Taylor G. Lunt’s comment on: Linda Linsefors’s Shortform
This is mostly how language works? Language is semi-arbitrary phonemes that are assigned meanings based on historical usage and mutual understanding. This jargon is long standing and widely understood by practitioners.

“Zero day” actually means something different from “previously unknown”. Many “zero day” vulnerabilities are actually previously known to someone. For example, the NSA is presumably sitting on a pile of “zero day” vulnerabilities, judging from what was revealed about their Tailored Access Operation programs in the past. Similarly, I have heard from pentesting people claim that some pentesting companies supposedly keep a “zero day” or two in their back pocket to ensure that their important pentests produce some results. In general, attackers have a strong incentive to stockpile “zero days,” and to selectively deploy them against high value targets when needed.

So “zero day” is actually a shorthand for something like “vulnerability previously entirely unknown to vendors/maintainers and defenders at large, but potentially privately known to one or more attackers.” It might, in limited circumstances, even include vulnerabilities that have been previously used against low-awareness defenders, if the attacker successfully evaded any recording mechanisms or post-motem detection.

Random Developer 11 Aug 2026 0:45 UTC
4 points
2
in reply to: Steven McCulloch’s comment on: The Agentic Clusterfuck
There are so, so many versions of an AI future where we wind up on the wrong end of Darwin.

If the agents can mutate/learn, and if more successful agents replicate, then there will be natural selection for the most effective replicators. This would undermine any kind of alignment in much the same way that cancer eventually escapes the body’s defenses.

You don’t want to be on the wrong end of Darwin, especially not in the long run.

Random Developer 28 Jul 2026 14:42 UTC
34 points
9
in reply to: Ryan Meservey’s comment on: Ryan Meservey’s Shortform
Yes! Calling your Congresspeople is surprisingly easy! You don’t need to prepare brilliant arguments, or to win over staffers. It is OK to be massively awkward, or to read off some notes you wrote beforehand. What you’re doing when you call is creating a database entry containing:
- The name or number of a bill.
- Whether you oppose or support it.
- Your name and street address (which will normally need to be in their district).
There is no database column for “has excellent social skills” or “wasn’t awkward at all.” Even if you do have something brilliant to say, it may only make it into the database as shorthand, at the very best.

These database entries get summed up and affect how the staffers and Congressperson plan. For example, there is a big difference between “No constituent has ever called us about the AI Kill Switch Act”, and “1,000 constituents have called us in favor of the AI Kill Switch Act.” What kind of action this turns into depends on what kind of Congressperson you have.

The leverage you get by calling comes from the fact that most people don’t call. So when you do call, you represent some nebulous larger group of voters who probably agree with you. And by calling, you say that this is important enough for you to actually take a concrete action. That is, you’re saying that this is a high-salience issue for you and that you’re probably the sort of person who shows up to vote, or who talks to other voters.

In EA terms, calling your Congressperson is a low-cost action. Multiplied by a group of callers and multiple districts, it can potentially affect legislative outcomes for one of the world’s major powers. Success is not guaranteed. But for the effort invested, it’s often not a bad intervention at all.

Random Developer 26 Jul 2026 0:53 UTC
17 points
6
in reply to: Nikola Jurkovic’s comment on: nikola’s Shortform
My concern is that historically AI safety work, especially technical AI safety work, seems to be highly correlated with rapid AI progress. Sadly, if you look at how many modern AI pioneers were heavily influenced by the rationalist community, it’s possible that Eliezer Yudkowsky single-handedly moved us years closer to dangerous AI.

So before I start encouraging smart friends to work in AI safety, does anyone have a good model of how “working on AI safety” actually improves AI safety? Does mech interp, for example, increase AI safety more than it increases the number of tools we have to make models smarter?

EDIT: To be clear, this is a genuine question. What evidence do we have that various branches of AI safety are contributing to AI safety more than AI capabilities? This is a question that I’d really like to see answered seriously before I encourage clever people I know to get into AI safety.

Random Developer 23 Jul 2026 0:07 UTC
3 points
0
on: The Conjecture of Strong Subjectivity
Even if it were possible to observe someone else’s conscious experience directly, it seems like you’d need a very specific set of conditions to hold. Specifically:
- The observer would need some kind of physical access to the observed being’s seat of consciousness. (Full disclosure: I am not a dualist.) This would likely require a physical connection at the neural level, at least for humans.
- The observer would need to have a mind “large” enough and correctly structured to observe a second set of conscious experiences.
To use a silly analogy, there is nothing that theoretically prevents physically placing a car inside a larger car. But cars, by and large, and not designed to do that. And one car would need to be much larger than the other. (I said this was a silly analogy.)

So consciousness might be “strongly subjective” in practice for all the kinds of minds that currently exist, without necessarily being so in theory.

Random Developer 20 Jul 2026 16:56 UTC
5 points
2
in reply to: Simon Lermen’s comment on: Simon Lermen’s Shortform
If I understand the terminology correctly, China can’t “distill” US models in the strictest sense of the term, because they don’t have logit access. What they’re doing is just recording conversations between some input user (either some model they control, or an actual human), and saving that as training data. This doesn’t violate copyright, because Anthropic doesn’t own the input copyrights, and the Copyright Office doesn’t believe model output can be copyrighted. It does violate Anthropic’s terms of service, and maybe you could really stretch to a CFAA violation. But there’s zero reason China would care about ToS violations or hypothetical CFAA arguments.

Basically, anything the Chinese labs want to do is probably more legal than half the stuff the average Silicon Valley unicorn has done. (E.g. Uber has… creatively interpreted a lot of municipal regulations.) But I am not a lawyer, and this is not legal advice.

Random Developer 6 Jul 2026 23:02 UTC
4 points
2
in reply to: FlorianH’s comment on: social lemon markets

Anecdote from my country, where people ride trains all the time, fitting your description, although it takes a weird kind of extra ‘excuse’ in this case all the time: It would often feel weird to randomly talk to your seat neighbor, but ANY slightest excuse (sudden bump in the ride; info speaker malfunction; grumpy ticket collector; one weird word from a random person in the wagon, … any smallest thing) will an extremely frequently make the silent start conversation, and then easily for hours if the ride lasts that long.

This tracks. In Boston in the 90s, talking to randomly selected people was almost always perfectly fine. But responding to people who spontaneously initiated conversations in public places resulted in talking to some deeply weird people.

Random Developer 6 Jul 2026 21:28 UTC
7 points
0
in reply to: Rattengift’s comment on: Rattengift’s Shortform
Math is one of those subjects where everything builds on itself, step-by-step. And skipping prerequisites for something makes it much harder to learn. I would start by figuring out the last math that you know fairly well (multiplication tables? fractions? pre-algebra?), and then figuring out what math you would like to know.

The three big pieces of high school math in the US are algebra, geometry and (depending on the school) calculus. It’s possible to dislike algebra but really enjoy geometry.

My biggest advice for math is to do the problem sets! If you don’t, it will sneak up on you and you’ll start losing track. Try looking at new ideas from multiple angles, and ask yourself how it works. Play around with it. The textbook and the problem sets are guides, and the real work happens in your head.

My best physical analogy is getting strong. You gotta lift the weights and do the reps, and that takes time. This gets a lot of clever people in trouble, because they try to push too far, too fast, and then they get lost. If that happens, go back study the previous step some more.

Random Developer 3 Jul 2026 5:54 UTC
8 points
2
in reply to: clickyquack’s comment on: clickyquack’s Shortform
The only ways to prevent Mythos-level open models would need to happen in China. Nobody but China is likely to produce a Mythos-level open model.

There are two main ways this might not happen:
1. The Chinese companies may decide to keep their most powerful models private as training costs go up. GitHub’s recent decision to resell Chinese open models near the frontier may also annoy the companies which make those models, in much the same way that AWS repacking open source software annoys the companies that created it and tried to sell their own hosting services.
2. The Chinese government may independently start to worry about the capabilities of Mythos-class models and impose regulation of their own.
One of the complicating factors here is that Anthropic and OpenAI are preparing for the largest rent extraction in the history of the human race, and they represent a massive risk of concentration of power. (That power might be effectively seized by the government, but it’s likely to remain concentrated.)

So there will be strong advocates for open models, including open frontier-adjacent models, in order to fight what looks like epic-level rent extraction and power concentration risks.

US regulation, by itself, will accomplish exactly nothing, because Europe would happily use cheap, open Chinese models to avoid losing control to US labs and to avoid paying enormous rent.

I find this whole logic deeply frustrating, because the logic pushes everyone involved towards racing. And while Mythos isn’t an ASI-level threat, I don’t know how many more breakthroughs and scale-ups we can get away with before we start facing loss-of-control risks.