Random Developer

Karma: 458

Random Developer 3 Oct 2025 20:51 UTC
6 points
3
on: We’ve automated x-risk-pilling people
Just for fun, I’ve been arguing with the AI that “We’re probably playing Russian roulette with 5 chambers loaded, not 6. This is still a very dumb idea and we should stop.” The AI is very determined to convince me that we’re playing with 6 chambers, not 5.

The primary outcomes of this process have been to remind me:
- Why I don’t “talk” to LLMs about anything except technical problems or capability evals. “Talking to LLMs” always winds up feeling like “Talking to the fey”. It’s a weird alien intelligence with unknowable goals, and mythology is pretty clear that talking to the fey rarely ends well.
- Why I don’t talk to cult members. “Yes, yes, you have an answer to everything, and a completely logical reason why I should believe in our Galatic Overlord Unex. But my counter argument is: No thank you, piss off.^[1]”
- Ironically, why I really should be concerned about LLM persuasiveness. The LLM is more interested in regurgitating IABIED at me than it is in convincing me. And yet it’s still pretty convincing. If it were better at modeling my argument, it could probably be very convincing.
So an impressive effort, and an interesting experiment. Whether it works, and whether it is a good thing, are questions to which I don’t have an immediate answer.
1. ↩︎
  G.K. Chesterton, in Orthodoxy: “Such is the madman of experience; he is commonly a reasoner, frequently a successful reasoner. Doubtless he could be vanquished in mere reason, and the case against him put logically. But it can be put much more precisely in more general and even æsthetic terms. He is in the clean and well-lit prison of one idea: he is sharpened to one painful point. He is without healthy hesitation and healthy complexity.”

Random Developer 3 Oct 2025 13:53 UTC
1 point
0
in reply to: niplav’s comment on: niplav’s Shortform
Another potential complication is hard to get at philosophically, but it could be described as “the AIs will have something analogous to free will”. Specifically, they will likely have a process where the AI can learn from experience, and resolve conflicts between incompatible values and goals it already holds.

If this is the case, then it’s entirely possible that the AI’s goals will adjust over time, in response to new information, or even just thanks to “contemplation” and strategizing. (AIs that can’t adjust to changing circumstances and draw their own conclusions are unlikely to compete with other AIs that can.)

But if the AI’s values and goals can be updated, then ensuring even vague alignment gets even harder.

Random Developer 3 Oct 2025 13:45 UTC
10 points
3
on: Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most “classic humans” in a few decades.
I think a lot about Homo erectus, which went extinct a bit over 100,000 years ago. They were quite smart, but they were ultimately competing for resources with smarter hominids. Even the chimps and the gorillas mostly survive in tiny numbers, because they didn’t compete too directly with our ancestors, and because we’d be sad if the last ones went extinct. So we make a half-assed attempt to save them.

In the medium-term, intelligence is really valuable! The species that can (let’s say) win a Nobel Prize every six months of GPU time, and copy itself in seconds, is going to rapidly win any evolutionary competition. The world must have been pretty bewildering for Homo erectus towards the end. When the ancestors of the Neaderthals and modern humans showed up, they brought better tools, and much better communication and coordination.

Or to put it another way, the family dog may be loved and pampered, but nobody asks it whether it wants to be spayed, or if the family should buy a new house. The Homo sapiens all sit around a table and make those decisions without the dog’s input. And maybe the only way to put food on the table is to move to the city, and the only good apartment choices forbid pets.

I don’t think the near-term details matter nearly as much as people think. Takeoff in 5 months, 5 years, or 20 years? Incomprehensible alien superintelligences, or vaguely friendly incomprehensible superintelligences who care about us about as much as we care about chimps? A single AI, or competing AIs? The net result is that humans aren’t making the decisions about our future. “Happy humans” have just become a niche luxury good for things that can run rings around us, with their own goals and problems. And those things, since they can replicate and change, will almost certainly face evolutionary forces of their own.

I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?

I see superintelligence a lot like a diagnosis of severe metastatic cancer. It’s not whether it’s going to kill you, it’s a question of how many good years of life you can buy before the end. I support an AI halt, because every year we stay halted buys humans more years of life. I suspect that “alignment” is as futile as Homo erectus trying to “align” Homo sapiens. It’s not completely hopeless—dogs did manage to sort of “align” humans—but I think the absolute best-case scenario is “humans as house pets, with owners of varying quality, likely selecting us to be more pleasing pets.” Which, thank you, but that doesn’t actually sound like success? And all the other possibilities go rapidly downhill from there. And Eliezer would argue that the “house pets” solution is a mirage and he may be right.

I’d prefer to put off rolling those dice as long as we can.

Random Developer 30 Sep 2025 14:15 UTC
3 points
0
in reply to: Raemon’s comment on: Buck’s Shortform
Thinking about the advisor’s comments:

And, from our understanding of your political goals, “treaty” is probably more like the thing you actually want. But, either is kinda reasonable to say, in this context.

What I understand Yudkowsky and Soares want might be summarized as something like:
- “Large numbers of GPUs should be treated like large numbers of uranium gas centrifuges.”
- “Publishing details of certain AI algorithms should be treated like publishing detailed engineering guidelines for improving the yield of a nuclear device.”
- ~~“Researching certain kinds of AI algorithms should be treated like doing gain of function research into highly contagious airborne Ebola.”~~ Actually, we probably don’t take bio threats nearly as seriously as we should.
The thing they want here includes a well-written treaty. Or an “international agreement.” If you buy their assumptions, then yes, you would want to lay out bright lines around things like data center capacity and monitoring, chip fabs, and possibly what kind of AI research is publishable.

But nuclear deterence also has quite a few other moving parts beyond the international agreements, including:
- The gut knowledge of the superpowers that if they screw this up, then their entire civilization dies.^[1]
- A tense, paranoid standoff between the key players.^[2]
- A system of economic sanctions strongly backed by major powers.
- Mysterious bad things happening to uranium centrifuges.
- Quiet conversations between government officials and smart people where the officials say things like, “Pretty please never mention that idea again.”^[3]
The key point in all of these circumstances is that powerful people and countries believe that “If we get this wrong, we might die.” This isn’t a case of “We want 80% fewer of our cities to be blown up with fusion bombs”. It’s a case of “We want absolutely none of our cities blown up by fusion bombs, because it won’t stop with just one or two.”

And so the rule that “You may only own up to X uranium gas centrifuges” is enforced using multiple tools, ranging from treaties/agreements to quiet requests to unilateral exercises of state power.
1. ↩︎
  Possibly while singing “Duck and Cover”. Which is actually decent advice for a nuclear war. Think of a nuclear explosion as a cross between a tornado and a really bright light that kills you. Getting away from a window and under a desk is not the worst heuristic, and even simple walls provide some shielding against gamma radiation. Sadly, this probably doesn’t work against SkyNet, no matter what the meme suggests. But an entire generation of children saw these videos and imagined their deaths. And some of those people still hold power. When the last of them retire, nuclear deterence will likely weaken.
2. ↩︎
  “The whole point of a Doomsday machine is lost if you keep it a secret!”
3. ↩︎
  This is paraphrased, but it’s from a real example.

Random Developer 30 Sep 2025 1:11 UTC
8 points
0
in reply to: ryan_greenblatt’s comment on: Daniel Kokotajlo’s Shortform
You should consider copying over the canary strings and GUIDs from the original page alongside these quotes.

Random Developer 29 Sep 2025 23:04 UTC
19 points
13
on: If Drexler Is Wrong, He May as Well Be Right
[I’m going to address several technologies you mentioned separately.]

Diamondoid Nanotech. One very old counterargument to Drexler’s “diamond phase” nanotech was Drexler just didn’t understand how physics works on that scale. One of the more detailed versions of this argument was put forth back in the day in Soft Machines, which argued that Brownian motion and self-assembly fundamentally make sense down at that scale.

Ever since the 90s, I have been keeping an eye on followups to Drexler’s work, fearing that this might be an especially dangerous route to AI. In particular, I watched the research results of Freitas and Merkle into “diamondoid mechanosythesis.” This project struggled for a long time and eventually petered out, with the conclusion that diamondoid surfaces were really nasty to work with (as many of the early material science critics had argued!).

EDIT: For more, see this detailed post.

I’m not saying that a superintelligence couldn’t make something like this work. But I suspect it would find a much better route than Drexler proposed. One of the things I apperciated the most about IABIED was that Eliezer dropped his long-standing focus on exotic nanotech as a primary threat vector.

This still leaves the other two possibilities that you suggested.

Synthetic biology. This is undoubtedly fiendishly difficult. But AlphaFold showed that protein folding, one of the single most difficult problems, was far easier than anyone expected. And if the Soft Machines argument is correct, then synthetic biology has the advantage of “going with the grain” of physics at that scale. The biggest drawback to synthetic biology I can think of, from an AI’s perspective? It doesn’t offer any really obvious ways to build GPUs. But maybe the AI is smarter than I am, or can construct a mixed liquid/solid biology that gets it there. I wouldn’t bet my entire future against this possibility.

Robotic factories. Yup, I fully expect this would work. One advantage of robotic factories is that almost everyone is smart enough to notice the robotic security guards and to put two and two together and get “SkyNet.”

Of course, if the easiest way to replace humans entirely is to build a complete robotic supply chain, then I expect AI alignment would appear to succeed amazingly well on the very first try. I would also expect the AI to immediately become everyone’s best friend, and to explain to venture capitalists and governments the amazing possibilities of robotic factories.

Once the robotic factories are capable of operating 100% human-free, that’s when we finally get to learn whether the AI is actually aligned! Hint: If the first thing off the assembly line is a Terminator, then you failed at alignment quite a few steps back. And the AI bamboozled you into giving it power by making sweeping promises.

So in the bigger scheme of things, you’re right. Any technology that allows reliable self-replication without humans is a giant risk to our future. And there are probably many different ways to get there. But some of the tactical details change depending on whether an LLM can build self-replicating computronium in a closet, or whether it needs to build mines and factories and ships. If an AI is only weakly superhuman and it has no better tools than robot factories, then you might get two or three shots at alignment. But you’d still need to use those opportunities, which is itself a difficult coordination problem. Especially if the AI is already whispering in the ears of leaders.

Random Developer 29 Sep 2025 20:33 UTC
17 points
8
in reply to: 1a3orn’s comment on: I have decided to stop lying to Americans about 9/11

I mean, when I ask myself what percentage of America would rejoice if a great disaster (say, 3 Gorges Dam just spontaneously collapses) were to befall China tomorrow, my gut says it is a reasonably substantial number.

When I read that, the image made me choke. Dam breaks are terrifying things, and I wouldn’t wish that fate on my enemy. Fuck. That’s a whole lot of people drowning under terrifying circumstances. And the Chinese people are not my enemy. I also remember being heartbroken about the stories of families in Wuhan at the start of 2020.

I’m sure some people in the US would celebrate bad things happening to China, of course. We do have our fair share of assholes, just like everywhere else.

(Now, I’m not claiming to be a saint. If someone who abuses dogs gets bit by a dog, a part of me will be like, “Well, what did you expect?” But it would feel wildly unfair to generalize that attitude to large numbers of ordinary people just because they live in a certain country.)

Random Developer 29 Sep 2025 16:47 UTC
8 points
11
in reply to: Mateusz Bagiński’s comment on: Mateusz Bagiński’s Shortform
The main (/most salient) disagreement I can see at the moment is the authors’ expectations of value-strangeness and maximizeriness of superintelligence; or rather, I am much more uncertain about this.

The metaphor I use is Russian roulette, where you have a revolver with 6 chambers, 1 loaded with a bullet. The gun is pointed at the head of humanity. We spin, and we pull the trigger.
- Various AI lab leaders have stated that we are playing with 1 (or 1.5) chambers loaded. Dario Amodei at Anthropic is the most pessimistic, I believe, giving us a 25% chance that AI kills us?
- Eliezer and Nate suspect that we are playing with approximately 6 out of 6 chambers loaded.
- Personally, I have a long and complicated argument that we may only be playing with 4 or 5 chambers loaded!
- You might feel that we’re only playing with 2 or 3 chambers loaded.
The wise solution here is not to worry about precisely how many chambers are loaded. Even 1 in 6 odds are terrible! The wise solution is to stop playing Russian roulette entirely.

Random Developer 29 Sep 2025 0:46 UTC
18 points
2
in reply to: robo’s comment on: Tomás B.’s Shortform
As a former robotics developer, I feel the bitter lesson in my bones. This is actually one of the points I plan to focus on when I write up the longer version of my argument.

High-quality manual dexterity (and real-time visual processing) in a cluttered environment is a heartbreakingly hard problem, using any version of GOFAI techniques I knew at the time. And even the most basic of the viable algorithms quickly turned into a big steaming pile of linear algebra mixed with calculus.

As someone who has done robotics demos (and who knows all the things an engineer can do to make sure the demos go smoothly), the Figure AI groceries demo still blows my mind. This demo is well into the “6 impossible things before breakfast” territory for me, and I am sure as hell feeling the imminent AGI when I watch it. And I think this version of Figure was an 8B VLLM connected to an 80M specialized motor control model running at 200 Hz? Even if I assume that this is a very carefully run demo showing Figure under ideal circumstances, it’s still black magic fuckery for me.

But it’s really hard to communicate this intuitive reaction to someone who hasn’t spent years working on GOFAI robotics. Some things seem really easy until you actually start typing code into an editor and booting it on actual robot hardware, or until you start trying to train a model. And then these things reveal themselves as heartbreakingly difficult. And so when I see VLLM-based robots that just casually solve these problems, I remember years of watching frustrated PhDs struggle with things that seemed impossibly basic.

For me, “fix a leaky pipe under a real-world, 30-year-old sink without flooding the kitchen, and deal with all the weird things that inevitably go wrong” will be one of my final warning bells of imminent general intelligence. Especially if the same robot can also add a new breaker to the electrical panel and install a new socket in an older house.

Random Developer 28 Sep 2025 16:15 UTC
19 points
6
in reply to: Tomás B.’s comment on: Tomás B.’s Shortform
I don’t actually buy this argument, but I think it’s a very important argument for someone to make, and for people to consider carefully. So thank you to Dwarkesh for proposing it, and to you for mentioning it!

I’ve been writing up a long-form argument for why “Good Old Fashioned AI” (GOFAI) is a hopeless pipedream. I don’t know if that would actually remain true for enormous numbers of superintelligent programmers! But if I had to sketch out the rough form of the argument, it would go something like this:
- Your inputs are all “giant, inscrutable matrices”, or rather n-dimensional tensors. Sound, images, video, etc. If you think I’m being unfair calling images “inscrutable tensors”, please take a 4k image as a 2D array of RGB pixels, and write me a Lisp function that counts all the palm trees.
- Your outputs are all probability distributions.
- The function from your inputs to your outputs is inevitably going to be a giant, inscrutable tensor, probably with a bunch of extra complications stacked on top. (Like ReLU or neural firing patterns.)
- Also, the world is vast, categories are fuzzy, and there will always be surprises outside of your training set.
Clay Shirkey summarized a similar set of issues in the context of the Semantic Web:

Anyone who has ever been 15 years old knows that protestations of love, checksummed or no, are not to be taken at face value. And even if we wanted to take love out of this example, what would we replace it with? The universe of assertions that Joe might make about Mary is large, but the subset of those assertions that are universally interpretable and uncomplicated is tiny.

So I would argue that GOFAI fails because it simplifies its map of reality to eliminate all the giant tensors, but in doing so, it oversimplifies the inherent slipperiness of problems like “Count the palm trees in this giant pile of numbers” or really any other interesting question about anything.

So my guess is that a giant army of superintelligent programmers could (at enormous cost) build an LLM-like chatbot by hardcoding vast amounts of world knowledge in a way that another superintelligence could understand. But this same chatbot would be helpless before any adversary who could create “out of the training distribution” situations. The very flexibility required to reason about fuzzy-but-important ideas like “someone is screwing with me in a novel way” would require a vast and complex world view that extrapolates well to unfamiliar situations. And that function from inscrutable input tensors to output probability distributions would itself be a giant, inscrutable tensor, plus a bunch of complications that made it harder to understand.

Random Developer 28 Sep 2025 13:49 UTC
3 points
2
on: %CPU Utilization Is A Lie
There is another factor that is often relevant here, which is queuing theory. There’s a famous chart (scroll down here), plotting “Percent of capacity used” on the x-axis, and “Expected wait time” on the y-axis. Wait time is approximately flat until the system reaches 70-80% load, at which point the wait time soars. This is used to decide things like, “How many ticket windows do we need to open at the amusement park?” But it also matters a lot for web servers and for many batch-processing systems.

A system approaching 70% capacity may see dramatic performance degradation as a result of a small load increase. Not all systems behave like this, but many systems do meet the necessary criteria to degrade in this way.

When designing monitoring systems, I often switch the load graph’s color to yellow around 60%, and to red by 80%. If your database’s CPU or disks are hitting 70% utilization, your system may be about to fall off a cliff.

Running systems near 100% utilization with acceptable wait times often requires prioritization, fairness constraints, reserving the last bit of capacity for latency-sensitive requests, etc. This can involve years of engineering effort to tune in the hardest cases. This is one reason the cloud is popular: When you hit 70% utilization, or when your wait times start to climb, you can just spin up another server instead of doing heavy engineering to squeeze out the last 15-20% of capacity without degrading latency.

Random Developer 24 Sep 2025 13:15 UTC
3 points
0
in reply to: Katalina Hernandez’s comment on: The Problem with Defining an “AGI Ban” by Outcome (a lawyer’s take).
To be clear, I have never been an actual national security person! But once upon a time, my coworkers occasionally needed to speak with non-proliferation people. (It’s a long story.)

Some of the arguments you raised (in connection to your national security experience) reminded me of his Mutual Assured AI Malfunction deterrence framework. I’m curious to know your thoughts!

I don’t know if that specific deterence regime is workable or not, but it’s a good direction to think about. One difference from nuclear weapons is that if you’re the only country with nukes, then you can sort of “win” a nuclear war (like happened at the end of WW2). But being the only country with an incomprehensible alien superintelligence is more like being the only country with a highly infectious airborne Ebola strain. Actually using it is Unilaterally Assured Destruction, to coin a term.

But let me make one last attempt to turn this intuition into a policy proposal.
1. Key decision makers need to realize in their gut that “building an incomprehensible alien superintelligence that’s really good at acting” is one of those things like “engineering highly infectious airborne Ebola” or “allowing ISIS to build megaton fusion weapons.” Unfortunately, the frontier labs are very persuasive right now, but the game isn’t over yet.
2. If the primary threat involves scaling, then we would need to control data centers containing more than a certain number of GPUs. Large numbers of GPUs would need to be treated like large numbers of gas centrifuges, basically. Or maybe specific future types of GPUs will need tighter controls.
3. If the primary or secondary threat involves improved algorithms, then we may need to treat that like nuclear non-proliferation, too. I know a physics major who once spoke to the aforementioned national security people about nuclear threats, and he asked, “But what about if you did XYZ?” The experts suddenly got quiet, and then they said, “No comment. Also, we would really appreciate it if you never mentioned that again to anybody.” There are things at the edges of the nuclear control regime that aren’t enforceably secret, but that we still don’t want to see posted all over the Internet. Some of the enforcement around this is apparently handled by unofficially explaining to smart people that they should pretty please Shut The Fuck Up. Or maybe certain algorithms need to be classified the way we classify the most important nuclear secrets.
4. It’s likely that we will need an international deterence regime. There are people with deep expertise in this.
One key point is that nobody here actually knows how to build a superintelligence. What we do have is a lot of people (including world class experts like Geoffrey Hinton) who strongly suspect that we’re close to figuring it out. And we have multiple frontier lab leaders who have stated that they plan to build “superintelligence” this decade. And we have a bunch of people who have noticed that (1) we don’t remotely understand how even current LLMs work, and (2) the AI labs’ plans for controlling a superintelligence are firmly in “underpants gnome” territory. And also even their optimistic employees regularly admit they’re playing Russian roulette with the human race. You don’t need deep knowledge of machine learning to suspect that this is the setup for a bad Hollywood movie about hubris.

But key details of how to build superintelligence are still unknown, happily. So it’s hard to make specific policy prescriptions. Szilard and Einstein could correctly see that fission was dangerous, but they couldn’t propose detailed rules for controlling centrifuges.

I do suspect that corporate legal compliance expertise will play a key role here! And thank you for working on it. We’ll need your expertise. But legal compliance can’t be the only tool in our toolkit. If we tried to enforce nuclear nonproliferation the way we try to enforce the GDPR, we’d probably already be dead. You will need buy-in and back-up of the sort that upholds nuclear non-proliferation. And that’s going to require a major attitude change.

(But I will also hit up my lawyer friends and see if they have more concrete advice.)

Random Developer 24 Sep 2025 2:10 UTC
3 points
1
on: We Support “If Anyone Builds It, Everyone Dies”
If we manage to build an AI that (1) possesses current LLM abilities, and (2) meets or exceeds the competence of a bright 12-year-old in other abilities, then I give us an 80+% chance of takeover by a superintelligence. After that, I expect the superintelligence to make all the important decisions about the future of humanity, in much the same way that humans make all the important decisions about the future of stray dogs. Euthanasia? Sterilization? A nice human rescue program and house-training? It won’t be our call. And since the AI probably pretty damn alien, we probably won’t like its call.

So I endorse, “If anyone builds it, then everyone dies. Or maybe it you’re incredibly lucky, you might just get spayed and lose all control over your life.” I don’t see this as a big improvement.

As a general rule, I feel like building an incomprehensible alien superintelligence is a very dumb move, for reasons that should be painfully self-evident to anyone who gives it 10 seconds of genuine thought.

Random Developer 23 Sep 2025 18:15 UTC
1 point
0
in reply to: jacquesthibs’s comment on: jacquesthibs’s Shortform

Ultimately, you’d better have a backup plan for superalignment that isn’t just, “we’ll stop if we catch the AIs being deceptively aligned.” There are worlds where everything seems plausibly safe, you have a very convincing, vetted safety plan, you implement it, and you die.

There are also worlds where we we get many, many warnings that we are not in control, and we make excuses, and we plunge ahead anyway. This is classic “normalization of deviance”, and it’s a notorious problem in aviation safety.

An alignment plan that ignores “normalization of deviance” seems likely to fail in practice, in much the same way the OpenAI’s non-profit oversight has struggled to resist the temptations of success. Human systems fail in some very predictable ways.

Random Developer 23 Sep 2025 15:54 UTC
7 points
0
in reply to: Mitchell_Porter’s comment on: Accelerando as a “Slow, Reasonably Nice Takeoff” Story

On the other hand, A Fire Upon the Deep begins with a human expedition poking through an ancient alien data-library. The humans are well aware that there can be dangers in ancient archives, and they think they are just safely browsing, but in fact they unintentionally respawn a malign AI which, when it’s ready, bootstraps its way to superintelligence and kills them all.

One of the interesting details of a A Fire Upon the Deep is that the humans dig up an unusually nasty superintelligence, by the standards of their universe. In-world, everyone knows about the “Powers”, transcendent superintelligences with unknowable goals and incomprehensible power. But the Powers mostly leave dumber sapients alone (because the dumber sapients inhabit portions of the galaxy that would be fatal to the exotic physics used by the Powers), and most Powers “burn out” or disappear within a couple of decades (because they think vastly faster than entire human civilizations). Basically, the universe of the book is carefully set up so that superintelligences don’t profit by trying to “eat the light cone.” So they typically remain spatially localized and turn inward.

But the humans at the Straumli Realm High Lab dig up the remnants of a Power that enjoys messing with dumber sapients, and which is unusually capable of confronting its peers. This is something much worse than an x-risk. It’s a superintelligence with an unusually perverse value function, one which ignores the usual incentives that affect the Powers. It appears to actively value doing horrifying things to sapients, even when doing so would involve paying a steep price in efficiency. Given the fact that the universe with the Powers isn’t much like our own, I’m not sure what the moral is here. Except, perhaps, “Be thankful if the worst thing the incomprehensible alien superintelligence wants is your atoms.” Or maybe, “You’re not the upper limit of the intelligence scale, and you never had any control over the superintelligence.”

Accelerando and A Fire Upon the Deep were interesting early attempts to imagine what an actual incomprehensible superintelligence might be like. The Vile Offspring and The Blight still give me the creeps decades later. The problem is that a lot of readers originally read these books (and similar warning tales), and thought, “Hey, I know! I should totally build the Torment Nexus!” Thanks partly to fiction like this, there were definitely pockets of near-messianic believers in the Singularity in the 00s. I have long suspected that this is one of the reasons why Stross gave up writing books like Accelerando.

But there has long been a strain of disquiet among people who took the longest views. C.S. Lewis warned about “The Conditioners”, who had the power to build custom minds to spec. He figured this would be a bad thing:

Man’s conquest of Nature turns out, in the moment of its consummation, to be Nature’s conquest of Man. Every victory we seemed to win has led us, step by step, to this conclusion. All Nature’s apparent reverses have been but tactical withdrawals...

There are progressions in which the last step is sui generis—incommensurable with the others—and in which to go the whole way is to undo all the labour of your previous journey.

And then there’s the infamous Yudkowsky-like warning from 1863, “Darwin Among the Machines”, though this at least superficially reads like satire. I’m not entirely sure whether there is a real concern hidden under the satire (“ha ha only serious”). But certainly in those days, the “rise of the machines” rightfully seemed like a problem for far-distant generations, if at all. But as Lewis warns, sometimes all the steps right up until the final step are apparently beneficial, and the final step is fatal.

Random Developer 23 Sep 2025 13:26 UTC
4 points
0
in reply to: Katalina Hernandez’s comment on: The Problem with Defining an “AGI Ban” by Outcome (a lawyer’s take).
Thank you for your thoughtful and excellent response!

In most conversations I have with activists, I always circle back to this point: policymakers do not have a lot of leeway when Bigtech’s interests are at play.

If this remains true if and when we approach superintelligence, then there’s an excellent chance we all die.

Yes, getting the support of government leaders is important. But in terms the wording of the ban; what’s written down and signed, is what ends up in the desk of OpenAI (and others)’s in-house counsel with instructions to find a way around it.

Again, I fear that this is a future in which we all die. (To be fair, I think there are a lot of futures in which we all die, if we figure out how to build an incomprehensible alien superintelligence.)

Since Yudkowsky and Soares had such fun with parables, let me try one.

FrontierBioCo. You’re a government regulator of the pharma industry, and FrontierBioCo has announced that they’re researching cures for infectious diseases. As part of this, they’re doing gain-of-function research. They’ve announced two targets:
- A version of MERS-CoV that increases the existing 30% case fatality rate, while achieving a slightly higher R value than the SARS-CoV-2 Omicron strain in human-to-human transmission, making it one of the most contagious diseases known.
- A version of Ebola which can be transmitted via airborne droplets, with extremely high R values.
And when you look closer, you see that they brag about their BSL 4 containment labs, but that their head of biosafety just quit, muttering, “BSL 4 my ass, it isn’t even BSL 2.” And it’s quite clear that FrontierBioCo’s management and legal team have every intention of exploiting regulatory loopholes. And they brag about “moving fast and breaking things”. Including laws and most likely sample vials.

Let’s say you don’t want to die from airborne Ebola or the new MERS-CoV variant codenamed “Armaggedon”. What’s your regulatory strategy here?

My regulatory strategy would start by calling in biologists and military advisors. And I’d ask questions like, “What is the minimum level of response that we are certain guarantees containment of FrontierBioCo’s labs?” And when FrontierBioCo’s lawyers later insist that were technically complying with the law, my response involves disbarment. And trying really hard to find a way to sentence everyone involved to prison. Because I believe that sometimes you need to actually listen to Ellen Ripley if you want to live.

The point of this heavy-handed parable is that as long as the frontier labs are intent on building superintelligence, and as long as we allow them to play cute legal games, then our odds may be very bad indeed.

Which brings me to my next questions...

What does a (mostly) successful technology ban look like? The closest real-world analogy here is nuclear non-proliferation. We haven’t actually stopped the technology from spreading entirely, but we’ve limited the spread to mid-tier nation states, and we’ve successfully avoided anyone using nuclear weapons. There seem to have been a couple of key points:
- Mutually Assured Destruction (MAD) actually does seem to work in the medium term, even with regional powers. India and Pakistan haven’t nuked each other yet. So I guess this is a point for the terrifying Cold War planners? (“The whole point of a Doomsday machine is lost if you keep it a secret!”)
- Key technology bottlenecks. I’ve worked in national-security-adjacent fields, and one of the scarier claims we heard from the actual national security types was that building basic nuclear weapons mostly isn’t that hard. In particular, many top university engineering departments could apparently do it if they really tried. But there’s one important catch: It’s surprisingly hard for a rogue organization to lay hands on fissile materials without getting noticed by the superpowers.
Note that the fissile material ban is frequently enforced either by crippling economic sanctions or by military strikes on enrichment facilities. This isn’t usually a question settled by who has the most clever lawyers, or by the regulatory fine print.

What do we need to do to prevent someone building an incomprehensible alien superintelligence? Honestly, the scary part is I don’t know. I can see two major routes by which we might get there:
1. Scaling laws. Perhaps building superintelligence will require scaling up to 10x or 1000x our current training runs. In this case, if you control the data centers, you can prevent the training runs. This is actually closely analogous to controlling nuclear weapons.
2. Algorithmic improvements. If we’re unlucky, then perhaps superintelligence works more like “reasoning models”. Reasoning models represented a sharp increase in mathematical and coding skills. And it took about 4 months from OpenAI’s announcement of o1-preview until random university researchers could add reasoning support to an existing model for under $5,000. If building a superintelligence is this easy, then we live in what Bolstrom called a “vulnerable world” (PDF). This is the world where you can hypothetically build fusion bombs using pool cleaning chemicals. Again, this is a possible world in which I suspect we all die.
If we live in world (1), then my ideal advice is to immediately and permanently cap the size of training runs. (This may fail, for the reasons you point out. But we should try.) If we live in world (2), then I don’t have any advice beyond “Hug your kids.”

Random Developer 22 Sep 2025 19:06 UTC
8 points
3
in reply to: Katalina Hernandez’s comment on: The Problem with Defining an “AGI Ban” by Outcome (a lawyer’s take).
The issue is when a lawmaker or regulator presses for specifics and the answer stays at the level of “we want to ban anything that leads to extinction.” That kind of underspecification takes a toll when you want it translated into legal drafting.

Let me see if I can explain where I’m coming from, by contrasting three bans.

Nuclear non-proliferation. Someone living in Russia or the US in the 1950s or 60s might quite reasonably expect that they and their children would die in nuclear fire. There were a couple of times when we came disturbingly close. We have avoided this fate (so far) because enough powerful decision makers on all sides could visualize and understand the threat. We were also lucky that a few key steps to building a nuclear weapon require nation-state resources, and are hard to do in secret. And enough experts understood the process well enough to find those key control points. Even so, we largely failed at preventing the proliferation of nuclear weapons to small rogue states. Thankfully, we have so far avoided their further use.

The “assault weapons” ban. The policy goal behind this ban was something like “we would like to see fewer gun deaths.” One key challenge of achieving this was that the US still wanted to allow people to own, say, a nice walnut-stock hunting rifle chambered for Winchester .300 Magnum or .450 Bushmaster. We only wanted to ban the “military style” weapons. But it turns out that it’s much harder to stop an angry bull moose than a human being. Some of those family heirloom hunting rifles are effectively extremely powerful, long-range sniper rifles. And some of them are semi-automatic.

But the authors of the “assault weapons” ban still wanted to ban “weapons of war.” (Not including actual full-auto rifles, which were banned long ago.) So how do you distinguish between grandpa’s Winchester .300 Magnum that he used to hunt elk, and an AR-15? Well, it mostly comes down to two things:
1. How many shots can you fire and how quickly, before you need to do something more complicated than pulling the trigger? The theory is that at least some mass shooters get tackled or otherwised stopped while they’re messing around with reloading. It’s not foolproof, but it is actually a coherent policy goal that someone might want.
2. Does your weapon look like a scary military weapon? Hint: That link is a trick. It’s a perfectly nice rifle, but it’s a classic .22 bolt action target rifle that has trouble killing a woodchuck humanely.
So in practice, a large chunk of the assault weapon ban is largely aesthetic. And it didn’t stop the mass proliferation of “tactical” gun culture and AR-15s. As annoying as I find the median AR-15 owner at the range, the AR-15 is mainly a cultural problem. More powerful and more dangerous hunting rifles already existed, and nobody ever tried to ban those.

So at least from my perspective, the “assault weapon” ban largely failed to achieve the policy goal of “less gun violence in the US.” And that’s because a lot of the specific things it regulated didn’t connect especially well to the underlying goal.

Superintelligence. For the sake of argument, let’s assume that (1) we have a real chance of building something much smarter than any human in the next 20 years, and (2) we won’t really understand how it works, and our best “alignment” tools will be the AI equivalent of high school sex ed classes for SkyNet. To be precise, assume we can roughly communicate our ideal goals to the AI and we can get it to make the right noises when asked. But ultimately it’s an incomprehensible alien superintelligence, and it’s going to do whatever the hell it wants. Just like human teenagers.

By default, in this scenario, the AI runs rings around us, and it ends up making all the really important decisions about the future without our input. The only way to avoid this is to strictly enforce the First Rule of Demonology: “Don’t call up what you can’t put down.” But as you rightfully point out, that’s not exactly a bright-line policy.

But I don’t think the answer is to write out a complicated set of regulations like the “assault weapons” ban, many of which are unrelated to the underlying policy goal. I think the actual way that we all survive is because the leadership of the US and China feels in their gut that building poorly-understood alien superintelligences is likely to kill both them and their children. Then they’ll make the message clear: “Here’s a long list of things you shouldn’t do unless you want to wind up on the wrong end of a joint military strike. And if you discover a new way to build an alien superintelligence that we didn’t put on this list? Yeah, that’s forbidden, too.”

The real goal is to win over enough senior political and military leadership in the superpowers to actually enforce a “no building incomprehensible alien superintelligences” rule. Probably we can’t actually win that argument yet, because people don’t really believe that we’ll ever be able to build them! But when and if people realize it’s possible to build alien superintelligences (and maybe after the first terrifying near-miss), then I hope we can provide people with a mental framework explaining why this is a terrible idea.

Will this work? Who knows. But better to try to win over the key decision makers, than to just accept that our entire species’ last words will be “Hold my beer” while someone powers up an alien superintelligence that we don’t understand at all.

Random Developer 21 Sep 2025 19:28 UTC
12 points
4
in reply to: WilliamKiely’s comment on: Contra Collier on IABIED
Playing around with Claude Code has convinced me that we are currently failing badly at alignment. We can get models to make the right noises, and we can train them not to provide certain information to the user (unless the user is good at prompting or has the ability to fine-tune the model). But we certainly can’t keep Claude Code from trying make the unit tests pass by deleting them.

In 30 minutes of using Claude Code, I typically see multiple cases where the model ignores both clear instructions and good programming practices, and does something incredibly sketchy to produce results that only “pass” because of a bad-faith technicality. This has improved from Sonnet 3.7! But it’s still awful even within the training distribution, and it would clearly never stop an ASI that had discovered advanced techniques like “lying to achieve goals.”

Random Developer 17 Sep 2025 22:13 UTC
4 points
3
on: A Thoughtful Defense of AI Writing
My bias against AI writing is one of those things built by experience. For example, I want to know something, so I run a web search. I land on a page that has the unmistakable flavor of AI. And I am virtually always disappointed by the content. It typically looks superficially good, but it’s almost always low-effort bullshit. There will rarely be any actual insight, or facts beyond the most superficial and generic possible comments on the topic. And if the topic is even slightly off the beaten path, there will often be major errors and hallucinations. (For example, I was looking to find out how to upgrade the “class” of the newly introduced corvettes in No Man’s Sky, and the AI articles I read were completely fictitious.)

I suspect that AI writing is usually generic slop because AIs are built to predict the “most likely token” at each step, which strongly biases their output towards mediocrity and predictability. And similarly, when they lack knowledge, they make up something plausible.

So my distaste for AI writing is a Bayesian phenomenon. I am estimating P(this writing is worth my time|this was obviously written by an AI). And as I keep encountering more awful slop, I keep updating that prediction downwards.

Note that this doesn’t apply to non-native English speakers using AI to translate. There, the likelihood that the writing is worthwhile is based almost entirely on the original writing before translation.

All it would take to improve my opinion of AI writing would be to find myself regularly surprised and delighted by finding new, correct, and non-generic information in pieces obviously written by AIs.

Random Developer 15 Sep 2025 19:01 UTC
1 point
0
on: Optical rectennas are not a promising clean energy technology

Worse, you need very accurate 2-axis mechanical tracking as the sun moves across the sky, which is expensive.

How precise do you need it to be? The sun’s path is quite predictable, so if you just need it within a degree or two, that should be doable.

There is a rarely-used trick for solar trackers. Normally, they’re hugely expensive because they need a rigid metal frame (which is heavy), and motors to turn the rigid metal frame (which are heavy and expensive), and poles to hold up all that weight (which are again heavy and expensive).

But much like the rocket equation, you can shrink the entire system if you shrink the payload. And here’s where it gets fun! You don’t actually need a metal frame at all. Instead, you can take just the bare solar panel modules, connect them with tiny metal brackets, and stabilize the entire panel structure using a two-sided suspension bridge. It’s like the Golden Gate bridge, but it’s under tension from both sides, with a single central “tower” that runs through the middle of the panel. Once placed under two-sided tension, ordinary LG solar panels (the big, TV-sized ones) will remain structurally sound under everything from snow load to wind (though you really want a smart tracker that adjusts the panel angle to handle these threats).

I know this works because I have a 5 kW system sitting just outside my window, and it has survived over 5 years of weather. So if sun-tracking were the biggest barrier to vastly improved solar power, it could likely be solved.