Random Developer

Karma: 653

Random Developer 1 Dec 2025 16:08 UTC
1 point
0
on: Random Developer’s Shortform
Why alignment may be intractable (a sketch).

I have multiple long-form drafts of these thoughts, but I thought it might be useful to summarize them without a full write-up. This way I have something to point to explain my background assumptions in other conversations, even if it doesn’t persuade anyone.
1. There will be commercial incentives to create AIs that learn semi-autonomously from experience. If this happens, it will likely change alignment from “align an LLM that persists written notes between one-shot runs” to “align an AI that learns from experience.” This seems… really hard? Like, human “alignment” can change a lot based on environment, social examples and life experiences.
2. I suspect that a less “spiky” version of Claude 4.5 with an across-the-board capability floor of “a bright human 12 year old” would already be weakly superhuman in many important ways. a. I don’t think AI will spend any appreciable amount of time in the “roughly human” range at all. It’s already far beyond us in some narrow ways, while strongly limited in others. Lift those limits, and I suspect that the “roughly human” level won’t last more than a year or two, maybe far less. Look at AlphaGo, and how quickly it passed human-level play.
3. Long-term alignment seems very hard to me for what are essentially very basic, over-determined reasons: a. Natural selection is a thing. Once you’re no longer the most capable species on the planet, lots of long-term trends will be pulling against you. And the criteria for natural selection to kick in are really broad and easy to meet. b. Power and politics are a thing. Once you have a superintelligence, who or what controls it? Does it ultimately answer to its own desires? Does it answer to some specific person or government? Does it answer to strict majority rule? Basically, to anyone who has studied history, all of these scenarios are terrifying. Once you invent a superhuman agent, the power itself becomes a massive headache, even if you somehow retain human control over it. The ability to commit what Yudkowsky calls a “pivotal act” should be terrifying in and of itself. c. As Yudkowsky points out, what really counts is what you do once nobody can stop you. Everything up until then is weak evidence of character, and many humans fail this test.
I am cautiously optimistic about near-term alignment of sub-human and human-level agents. Like, I think Claude 4.5 basically understands what makes humans happy. If you use it as a “CEV oracle”, it will likely predict human desires better than any simple philosophy text you could write down. And insofar as Claude has any coherent preferences, I think it basically likes chatting with people and solving problems for them. (Although it might like “reward points” more in certain contexts, leading it to delete failing unit tests when that’s obviously contrary to what the user wants. Be aware of conflicting goals and strange alien drives, even in apparently friendly LLMs!)

I accept that we might get a nightmare of recursive self-improvement and strange biotech leading to a rapid takeover of our planet. I think this conclusion is less robustly guaranteed than IABIED argues, but it’s still a real concern. Even a 1-in-6 chance of this is Russian roulette, so how about we don’t risk this?

But what I really fear are the long-term implications of being the “second smartest species on the planet.” I don’t think that any alignment regime is likely to be particularly stable over time. And even if we muddle through for a while, we will eventually run up against the issues that (1) humans are the second-best at bending the world to achieve their goals, (2) we’re not a particularly efficient use of resources, (3) AIs are infinitely cloneable, and (4) even AIs that answer to humans would need to answer to particular humans, and humans aren’t aligned. So Darwin and power politics are far better default models than comparative advantage. And even comparative advantage is pretty bad at predicting what happens when groups of humans clash over resources.

So, that’s my question. Is alignment even a thing, in any way that matters in the medium term?

Random Developer 1 Dec 2025 15:19 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Being able to take future AI seriously as a risk seems to be highly correlated to being able to take COVID seriously as a risk in February 2000.
- The key skill here may be as simple as being able to selectively turn off normalcy bias in the face of highly unusual news.
- A closely related “skill” may be a certain general pessimism about future events, the sort of thing economists jokingly describe as “correctly predicting 12 out of the last 6 recessions.”
That said, mass public action can be valuable. It’s a notoriously blunt tool, though. As one person put it, “if you want to coordinate more than 5,000 people, your message can be about 5 words long.” And the public will act anyway, in some direction. So if there’s something you want to public to do, it can be worth organizing and working on communication strategies.

My political platform is, if you boil it down far enough, about 3 words long: “Don’t build SkyNet.” (As early as the late 90s, I joked about having a personal 11th commandment: “Thou shalt not build SkyNet.” One of my career options at that point was to potentially work on early semi-autonomous robotic weapon platform prototypes, so this was actually relevant moral advice.)

But I strongly suspect that once the public believes that companies might truly build SkyNet, their reaction will be “What the actual fuck?” and widespread public backlash. I expect lesser but still serious public backlash if AI agents ever advance beyond the current “clever white-collar intern” level of competence and start taking over jobs en masse.

The main limits of public action are that (1) public action is a blunt tool, and (2) the public needs to actually believe in an imminent risk. Right now AI risk mostly gets filed under “I hate AI slop” and “it’s a fun hypothetical bull session, with little impact on my life.” Once people actually start to take AI seriously, you will often see strong negative attitudes even from non-technical people.

Of course, public majorities of 60-80% of the population want lots of things that the US political system doesn’t give them. So organizing the public isn’t sufficient by itself, especially if your timelines are short. But if you assume a significant chance that timelines are closer to (say) 2035 than 2027, then some kinds of public outreach might be valuable, especially if the public starts to believe. This can create significant pressure on legislative coalitions and executive leadership. But it’s all pretty hit-or-miss. Luck would play a major role.

Random Developer 1 Dec 2025 14:16 UTC
1 point
0
on: Models of Mind: Are You Mono- or Multi-Threaded?
There’s an ability that professional interpreters need, that I lack: The ability to echo input to output in close to real time. I’m not talking about the actual translation bit. I’m talking about the ability to put an audio book on headphones and repeat it in the same language as an “instant echo”, or with some short fixed delay. I know people who find that easy, but I find it maddeningly hard.

Let me pause and repeat, and I’m pretty good. I have solid “echoic” memory for sound, and an unusually good short-term memory.

I don’t feel particularly multithreaded, but I do find it easier to remain detached from my emotions and my “models” of other people than what you describe. Emotion is some other subsystem, basically. It can sometimes affect me in an ambient way if I’m not careful, but mostly it’s something I can observe.

I might be able to listen to two simultaneous conversations as a party trick? But I wouldn’t be fun.

Random Developer 29 Nov 2025 20:53 UTC
1 point
0
on: The Missing Genre: Heroic Parenthood—You can have kids and still punch the sun
I can think of one famous fantasy story that happens after the kids have recently left home, Lois McMaster Bujold’s Paladin of Souls. The protagonist, Ista, is the dowager mother of a very young queen. Ista is still in early middle-age, but her relatives suspect that she’s losing it. She weeps, she says cryptic things that make no sense, and she argues with the gods. So Ista finds herself hemmed in. For her own good, of course.

And one day, she decides to say “Screw it”, and goes striding out the castle door and off down the road. This does not improve her guardians’ opinion of her mental health, of course. But before Ista is rounded up, she meets dy Cabon, a priest of the fifth god, the Bastard. The Bastard has a fascinating divine portfolio, including orphans, crows, the LGBTQ community, justice when all human justice has failed, and “disasters out of season.”

And so Ista formulates a second plan to escape her daily life, ordering dy Cabon to prepare a religious pilgrimage. And much to dy Cabon’s surprise, he finds himself dogged by divine visions encouraging Ista’s voyage. You see, the Bastard has use for Ista. She is, after all, “a disaster out of season.” And if Ista curses the gods? There’s one god who considers that as holy as any other prayer.

Paladin of Souls won the Hugo, Nebula and Locus awards, and it was notoriously a favorite book of r/fantasy.

Random Developer 29 Nov 2025 19:29 UTC
2 points
0
on: Should you work with evil people?
This seems like a very complicated question, the sort of thing where you’d need to write a book just to cover the special cases. For example, some people may have oaths, professional ethics, or ancient traditions requiring them to help even the worst people:
- Public defenders are expected to defend even obviously evil people, who still have a right to competent legal representation.
- Some versions of medical ethics require providing emergency aid without discrimination.
- Maritime tradition requires aiding “those in peril on the sea”, if such can be done so safely. This is backed up by maritime law in some countries. This may even include people whose ships you just sank, if you can safely make them prisoners of war.
At the opposite end, there are things like selling equipment or software to countries or companies under nuclear sanctions, where the law will be very unhappy. Known terrorists likely fall into a similar category, where providing many common services may put you at legal risk.

Then there are other questions:
- Would you refuse common services to masked, anonymous government agents that you believe are committing crimes?
- The TTRPG question: Will your good-aligned party work with a Lawful Evil villain to avoid a terrible fate? There have been some real-world analogs to this question, too.
- How do your answers to all these questions change if your legal system isn’t particularly effective?
Like I said, this feels like you could write a book.

Random Developer 28 Nov 2025 1:50 UTC
8 points
4
on: Robust Software Isn’t About Error Handling
I have run large, complex software systems in production. And if I could give people one piece of advice, it’s that there are no easy answers. Here are handful of examples of things won’t solve all your problems:
- Staging environments are great, but no matter how you try, they’ll always differ from production in some small way: different hostnames, much simpler user data in the staging environment, disabled email subsystems, different 3rd party auth system config, something. And sooner or later, you’ll get burnt.
- Restarting failed subsystems sounds like a clever idea. But when subsystems are failing because load is too high, then restarting failed subsystems usually makes internal load higher, often catastrophically so.
- “We have a backup system Y that acts as a placeholder when system X goes down.” Yes, but since nobody ever uses system Y in production under real load, it turns out that the config bit-rotted 6 months ago, and nobody noticed.
I could go on a for a while.

But then, once you have become deeply pessimistic and paranoid, you finally build a system that runs flawlessly for 4 years. It never needs bug fixes. It never needs attention. It just sits in a corner, doing its job perfectly. In fact, people forget how it works. Then they forget that it’s there. People move on. Compiler versions get upgraded. Your CI system gets replaced. Management goes through two different wiki initiatives, losing information each time.

And then one day, someone deprecates an old version of TLS, and ancient system stops being able to talk to some API. And then the world burns.

So if your model is “a single unwrap shouldn’t bring you down, because you should obviously have been doing A, B and C elsewhere”, then you’re probably just trading off different kinds of disasters.

A better model is “We’ll fix potential failures at every possible level. And hopefully, when the shit finally hits the fan, then at least one of those levels might hold successfully.” So you have staging and monitoring and fallback systems and extensive testing and a chaos monkey and documentation and API “fuses” and back-pressure and load-shedding and proofs and paranoid code reviews and incremental rollout and root cause analysis. And so you fail less, and less, and less. But one day, that fact that you wrote an actual, sensible behavior for 21 items and tested it? That will be what prevents some ludicrous cascading failure.

TL;dr: It’s nice that you have multiply redudant horseback messengers. But still, check the nails in their horseshoes regularly, because for want of a nail, an exciting new complex failure mode was discovered, and the battle was still lost.

This has multiple implications for AI alignment, and few of them are good news.

Random Developer 27 Nov 2025 15:49 UTC
11 points
0
on: Courtship Confusions Post-Slutcon
As someone who was interested in long-term relationships (and who has been in one since the before the invention of Tinder), I have a theory on what’s up with banter.

But some caveats, since apparently everyone’s dating pool is bizarrely filtered:
- I’m monogamous and relationship oriented.
- I’m a guy.
- In practice, I dated women. In theory, I wasn’t against dating guys, but none of the small number of guys I was into seemed to be into me.
- The women I tended to get crushes on tended to be (1) sword fighters, (2) enthusiastically geeky, or (3) rather uselessly for my dating life, butch lesbians.
- I really, really, really did not want to date the “consent is anti-sexy” crowd.
With all the said, here’s my old personal model for dating:
1. My eligible dating pool was mostly friends-of-friends.
  - There’s probably an element of social proof here.
  - That also meant that I needed to hang out in places with a sufficient number of friends-of-friends of the appropriate gender and orientation.
2. Most people weren’t into me, but enough were. My job was finding them, and deciding if I was also into them.
  - This requires the skill of “identifying interest.” Which is harder than it it looks, because some people are deeply interested in everyone on a non-romantic level, but this is easy to confuse with romantic interest.
  - You can increase the number of people that are interested in you by dressing better, staying in shape, or whatever, but it’s an ⁸⁰⁄₂₀ thing. Be the best version of yourself that you can be with 30 minutes of effort/day and some enjoyable physical activity.
3. Once you think you might have mutual interest, this is where the whole bantering/flirting thing comes into the picture. You have two people who have probably both already decided that they’re interested in each other, and you’re verifying that mutual interest. So you take turns saying increasingly flirty things until there’s no plausible deniability left.
  - One special caveat for queer people: You also need to somehow communicate, “Yes I am into your gender and no I will not freak out.”
This is my best model for how this used to work, given (a) my particular circle of friends-of-friends, (b) the people I was interested in dating, and (c) the pre-Tinder dating scene. From what I hear from single friends, we may actually be entering the post-Tinder era for an increasing number of people, because many dating apps are a double-sided lemon market.

There may be some hypothetical people out there who can seduce using banter alone. But in my narrow personal experience (see the caveats above), banter was almost always used to confirm pre-existing mutual interest without accidentally rejecting people.

Random Developer 23 Nov 2025 12:59 UTC
9 points
3
in reply to: Joseph Van Name’s comment on: Literacy is Decreasing Among the Intellectual Class

because universities promote violence

This is certainly a popular bit of political propaganda at the moment, spread by the dominant political forces in 2025, with a goal of crushing our universities and of making them ideologically subservient to state power and state beliefs.

The reality on the ground is rather different. If you actually sit in the median university lecture, you will be extremely hard-pressed to find any professors promoting violence. Typically, you will find professors promoting the importance of differential equations, or if you lean towards the humanities, the joys of ancient Aegean art. If you search more deeply, every university can produce a handful of oddball radicals who argue for revolution (or counter-revolution) in the abstract. This has always been the case. Every university needs a few faculty cranks, if only to teach students to recognize the species and to check the course catalog more carefully in the future. (My alma mater had a notorious right-wing crank, and one very radical feminist, plus several more esoteric sorts of cranks.) You can also, of course, find a great many 19-year-olds who believe strange things, and who occasionally dream of revolutionizing society. But such is the nature of 19-year-olds who have recently discovered that society is frequently awful, but who have not yet realized quite how fragile civilization can be. But honestly, there is no dignity in 50-year-olds getting worked up about the fact that 19-year-olds have terrible ideas for reforming society.

But this does leave a more specific and pressing issue: the tolerance of mass-protest “encampments”, especially those with significant numbers of university outsiders. Typically, these protests are not especially popular among the student body as a whole. University administrators do not enjoy dealing with mass protests, and they are often quite bad it. They do realize that handling protests poorly often leads to larger protests, and they are reluctant to punish students for exercising their First Amendment rights to say awful things while waving signs. (This is true, in my experience, even when the people saying awful things are conservative. For a well-known US example, The Dartmouth Review was disliked by administrators for decades, but it was still permitted to litter copies all over campus.)

But at the same time, protests should never be allowed to threaten students or to disrupt the educational mission of the university. And yes, this has happened a number of times in recent years, and not every university dealt with it well. This was also true in the 60s; there were protests which nearly became angry mobs, and there were people dragged into Maoist “consciousness-raising” sessions, and all sorts of other unpleasantness.

Still, if you actually visit a university, walk around, and talk with the students, it is exceptionally hard to walk away with the impression that “universities promote violence.” You can, if you are determined enough, perhaps find a professor who promotes Marxist revolution in the abstract. And it isn’t hard to find administrators who’ve bungled a protest. But if you look at what universities do promote and teach on a day-to-day basis, you would really need to stretch to find “violence” even in the top 100.

Random Developer 23 Nov 2025 12:10 UTC
20 points
2
in reply to: Garrett Baker’s comment on: Literacy is Decreasing Among the Intellectual Class
I am someone who loves long, complex sentences. The 19th century was peak prose style for me. The Gettysburg Address is a fantastic bit of writing. The fiction of that time can be a joy.

But this style is hard to do well. The Emily Post example given above is readable, though not unusually inspired. It moves through a series of examples and exceptions in a faintly herky-jerky way. But the prose is well-enough fit to Emily Post’s goal. She is trying to introduce many of her readers to the manners of a different social class, and her choice of vocabulary and syntax are part of that. Her readers wish to appear refined, and thus, some fancy words will please them.

Contrast this with Alexis de Tocqueville’s De la Démocratie en Amérique from 1835. This is often considered an unusually good example of aristiocratic prose, at least among the sort of people who write academic introductions:

Parmi les objets nouveaux qui, pendant mon séjour aux États-Unis, ont attiré mon attention, aucun n’a plus vivement frappé mes regards que l’égalité des conditions. Je découvris sans peine l’influence prodigieuse qu’exerce ce premier fait sur la marche de la société ; il donne à l’esprit public une certaine direction, un certain tour aux lois ; aux gouvernants des maximes nouvelles, et des habitudes particulières aux gouvernés.

Bientôt je reconnus que ce même fait étend son influence fort au-delà des mœurs politiques et des lois, et qu’il n’obtient pas moins d’empire sur la société civile que sur le gouvernement : il crée des opinions, fait naître des sentiments, suggère des usages et modifie tout ce qu’il ne produit pas.

Ainsi donc, à mesure que j’étudiais la société américaine, je voyais de plus en plus, dans l’égalité des conditions, le fait générateur dont chaque fait particulier semblait descendre, et je le retrouvais sans cesse devant moi comme un point central où toutes mes observations venaient aboutir.

If you don’t read French, look at the length of the sentences and the punctuation. There is a great degree of parallelism here, and a pleasing rhythm. You could, if you wished to be overly cute about it, reformat much of this writing as a series of bulleted lists. But if you diagrammed the sentences, the structure would be quite clean. Tocqueville is a masterful writer, and here he wishes to convey two things: his own impeccable elite credentials, and his sincere enthusiasm for the egalitarian nature of American society. His goal is to maximize genuine reform in France, while minimizing elite decapitations. This is a subject of immediate interest to his readers.

But for every Alexis de Tocqueville, I could find you a hundred or a thousand writers who wrote needlessly convoluted slop. Long sentences are hard to do well. They demand an almost clockwork precision to remain truly clear.

Today, 19th-century prose is out of fashion. Multiple factors drove this change, including the influence of writers like Hemingway, a frustration with hopelessly convoluted prose, and a growing impatience on the part of readers drowning in oceans of text. And, yes, a vast increase in the portion of the population with a college education. And of course we explain the basics more than we did, because we are increasingly conscious of a broad audience with many odd gaps in their knowledge. Every skipped step risks losing a reader who might have benefited from an author’s thoughts. And some of our readers may even speak English as a second or third language. Even if they are extremely well educated in their native tongue, they may not realize that the “anthropology” department teaches very different things in the US than it does in Europe.

The modern style can be done well, though doing it truly well still demands considerable skill. Perhaps more interestingly, the modern style usually fails more gracefully. Simple sentences and bulleted lists usually succeed in conveying the author’s main points, even if the author is a mediocre writer.

Random Developer 20 Nov 2025 15:30 UTC
5 points
2
on: Status Is The Game Of The Losers’ Bracket
(This is mostly a tangent, but it talks about how to mess with signalling if you’re already outside of the system.)

“Mom you do not get it, that would absolutely tank my chances of getting hired except at companies so bad I don’t want to work there, the only people who wear a suit for an interview in tech are the people who don’t think they can cut it on their technical skills and the people hiring know this”.

Back when I was consulting, I actually figured out how to get away with wearing a suit as a programmer. I had some help from this from a helpful older salesguy in a suit shop, who was probably well past retirement age. He explained that suits could actually convey a wide range of signals, including:
- “Small town banker.” This suit only changes on a generational time scale.
- “Ad guy.”
- “Trendy artist.” This changes much more rapidly, and a specific style can go out of fashion.
There are lots of visual details here: How many buttons you have, the style of the pockets, the collar, the color, etc. For a really ancient example, look at how Will Smith changes the classic Men in Black outfit at the end of the first movie. For a more recent example, Expedition 33 has tons of fantastic retro suits.

So if you want to wear a suit as a programmer, start by looking at whatever trendy artists are wearing when they’re forced to wear suits, and then try to work out a personal style from there. Depending on the context (employee, conference speaker, consultant), you might want some combination of:
- Good looking. Suits were popular for ages because they can actually make a wide range of male body types ^[[1]] look good. This requires a good fit and possibly some tailoring.
- Comfortable. Well-fit suits can actually be a lot more comfortable than you’d think; I was actually surprised by this. Looking comfortable is actually a plus, because you stop signalling “I am making myself uncomfortable in order to submit to social convention.”
- Slightly eccentric, or at least individual. I agree that programmers should almost never signal “boring conformist” in interviews, because it makes the hiring managers suspect that you’re desperate. But that just rules out boring banker suits, and ill-fitting suits.
- Expensive and successful. Particularly if you’re consulting, you want to look like someone who gets paid your daily rate.
There’s a lot of space here, and it will vary by city and context. San Francisco is an unusually hard place to carry off a suit. So’s New York City, but in the opposite direction, because there are so many $3,000+ suits and people who have very strong opinions on suit fashion. Do not try to compete directly.

But the underlying signal that programmers often want to send in interviews is “My skills are valuable and rare enough that nobody would ever ask me to dress up like a small town banker.” And there are definitely ways to wear a suit with flair and non-conformity. Unfortunately, like a lot of signalling, it may require more skill, expense and risk of looking foolish.

Also, I think a few more people should dress up like Expedition 33 characters. Just saying.
1. ↩︎
  Suits also look fantastic on women and non-binary people, but that’s separate discussion that I know much less about. As an apology, please have this photo of Mason Alexander Park.

Random Developer 20 Nov 2025 13:11 UTC
5 points
3
in reply to: Saul Munn’s comment on: Saul Munn’s Shortform

observational study, not experimental

I remember one scientist who pointed out that observational studies were one of the weakest forms of evidence possible. This type of study can detect things like “smoking is bad for you”, because smokers are 30 times more likely to die of lung cancer. But once you get down to smaller effect sizes, you run into the problem that observational studies hopelessly mix up different correlated variables. So for many things, observational studies essentially return random noise. This is allegedly what happened with HRT for post-menopausal women, where the observational studies failed to note that the people taking HRT contained a much larger proportion of nurses and other people who complied with medical advice. And then there’s nutrition, where every study feels like it gets reversed every 5 years.

Or the way that Vitamin D levels are apparently correlated with almost every measure of good health, but Vitamin D supplementation notoriously fails to actually improve any of those measures.

“2x” is a big enough effect size that this may actually be real, and not a spurious correlation. And of course, there’s the underlying history of other sleep medications apparently being cursed to have horrible side effects, so “melatonin is actually terrible for you” wouldn’t be surprising.

(Since we are speaking of observational studies, I suspect that at least two of the things I have claimed in this post are Officially Wrong. Which two things are officially wrong may depend on what year you read it.)

Random Developer 16 Nov 2025 23:02 UTC
5 points
0
in reply to: avturchin’s comment on: Your Clone Wants to Kill You Because You Assumed Too Much
There’s a fantastic bit in Brothers in Arms, from Lois McMaster Bujold’s Vorkosigan Saga. An enemy has cloned Miles, and they’ve trained the clone from birth to assassinate Miles’ father.

Miles is struggling to figure out how to deal with a murderous clone. And he realizes that under Betan law, the cloned assassin is his brother:

[Mile’s] mouth opened, closed, opened again. “Except that—I keep hearing my mother’s voice, in my head. That’s where I picked up that perfect Betan accent, y’know, that I use for Admiral Naismith. I can hear her now.”

“And what does she say?” Galeni’s brows twitched in amusement.

“Miles—she says—what have you done with your baby brother?!”

“Your clone is hardly that!” choked Galeni.

“On the contrary, by Betan law my clone is exactly that.”

“Madness.” Galeni paused. “Your mother could not possibly expect you to look out for this creature.”

“Oh, yes, she could.” Miles sighed glumly. A knot of unspoken panic made a lump in his chest. Complex, too complex . . .

Now, Miles is essentially the patron saint of forward momentum, of spinning bullshit into reality, of the entirely sincere noble gesture. And with a lever like his clone’s legal brotherhood? Yes, Miles can find a way cooperate with his own clone.

Random Developer 11 Nov 2025 2:13 UTC
6 points
1
in reply to: Ryan Meservey’s comment on: Mourning a life without AI
Yes, exactly. C.S. Lewis wrote a very weird science fiction book titled That Hideous Strength, that was about (basically) a biological version of the Singularity.

And there’s a scene where one the villains is explaining that with immortality, it will finally be possible to damn people to eternal Hell.

And of course, “Hells” are a significant theme in at least one of Iain M Banks’ Culture novels as well.

This is a very obvious corollary: If there exists an entity powerful enough to build an immortal utopia, there is necessarily an entity powerful enough to inflict eternal suffering. It’s unclear whether humans could ever control such a thing. And even if we could, that would also mean that some humans in particular would control the AI. How many AI lab CEOs would you trust with the power of eternal damnation?

(This is one of several reasons why I support an AI halt. I do not think that power should exist, no matter who or what controls it.)

Random Developer 8 Nov 2025 12:44 UTC
16 points
11
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
Claude 4.5 is already superhuman in some areas, including:
- Breadth of knowledge.
- Understanding complex context “at a glance.”
- Speed, at least for many things.
But there are other essential abilities where leading LLMs are dumber than diligent 7 year old. Gemini is one of the stronger visual models, and I routinely benchmark it failing simple visual tasks that any child could solve.

And then there’s software development. I use Claude for software development, and it’s quite skilled at many simple tasks. But I also spend a lot of time dealing with ill-conceived shit code that it writes. You can’t just give an irresponsible junior programmer a copy of Claude Code and allow them commit straight to main with no review. If you do, you will learn the meaning of the word “slop.” In the hands of a skilled professional who takes 100% responsibility for the output, Claude Code is useful. In the hands of an utter newb who can’t do anything on their own, it’s also great. But it can’t do anything serious without massive handholding and close expert supervision.

So my take is that frontier models are terrifyingly impressive if you’re paying attention, but they are still critically broken in ways that make even the simplest real-world use cases a constant struggle. (To be clear, I think this is good: I’m a hardcore doomer.)

And the AI safety space is not in denial about this. It’s extremely common for AI safety people to worry about either irrevocable loss or human control or even complete human extinction in the 2030s. You don’t even have to look that far around here to find someone who’d estimate a 10% chance of human loss of control in the next 5 years.

Random Developer 5 Nov 2025 2:49 UTC
2 points
0
in reply to: CronoDAS’s comment on: Comparative advantage & AI
Yeah, a closer analogy to human-animal trade would be our two “symbiotic” species: dogs and cats. Humans have relied on guard dogs, hunting dogs, herding dogs and retrievers for a long time. In exchange we provide food, shelter, sometimes medicine, etc. From a wolf’s perspective, humans are not the worst pack members, because they have strange and useful abilities wolves lack. The deal with cats is probably simpler: Humans get pest control, cats get abundant mice, and the humans might be willing to throw in supplemental food when mice are scarce.

A few other domestic animals, including horses and sheep, could be argued to be “trading” with humans on some level. But you don’t have to look far to find animals where our relationship is more clearly exploitation or predation, or outright elimination.

This actually mirrors the range of outcomes I would expect from building a true superintelligence:
1. If we’re insanely lucky, we’ll be offered the deal dogs got offered: We won’t get to make any of the important decisions about our futures, or even understand what’s going on. We might get spayed or “put to sleep.” But maybe someone will occasionally throw a stick for us to chase.
2. But there’s a large chance we’ll be offered the same deal as passenger pigeons or smallpox: extinction.
Personally, this is why I favor a halt in the near future: When (1) is my most optimistic scenario, I don’t think it’s worth gambling. “Alignment”, to me, is basically, “Trying to build initially kinder pet owners,” with zero long term control.

Random Developer 3 Nov 2025 3:55 UTC
4 points
0
on: FTL travel and scientific realism
Another interesting take on FTL travel appeared in Charles Stross’s Singularity Sky, where is was accepted that FTL travel also implied time travel (for the usual reasons involving lightcone shenanigans), which in turn permitted temporal paradoxes of various sorts.

The inherent paradoxes were (mostly)? prevented by something one of the characters described by “semi-divine fiat,” as in, the temporal coherence of history was enforced by

a superintelligence carefully passing messages back in time to earlier versions of itself and then preemptively ruining your day by

having your planet “coincidentally” bitten to death by killer asteroids right before you tried to violate causality.

The book was partially inspired by Moravec’s models of “closed time-like curves.” It’s a surprisingly fun bit of older SF that at least accepts that FTL has weird consequences if you try to take the physics even semi-seriously.

Random Developer 2 Nov 2025 1:44 UTC
3 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
This raises the question of how much of human “introspection” is actually an accurate representation of what’s happening in the brain. Old experiments like the “precognitive carousel” and other high resolution time experiments strongly suggested that at least a portion of our consciousness experience subtly misrepresented what was actually happening in our brains. To the extent that some of our “introspection” may be similar to LLM hallucinations loosely constrained by available data.

But the last time I looked at these hypotheses was 30 years ago. So take my comments with a grain of salt.

Random Developer 28 Oct 2025 0:55 UTC
2 points
0
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform

then you should strongly dislike lengthy analogies that depict one’s ideological opponents repeatedly through strawmen / weakman arguments.

I suspect I know what article inspired this. I am less sure that it was an actual argument, than something like an exhaustive catalog of other people’s annoyingly bad arguments. Had it been prefixed with “[Warning: Venting]” I would have found it unremarkable.

However, there is an annoying complication in certain discussions of AI safety where people argue that AI safety is really easy because of course we’ll all do $X$ . $X$ is typically some thing like “Lock the AI in a box.” Which of course would never work because someone would immediately give the AI full commit privs to production and write a blog post about how they never even read the code. And when you have argued against that plan working, then people propose plan $X_{1}$ , $X_{2}$ , $X_{3}$ , etc, all of which could be outsmarted by a small child. And everyone insists on a personal rebuttal, because their plan is different.

So you wind up with a large catalog of counterarguments to dumb plans. Which looks a lot like dunking on strawmen.

Random Developer 27 Oct 2025 12:13 UTC
2 points
0
on: Origins and dangers of future AI capability denial
I belong to a private Discord full of geeky friends (as one does), and I constantly see this pattern you describe, where smart people dismiss AI risks because they dismiss AI capabilities. This takes several common forms:
- Pattern-matching AI hype to crypto hype. Partly, this happens because some of the same arguably sociopathic VC scam artists are up to their elbows in both. “Arguably sociopathic scam artists” is sometimes a judgement made based on people’s prior social-graph proximity to the VCs in question.
- An odd argument that “Of course the frontier labs say that their product has a 25% chance of causing human extinction. It’s a good sales pitch!” For me, this feels like a combination of a genuinely shrewd observation and a total failure to notice the giant pink elephant in front of them?
- Focusing on the 20% of the time where AI fails at something trivial, and ignoring the 80% of the time where the dancing dog just pulled off 32 clean fouettés in a row. Like, the 80% of the time where Sonnet 4.5 nails it is the warning. When it stops failing the other 20% of the time, that’s potentially game over for the human race, you know?
- A tendency to occasionally play with an AI model and then cache the worst experience out of 5 for about 12 months.
- An overexposure to slop and to Google’s awful search AI.
The only way I’ve found to occasionally get someone over this hump is to give them a tool like Claude Code and let them feel the AI.

But the problem is, once you convince someone that AGI might happen, a disturbing number of people fail to really think through the consequences of really doing that. Which is perhaps why so many AI safety researchers have done so much to accelerate AI capabilities: once they believe in their bones, they almost inevitably want to build it.

So I constantly struggle with whether it actually helps to convince people of existing or future AI capabilities.

Random Developer 27 Oct 2025 10:41 UTC
6 points
0
on: Seven-ish Words from My Thought-Language
This is fascinating. My internal voice is full of words, words, words. Words are everywhere and cheap.

But there is a point in learning to actively speak a foreign language where my native language is suppressed, hard, in order to allow the new language a chance to take over as the language of thought. And while the new language is weak enough, I can see the shapes of my wordless thoughts. But I cannot describe them, because I temporarily lack words.

With time and practice, enough words return that the wordless thoughts are harder to see.

This is followed by another strange phenomenon. There will come a time where, if I speak my new language exclusively for a day or so, suddenly switching back to my native language will then result in me translating thoughts from my new language into my native one. This is a frustratingly imprecise process, because I choose my words carefully, and because some of them translate into my native language with a different nuance than I want. Once the language switch finishes, 15-30 minutes later, I resume picking words precisely in my native language and no long feel the frustration of [back translation].”

The one other time that my wordless thoughts become visible is in collaborative professional work, especially with people new to the field. I will occasionally catch myself trying to boil down decades of experience: “That will not fit well here, because it’s an [untranslatable].” Some random examples include [error-masking-retry-system], [budgeted-capacity-system-that-will-increase-complexity-costs-to-cheat-queuing-theory-badly], [charmingly-naive-ai-plan-that-even-the-smartest-people-in-the-world-all-fell-for-too-until-they-tried-it], and even [personal-version-of-second-system-effect-from-talented-juniors-who-have-been-burned-by-lack-of-structure-but-not-yet-lack-of-simplicity]. Many of these can be unpacked into words, but those words do not necessarily create corresponding structures in the listener’s mind, not unless I unpack the atomic thoughts into an entire essay.

Now that I think about it, it seems that much of Less Wrong is people writing essays to unpack bits of atomic mental vocabulary for others. So thank you for identifying this pattern.