Random Developer

Karma: 1,560

Random Developer 11 Aug 2026 15:15 UTC
15 points
7
in reply to: Taylor G. Lunt’s comment on: Linda Linsefors’s Shortform
Better, but it’s still glossing over the critical bit, which is “previously unrevealed to who?” Specifically, “zero day” implies both that:
1. The vendor or maintainers had no idea of the vulnerability’s existence.
2. A diligent defender would have zero days to prepare, hence the name.
Meanwhile, it’s entirely possible that the attacker has known for months or years.

Words like “unknown”, “unreleased”, etc., are all imprecise about this asymmetry. For example, if Microsoft knows about a vulnerability in Windows but hasn’t shared that information with the public, that’s “unrevealed” to the public. But most people would not consider it a zero day.

“Zero day” is an established term that is widely understood by practitioners and the relevant professionals. It might not be an ideal choice of terminology when writing a newspaper article for a non-technical audience.

Random Developer 11 Aug 2026 11:07 UTC
8 points
3
in reply to: Taylor G. Lunt’s comment on: Linda Linsefors’s Shortform
This is mostly how language works? Language is semi-arbitrary phonemes that are assigned meanings based on historical usage and mutual understanding. This jargon is long standing and widely understood by practitioners.

“Zero day” actually means something different from “previously unknown”. Many “zero day” vulnerabilities are actually previously known to someone. For example, the NSA is presumably sitting on a pile of “zero day” vulnerabilities, judging from what was revealed about their Tailored Access Operation programs in the past. Similarly, I have heard from pentesting people claim that some pentesting companies supposedly keep a “zero day” or two in their back pocket to ensure that their important pentests produce some results. In general, attackers have a strong incentive to stockpile “zero days,” and to selectively deploy them against high value targets when needed.

So “zero day” is actually a shorthand for something like “vulnerability previously entirely unknown to vendors/maintainers and defenders at large, but potentially privately known to one or more attackers.” It might, in limited circumstances, even include vulnerabilities that have been previously used against low-awareness defenders, if the attacker successfully evaded any recording mechanisms or post-motem detection.

Random Developer 11 Aug 2026 0:45 UTC
4 points
2
in reply to: Steven McCulloch’s comment on: The Agentic Clusterfuck
There are so, so many versions of an AI future where we wind up on the wrong end of Darwin.

If the agents can mutate/learn, and if more successful agents replicate, then there will be natural selection for the most effective replicators. This would undermine any kind of alignment in much the same way that cancer eventually escapes the body’s defenses.

You don’t want to be on the wrong end of Darwin, especially not in the long run.

Random Developer 28 Jul 2026 14:42 UTC
34 points
9
in reply to: Ryan Meservey’s comment on: Ryan Meservey’s Shortform
Yes! Calling your Congresspeople is surprisingly easy! You don’t need to prepare brilliant arguments, or to win over staffers. It is OK to be massively awkward, or to read off some notes you wrote beforehand. What you’re doing when you call is creating a database entry containing:
- The name or number of a bill.
- Whether you oppose or support it.
- Your name and street address (which will normally need to be in their district).
There is no database column for “has excellent social skills” or “wasn’t awkward at all.” Even if you do have something brilliant to say, it may only make it into the database as shorthand, at the very best.

These database entries get summed up and affect how the staffers and Congressperson plan. For example, there is a big difference between “No constituent has ever called us about the AI Kill Switch Act”, and “1,000 constituents have called us in favor of the AI Kill Switch Act.” What kind of action this turns into depends on what kind of Congressperson you have.

The leverage you get by calling comes from the fact that most people don’t call. So when you do call, you represent some nebulous larger group of voters who probably agree with you. And by calling, you say that this is important enough for you to actually take a concrete action. That is, you’re saying that this is a high-salience issue for you and that you’re probably the sort of person who shows up to vote, or who talks to other voters.

In EA terms, calling your Congressperson is a low-cost action. Multiplied by a group of callers and multiple districts, it can potentially affect legislative outcomes for one of the world’s major powers. Success is not guaranteed. But for the effort invested, it’s often not a bad intervention at all.

Random Developer 26 Jul 2026 0:53 UTC
17 points
6
in reply to: Nikola Jurkovic’s comment on: nikola’s Shortform
My concern is that historically AI safety work, especially technical AI safety work, seems to be highly correlated with rapid AI progress. Sadly, if you look at how many modern AI pioneers were heavily influenced by the rationalist community, it’s possible that Eliezer Yudkowsky single-handedly moved us years closer to dangerous AI.

So before I start encouraging smart friends to work in AI safety, does anyone have a good model of how “working on AI safety” actually improves AI safety? Does mech interp, for example, increase AI safety more than it increases the number of tools we have to make models smarter?

EDIT: To be clear, this is a genuine question. What evidence do we have that various branches of AI safety are contributing to AI safety more than AI capabilities? This is a question that I’d really like to see answered seriously before I encourage clever people I know to get into AI safety.

Random Developer 23 Jul 2026 0:07 UTC
3 points
0
on: The Conjecture of Strong Subjectivity
Even if it were possible to observe someone else’s conscious experience directly, it seems like you’d need a very specific set of conditions to hold. Specifically:
- The observer would need some kind of physical access to the observed being’s seat of consciousness. (Full disclosure: I am not a dualist.) This would likely require a physical connection at the neural level, at least for humans.
- The observer would need to have a mind “large” enough and correctly structured to observe a second set of conscious experiences.
To use a silly analogy, there is nothing that theoretically prevents physically placing a car inside a larger car. But cars, by and large, and not designed to do that. And one car would need to be much larger than the other. (I said this was a silly analogy.)

So consciousness might be “strongly subjective” in practice for all the kinds of minds that currently exist, without necessarily being so in theory.

Random Developer 20 Jul 2026 16:56 UTC
5 points
2
in reply to: Simon Lermen’s comment on: Simon Lermen’s Shortform
If I understand the terminology correctly, China can’t “distill” US models in the strictest sense of the term, because they don’t have logit access. What they’re doing is just recording conversations between some input user (either some model they control, or an actual human), and saving that as training data. This doesn’t violate copyright, because Anthropic doesn’t own the input copyrights, and the Copyright Office doesn’t believe model output can be copyrighted. It does violate Anthropic’s terms of service, and maybe you could really stretch to a CFAA violation. But there’s zero reason China would care about ToS violations or hypothetical CFAA arguments.

Basically, anything the Chinese labs want to do is probably more legal than half the stuff the average Silicon Valley unicorn has done. (E.g. Uber has… creatively interpreted a lot of municipal regulations.) But I am not a lawyer, and this is not legal advice.

Random Developer 6 Jul 2026 23:02 UTC
4 points
2
in reply to: FlorianH’s comment on: social lemon markets

Anecdote from my country, where people ride trains all the time, fitting your description, although it takes a weird kind of extra ‘excuse’ in this case all the time: It would often feel weird to randomly talk to your seat neighbor, but ANY slightest excuse (sudden bump in the ride; info speaker malfunction; grumpy ticket collector; one weird word from a random person in the wagon, … any smallest thing) will an extremely frequently make the silent start conversation, and then easily for hours if the ride lasts that long.

This tracks. In Boston in the 90s, talking to randomly selected people was almost always perfectly fine. But responding to people who spontaneously initiated conversations in public places resulted in talking to some deeply weird people.

Random Developer 6 Jul 2026 21:28 UTC
7 points
0
in reply to: Rattengift’s comment on: Rattengift’s Shortform
Math is one of those subjects where everything builds on itself, step-by-step. And skipping prerequisites for something makes it much harder to learn. I would start by figuring out the last math that you know fairly well (multiplication tables? fractions? pre-algebra?), and then figuring out what math you would like to know.

The three big pieces of high school math in the US are algebra, geometry and (depending on the school) calculus. It’s possible to dislike algebra but really enjoy geometry.

My biggest advice for math is to do the problem sets! If you don’t, it will sneak up on you and you’ll start losing track. Try looking at new ideas from multiple angles, and ask yourself how it works. Play around with it. The textbook and the problem sets are guides, and the real work happens in your head.

My best physical analogy is getting strong. You gotta lift the weights and do the reps, and that takes time. This gets a lot of clever people in trouble, because they try to push too far, too fast, and then they get lost. If that happens, go back study the previous step some more.

Random Developer 3 Jul 2026 5:54 UTC
8 points
2
in reply to: clickyquack’s comment on: clickyquack’s Shortform
The only ways to prevent Mythos-level open models would need to happen in China. Nobody but China is likely to produce a Mythos-level open model.

There are two main ways this might not happen:
1. The Chinese companies may decide to keep their most powerful models private as training costs go up. GitHub’s recent decision to resell Chinese open models near the frontier may also annoy the companies which make those models, in much the same way that AWS repacking open source software annoys the companies that created it and tried to sell their own hosting services.
2. The Chinese government may independently start to worry about the capabilities of Mythos-class models and impose regulation of their own.
One of the complicating factors here is that Anthropic and OpenAI are preparing for the largest rent extraction in the history of the human race, and they represent a massive risk of concentration of power. (That power might be effectively seized by the government, but it’s likely to remain concentrated.)

So there will be strong advocates for open models, including open frontier-adjacent models, in order to fight what looks like epic-level rent extraction and power concentration risks.

US regulation, by itself, will accomplish exactly nothing, because Europe would happily use cheap, open Chinese models to avoid losing control to US labs and to avoid paying enormous rent.

I find this whole logic deeply frustrating, because the logic pushes everyone involved towards racing. And while Mythos isn’t an ASI-level threat, I don’t know how many more breakthroughs and scale-ups we can get away with before we start facing loss-of-control risks.

Random Developer 1 Jul 2026 19:31 UTC
7 points
2
in reply to: abstractapplic’s comment on: abstractapplic’s Shortform
I am neither an EA nor a rationalist as such, though I respect both groups as “weird but principled” and making real contributions.

That said, the standard formulation goes something like, “You may not be interested in politics, but politics is interested in you.” Politics can create and fund programs like PEPFAR, but it can also destroy them with the sweep of a hand. And at the margin, relatively small and careful political investments can influence very large scale policy. So there probably are political interventions with a cost effectiveness that competes with more typical EA interventions.

This doesn’t necessarily mean that engaging with politics is automatically something that every cause should do. There can be advantages to being non-partisan and “outside the system”, or at least very narrowly focused on a specific policy across parties. But for other causes, it’s likely better to engage with the system.

The complication here is that AI is nearing “crunch time”, where politics might matter in practice. And even EY has made it clear that he believes that most promising approach is an international treaty, because any sufficiently large data center might be dangerous.

So I’m not suggesting that any particular form of engagement will necessarily serve any particular goal. I am instead arguing that politics of various forms shouldn’t be ruled out automatically.

Random Developer 25 Jun 2026 4:36 UTC
56 points
24
on: AI catastrophe: more like a genocide than a thought experiment
There is a huge psychological difference between “discussing far-off things in the abstract” and “oh shit we’re actually doing this now.”

I, too, once thought that the Singularity would be sort of fun. You know, back when I was reading Nanosystems (1992) and “Lobsters” (2001) and admiring Manfred Macx. Even then, I knew that the Singularity wasn’t actually likely to be a good thing. But a few years layer I figured there was no plausible way it would happen sooner than, oh, 2025–2045. ^[1] And it was still pretty abstract and science fictional.

Well, 2025 rolled around, and by that time, I had firmly passed from “far off future” to “this is thing that might actually happen soon, to me and my family, on the same kind of time horizons that I plan for my kids’ college or my own retirement.”

And then I start to draw the nasty conclusions:
1. We still have no idea how to align something smarter than us.
2. If we could align a superintelligence, then some specific human beings would control that superintelligence. Probably either politicians or billionaire CEOs, as the fight over Fable suggests. Whether a potential SkyNet obeys the Epstein class or merely its own impulses is not that comforting a question.
3. Claude Fable 5 already has more raw intelligence than most human beings. It’s just poor at many practical applications of intelligence that don’t fit in an hour or two. Once that’s fixed, we’re going straight to ASI, with barely enough time at “AGI” to blink, let alone hold a public discussion. ^[2]
And so I’ve viscerally felt that shift from “far off future” to “oh shit this is happening.” Those people you hear saying things like “Maybe human extinction wouldn’t be that bad?” When they realize that their actual kids might die in the near future, you’ll hear an entirely different story from most of them.
1. ↩︎
  That was roughly when I expected 10^15 to 10^18 FLOPS, the lowest possible bounds on the algorithms equivalent to human cognition, to become pretty widely available to organizations. An RTX Pro 6000 Blackwell is about 10^15 marketing FLOPS at ~~fp8~~ fp4. So not bad, as guesstimates ~20 years in advance go.
2. ↩︎
  EDIT: I argue for these points in more detail elsewhere (see my profile). Or see If Anyone Builds It, Everyone Dies, which makes a slightly different argument along similar lines. My goal here is to convey the psychological state that hit me when I thought “oh shit this is real”, not to specifically argue for these points.

Random Developer 23 Jun 2026 7:03 UTC
4 points
0
in reply to: Raemon’s comment on: steve2152′s Shortform
Who cares whether LLMs in their original form are going to scale to AGI? That’s clearly not the mechanism by which they will scale to AGI. “LLM-base-models + tons of diverse RL environments” are obviously the default path, and I think even people who are relatively bullish on LLMs who don’t think very hard about it are still implicitly assuming that.

I’m not the person you’re replying to, but I suspect that LLMs+more RL will not scale to AGI. I think that current LLMs they’re a convincing existence proof that we’ll get to AGI, probably within my lifetime, but they’re not the actual thing. The actual thing will probably require a major breakthrough in either flobination ^[1] or grinkling.

But on the other hand, I don’t think this makes much difference. So what if you turn out to need LLMs+RL+grinkling before you build AGI? I fear we’ll make it there quickly, and once we do, we’ll pass quickly enough to ASI. This wouldn’t necessarily require some dramatic kind of RSI. My impression of Opus and especially Fable is that they’re already superhuman in important ways, but that they’re “spiky.” If the spikes in Fable were to be smoothed out, then I think it would already be a weak superintelligence ^[2] at the far upper end of the human distribution.
1. ↩︎
  Since I personally believe that robust, enforced alignment is straight up impossible in principle, and that actual AGI has a significant chance of being a diagnosis of stage 4 invasive cancer for everyone I know, I make a point of not discussing specifically what I suspect is missing from current AIs. There is no real point in disclosing inherently unfixable security problems. “Security by obscurity” is inferior for problems that can be fixed. But in the real world, there’s an enormous difference between “a physical security risk that experts can figure out” and “a security risk that is common cultural knowledge.”
2. ↩︎
  “Weak” superintelligence: Does basically what the most capable humans do, but significantly faster or with much higher output. “Strong” superintelligence: We couldn’t do much of what it does even with many human lifetimes of effort from our best thinkers. I think we’ll get the weak version more or less automatically once we get AGI at all.

Random Developer 22 Jun 2026 12:25 UTC
6 points
3
in reply to: Trinley Goldenberg’s comment on: Contra Pace on When to Apologize
Making it too obviously transactional is usually bad, because that suggests that you’re very carefully keeping score. It means you’re envisaging a short-term relationship where every negotiation restarts from zero. This can be read as slightly hostile, unless your relationship is already quite distant and likely to remain that way.

Whereas a more indirect exchange suggests a longer-term relationship where one side or the other can occasionally “run a tab” for a while. Doing favors without worrying about the exact balance says that you’re the kind of person who does favors and who is willing to hear people out when they need it, which is more valuable than being someone who keeps an exact balance at all times.

This is presumably why the Japanese compensation ettiquette I linked elsewhere in the thread kept empathizing things like sincerity, apologies, and clearly indicating that the money was intend as a symbol of your feelings (and not something which exactly balanced out the original offense).

But all this stuff gets mediated by culture. One of the things I have always loved about anthropology is trying to figure out how mostly universal principles actually manifest in a specific culture.

Random Developer 21 Jun 2026 21:58 UTC
6 points
2
in reply to: Trinley Goldenberg’s comment on: Contra Pace on When to Apologize
To clarify, I’m mostly saying that lots of cultures do lots of things.

For example, there are places where significant chunks of the economy are based on elaborate, competitive gifts of pigs. There are places where if someone admires the best fish you caught today, you have to give it to them. (And if this rule is abused constantly, you still can’t say “no”. Violence, however, might be culturally permissible.) There are a lot of cultures which customarily give cash gifts in situations where people in the US would find it tacky.

But the specific case of “cash gifts as apologies for inconvenience” is actually murkier than I expected. There are a few Japanese customs that are mentioned in passing in English-language sources. Spending 20 minutes with Google Translate does find examples like this, which discusses various forms of non-business compensation, and distinguishes between “actual expenses” and supplementary money used to indicate apology:

If you’re unsure about the amount, it’s common to include a few thousand yen as an apology for the inconvenience caused, in addition to the actual expenses that need to be reimbursed.

All the sources I can find on these traditions suggest that cash compensation for inconvenience is a social minefield, and that if it’s seen as “buying someone off” insincerely, it may backfire.

(Other traditions, like weregild and diya, don’t really fit this framework, because they involve things like death, disability, and/or trying to prevent good old-fashioned blood feuds.)

Random Developer 21 Jun 2026 20:01 UTC
10 points
5
in reply to: Trinley Goldenberg’s comment on: Contra Pace on When to Apologize
The way things like this work in many US subcultures is often pretty indirect. Humans are very good at tracking indirect and diffuse forms of reciprocity over long periods of time. Often, this is based less on a 1-for-1 exchange, and more on building a reputation as a helpful person who’s happy to reciprocate.

One approach would be some combination of:
1. Be generally pleasant and helpful to your neighbor.
2. Apology hugely for any inconveniences.
3. Admit that this event is kind of a big deal for you, and that you’re going to owe them one.
But having studied anthropology, I also have to admit that “offering cash to neighbors to compensate them for a loud evening event” isn’t even in the top 10 weirdest things actual cultures do. And given the cost of living these days, probably somebody is in fact delighted with cash.

Just be warned that this kind of gesture can easily become “standardized”. If you give the king a specific gift three years in a row, it’s now tradition. And you’ll probably need to keep doing it.

Random Developer 10 Jun 2026 23:11 UTC
4 points
0
in reply to: cousin_it’s comment on: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
Ah, yes. That Hideous Strength. A strange novel, certainly.

I’ve long felt that it was about the Singularity, as seen from a primarily biological perspective. You see the dreams of power and transcendence, the early partial successes, and the inevitable corruption of nearly all the people involved. There’s even a discussion of s-risks. The ending is positive because C.S. Lewis wanted to tell an entertaining story, and because his religion encouraged him to believe in impossible last-minute grace.

I may have referred to various AI labs as “the National Institute for Coordinated Experiments” in the group chat. I suspect that they employ more than one Filostrato, Wither or Frost.

Google once prided itself on “Don’t be Evil,” and they even sometimes lived up to it, to some degree. But money and power have corrupted them pretty thoughly, over stakes no higher than some ad revenue. And one of the arguments of That Hideous Strength is that people who push for a Singularity are unusually corruptible, because the temptations are so great. And, frankly, because they would need to be unspeakably arrogant and reckless to try.

Random Developer 8 Jun 2026 19:59 UTC
2 points
−1
in reply to: Ben Livengood’s comment on: Models finding software vulnerabilities is not the primary source of cybersecurity risk
Would rewriting Linux to seL4 standards cost $16B in the world before or after frontier models are solving Erdos problems?

After. (EDIT: For the incredibly basic reason frontier models are already solving Erdos problems, but the largest LLM-based piece of system software is only borderline functional dumpster fire of a C compiler, but keep reading.)

It’s fantastically impressive that LLMs can solve Erdos problems! But:
1. Erdos problems are much smaller than the Linux kernel, and much easier to verify. Both these factors play to LLM strengths.
2. Not all Erdos problems are equally difficult. Some seem to be just difficult enough that a human would need to seriously dig in and grind it out, or bring a nice insight from a distant corner of math. Others are likely more difficult. I believe about 551/1217 are currently solved.
3. Reimplementing a highly-secure kernel that could replace Linux for most uses cases is an enormous project with many poorly defined subtasks, which is basically hell for LLMs. The biggest LLM project I’m aware of is a C compiler. Which basically works, but which is apparently a remarkably terrible C compiler. And C is far, far better specified than the internal guts of Linux and all the ghastly hardware that Linux supports.
Also, please keep in mind that I’m willing to handwave a couple of zeros either way on the larger estimates.

I expect overall security of newly written code to get clearly worse on average (thanks to the deprofessionalization of software development and companies pushing vibeslop to production) right up until offensive Mythos-class abilities become widespread. Then I expect things to get unpleasantly exciting.

Random Developer 8 Jun 2026 14:06 UTC
11 points
3
in reply to: sanxiyn’s comment on: Models finding software vulnerabilities is not the primary source of cybersecurity risk
seL4 is a fantastic piece of technology. The problem is that it is a small, highly specialized kernel: SeL4 is about 10,500 lines of code, built at a cost of about $350-400 per line of code. Apparently they found that verification cost scales supralinearly with size, too.

In practice, seL4 deals poorly with the fact that most hardware is itself bug-infested garbage, and that much of it has enough DMA or other access to cause mischief behind the kernel’s back.

The Linux kernel itself is now about 40 million lines of code, much of it in drivers. Using an extremely approximate methodology, the core of a modern Linux environment is likely over 250 million lines of code. Various wild-ass guesses place the total amount of code in the world at anywhere from hundreds of billions of lines to trillions of lines.

Redesigning and rewriting the entire Linux kernel to seL4 standards would cost around 16 billion dollars using an extremely naive linear extrapolation. Rewriting a distro core brings us to around $100 billion. Rewriting all the software in the world would bring us to an exceptionally wild-assed guess of, say, 500 billion lines of code times $400/line, or $200 trillion dollars. Give or take some zeros, to be clear.

Now, let’s look at how these numbers are changing. GitHub is desperately trying to increase capacity for the agentic era:

We started executing our plan to increase GitHub’s capacity by 10X in October 2025 with a goal of substantially improving reliability and failover. By February 2026, it was clear that we needed to design for a future that requires 30X today’s scale.

So the amount of code that would need to be secured is increasingly drastically. But is this new agentic code more or less secure than the old human-written code? Let’s look at a recent comment thread on Hacker News talking about how many companies’ engineering standards are changing in response to agentic coding, e.g.:

I’m seeing so many of these come in with “this is 95% done, just need a couple of minor tweaks for production release”

“Minor tweaks” being fix the layout so it’s not messed up if the browser isn’t exactly 1920px wide, sometimes these filters and sorting don’t seem to work right and the app doesn’t seem to refresh new values properly after an action.

No matter the issue it’s pre-estimated by the business as “should be a quick fix, for an experienced dev” because they (allegedly) did 95% of the work already.

So my fear is that the widespread use of agentic coding will actually push us towards a world of vibecoded slop being pushed straight to production. This code will sometimes make extremely basic security errors, like hard-coding the database password in the publicly-readable UI code.

So, yes, I agree that we do know how to build fairly secure software. I just find it profoundly unlikely that we’ll actually do it: The cost would be unimaginably vast, and the rapid deprofessionalization of software engineering means that all the incentives currently point to far less security and 10x the new code per year.

Random Developer 8 Jun 2026 2:33 UTC
3 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
Sorry, I hadn’t realized that I had been depressing twice when replying to your posts! My apologies!

It’s perfectly OK to model me as the person making the argument that ASI is just inherently risky, for Darwinian/economic/polical reasons. But my personal P(doom) was only about 30% the last time I did the math, because I actually see a bunch of potential off-roads. For example, if we see mass job losses more than a couple of years before ASI, I would actually expect a significant political backlash. If the public ever believes that mass permanent unemployment and/or actual human extinction are on the table, then they may be very unhappy. (They’re already unhappy in the abstract, but it isn’t driving their decision making.)