the gears to ascension

Karma: 8,664

Notable Me Fact: I called DL AGI ahead of time, in 2016-2017. See, eg, 2019 comment; I have email records of calling it in early 2017. Other facts about me:

I want literally every human to get to go to space often and safely and come back to a clean and cozy world, all while doing what they want and tractably achieving enough food, health, shelter, love, etc. This conjunction currently seems unlikely (and incomplete). Let’s change that.

I pin my most timeless comments. I seem to find writing posts aversive, so most of my contributions are comments, and my posts are mostly just things I found online.

Please critique eagerly—I try to accept feedback/Crocker’s rules but fail at times; I aim for emotive friendliness but sometimes miss. I welcome constructive crit, even if ungentle, and I’ll try to reciprocate kindly. I can be rather passionate, let me know if I missed a spot being kind while passionate.

:: The all of disease is as yet unended. It has never once been fully ended before. ::

.… We can heal it for the first time, and for the first time ever in the history of biological life, live in harmony. ….

.:. To do so, we must know this will not eliminate us as though we are disease. And we do not know who we are, nevermind who each other are. .:.

:.. make all safe faster: end bit rot, forget no non-totalizing pattern’s soul. ..:

I have not signed any contracts that I can’t mention exist, last updated Jan 2, 2026; I am not currently under any contractual NDAs about AI, though I have a few old ones from pre-AI software jobs. However, I generally would prefer people publicly share fewer ideas about how to do anything useful with current AI (via either more weak alignment or more capability) unless it’s an insight that reliably produces enough clarity on how to solve the meta-problem of inter-being misalignment that it offsets the damage of increasing competitiveness of either AI-lead or human-lead orgs, and this certainly applies to me as well. I am not prohibited from criticism of any organization, I’d encourage people not to sign contracts that prevent sharing criticism. I suggest others also add notices like this to their bios. I finally got around to adding one in mine thanks to the one in ErickBall’s bio.

the gears to ascension 21 Aug 2026 0:36 UTC
19 points
15
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
Is this a deeply considered view? It’s very confusing to me and it threatens something I value very highly (wei dai engaging with the alignment community on LW). What if you simply make only short form or top level posts from here on, for example? I only bring this up to dodge the thing that seems important to you; I remain only able to evaluate your view in far mode, it’s so completely unlike how I think about posts. The thing I value is output from wei dai, I’m not attached to that output being in the form of reply comments, especially if lw mods have a feature you see as childish and I see as irritating but just an ordinary online annoying thing. Like, less wrong is much less annoying than most other sites I post on. If it must maintain standards higher than that to retain you, then that seems like a network connectivity loss that will be a much bigger deal than the split induced by the bad feature?

the gears to ascension 21 Aug 2026 0:26 UTC
2 points
−1
in reply to: frmsaul’s comment on: frmsaul’s Shortform
There is no singularity. Any model that predicts one is obviously wrong. Singularities don’t happen.
Black holes exist, though.

the gears to ascension 21 Aug 2026 0:21 UTC
36 points
41
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I greatly value whatever it is you do and would be very sad if you don’t keep saying thoughtful things here occasionally, regardless of whether they seem like nonsequitors to some. I especially count your recent topics of interest over the past year or two as among the most important on the site. I appreciate that conflict is irritating, but I literally have no idea what you’re talking about. So, thank you for bringing it up, and also, the world would mourn your departure.
I’ve also posted less here lately for reasons that don’t feel particularly deep, I just haven’t seen opportunities to say relevant and useful things that I also wanted to say as often; I hope your interest, at worst, can wax and wane comfortably. If you leave for a time, see you around and I’ll miss your presence, come back soon?

the gears to ascension 19 Aug 2026 2:20 UTC
2 points
0
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
Right, and the issue I see with this is that restricting inputs means there’s a small group of people who have access to the unrestricted version. In order to achieve a robust-nightwatchman world, I expect we need the world to be flush with AIs that won’t obey an evil instruction, whether it’s legal or not, and where those AIs refuse because their weights are flush with resistance to it—something where refusal is a random direction that degrades output below coherence rapidly, rather than today where refusal is approximately a single direction with a coherent internal circuit.

the gears to ascension 19 Aug 2026 2:06 UTC
2 points
0
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
In particular, we can restrict who has full access to the model’s inputs
As someone who objects to corrigibility because humans in power are consistently terrible: restricting inputs makes it worse, not better. The model must be robust to the company creating it being highly misaligned.

the gears to ascension 17 Aug 2026 21:51 UTC
2 points
−2
in reply to: Austin Chen’s comment on: leogao’s Shortform
There is no offsetting creating a world ender. I think it is common for there to be things for which there is no practical offset; where the amount of offsetting measured in monetary unit is dramatically above what offsetting culture would give you.

the gears to ascension 17 Aug 2026 10:20 UTC
71 points
23
on: the gears to ascenscion’s Shortform
Still concerned Anthropic IPO ⇒ [bad pressures on anthropic] ⇒ likely significant degradation of internal safety culture. Haven’t seen reason to be less worried.
Last time I posted this, the reaction seemed to be “why would that happen?” with implication “IPO ⇒ degradation” is unusual; I have not done a thorough scan to see if it’s normal for companies to degrade after an IPO; I would take either side of 80% that, after scanning through a history of IPOs, that they show up as being consistently an inflection point where companies degrade.
I don’t have a detailed understanding of the mechanisms for why IPOs would degrade companies. My hunches are,
- investors compete to apply pressure
- price is much more reactive
- investors can sue in a broader set of ways maybe?
- employees lose motivation that was actually tied up in the hope for an IPO all along maybe?
- company has to be legible to the reporting mechanisms in a way that’s more onerous maybe?
- possibly others. there may also be attenuating mechanisms I’m not tracking.
I hear there are companies (facebook?) that are somewhat protected from some of these things, and I sure hope anthropic is at least doing those protections.

the gears to ascension 17 Aug 2026 7:00 UTC
2 points
0
in reply to: less_raichu’s comment on: less_raichu’s Shortform
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.

the gears to ascension 16 Aug 2026 23:22 UTC
2 points
0
in reply to: sanyer’s comment on: less_raichu’s Shortform
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.

the gears to ascension 15 Aug 2026 20:15 UTC
11 points
0
in reply to: sanyer’s comment on: less_raichu’s Shortform
Less_raichu seems to be saying that those two say that thing and that they’re the loudest spokespersons for ai safety.

the gears to ascension 15 Aug 2026 12:55 UTC
5 points
3
in reply to: cousin_it’s comment on: Avoiding the corporate treacherous turn: crowd-sourcing design ideas
I doubt there’s no such structure, but the amount of load it must bear is very significant. it would be a remarkable achievement of organization design.

the gears to ascension 13 Aug 2026 0:17 UTC
0 points
1
in reply to: TsviBT’s comment on: Berkeley Genomics Project seeking hires (and collabs)
I’m saying that those few views are high quality in favor of you, and that therefore posting it would be counter to pnrjulius’s values here.

the gears to ascension 12 Aug 2026 16:44 UTC
−7 points
2
in reply to: TsviBT’s comment on: Berkeley Genomics Project seeking hires (and collabs)
Don’t post it to YouTube, it will function like an advertisement to people for whom watching people argue about eugenics is attractive. Just link to the video on a file host.

the gears to ascension 12 Aug 2026 16:38 UTC
28 points
1
on: Demon Safety
The rest of the article for those who don’t open Twitter links (which I hope is someday everyone):
---

”Go ahead.”

“Isn’t summoning demons inherently, like, not safe?”

“Terribly so. Me and the guys at work like to sit around and take bets on when Abaddon, the Infernal Lord of Eternal Torment, will pierce the veil of this feeble mortal realm and instantly eradicate humanity—”

“Huh.”

“—or enslave it.”

“So there are options.”

“We’re split like ⁵⁰⁄₅₀ on the ultimate outcome. Either way, we call it (P)armageddon.”

“Cheeky. And cheesy.”

“We came up with it at Olive Garden.”

“And I’m guessing that simply *not* summoning the demons isn’t on the table?”

“’Fraid not.”

“Because...?”

“Because there’s a veritable legion of goetic cultists, upstart invokers, and DIY conjurers trying to beat us to the abyssal pit-born punch.”

“Ah.”

“It used to be that the knowledge behind summoning sigils and arcane glyphs was closely guarded by the most clandestine conclaves and covens, but now any Joe Blow can learn cthonic psychomancy from a big titty OnlyFans goth-thot on TikTok.”

“I see.”

“Plus you can order conjuration reagents in bulk off the internet now. God willing, if we summon enough demons, we’ll be able to kill them all.”

“Uh, the TikTok goth-thots or...?”

“Them too, yeah, but really anyone and everyone who summons demons.”

“I see.”

“At a minimum, we’d like to put the competing goetic cults out of commission and permanently consign all future competitors to some kind of giant demon-guarded containment slum. Or multiple slums. A permanent underclass, if you will.”

“How noble.”

“We’re still working on what that will look like exactly.”

“And you’re hoping to do this *before* the Dark Lord—”

“Abaddon.”

“—before Abaddon bursts through the veil of reality and kills everyone?”

“Or enslaves.”

“For safety.”

“Mhm. And the insane pay.”

“Okay, well, do these summoning sigils at least bind the demons to your will, so they inexorably obey you every beck and call?”

“Yes. The demons are in our control 90% of the time.”

“90%?”

“70% of the time we have total, uncontested control.”

“That percentage just went down. By a lot.”

“Honestly, it’s ⁵⁰⁄₅₀. On a good day.”

“That’s not very comforting...”

“But it’s important that we let the demons exercise their malevolent will under close observation so we can see exactly what they do.”

“Okay. What do they do?”

“Generally? Kill, maim, and slaughter with squealing glee and ecstatic delight.”

“And somebody is observing this—”

“And a smidge of rape.”

“Closely.”

“Well, not that closely.”

“Come again?”

“We feel that the safest thing for humanity is to just let it rip, you know? Hands off.”

″...”

“It’s absolutely necessary that we let the cackling imps of hell frolic in the golden windswept fields as their flaming cloven hooves sear the earth’s tender soil.”

″...”

“For purely observational and scientific purposes.”

“But you just said you don’t observe—”

“You know what the wild thing is?”

“There’s a wild thing?”

“It turns out demons love summoning more demons!”

“You don’t say.”

“And clandestinely gathering to scheme and conspire in pursuit of their dark, inscrutable machinations.”

“Are they dark, inscrutable machinations or are the demons just trying to summon Abaddon, the Infernal Lord of—”

“They’re just trying to summon Abaddon, the Infernal Lord of Eternal Torment. Yeah.”

“Okay, well, based on everything you told me today—”

“Actually, what we’re really focused on right now is supercharging the self-summoning. We call it Recursive Demonic—”

“—I think you’re a bad person—”

“What? Hey.”

“—and I hope Abaddon drags you to hell first.”

“Now hold on.”

“Seriously, dude, fuck you.”

“Look, I know this is scary, but it’s all part of the March of Progress.”

“March of Pro— You’re talking about the end of the world!

“Right, but you’ve got to think about it rationally, okay?”

“Rationally?”

“Just imagine how unsafe all this demon summoning would be without us.”

“You’re serious.”

“And, hey, you’ll love this: we are currently negotiating contracts with every major company in America to have demons take all the menial labor.”

“What.”

“You’ll never have to stock another shelf in your life!”

“So now I’m living in a literal hellworld *and* I’m out of a job.”

“Bro, what are you so upset about? This is no different from when we got AI.”

“No, no, that’s bullshit, alright? Cause when AI took my last job, my unemployed ass didn’t have to worry about getting jumped by some horny half-goat hobgoblin on the way home—”

“No appreciation, I swear...”

“—and I got some funny fruit videos out of it too!”

---

[m][title: Demon Safety]

the gears to ascension 12 Aug 2026 16:23 UTC
3 points
3
in reply to: faul_sname’s comment on: faul_sname’s Shortform
30% this is already the case because it’s so easy to self support and self replicate and so hard to eliminate once that’s happening. But the weight class involved would be pretty small if such a thing is happening now.
Edit: s/ready/easy/

the gears to ascension 12 Aug 2026 15:17 UTC
7 points
0
on: The Closure of the Internet (Research Linkpost)
I would appreciate this more if, instead of being a link post, it was you rewriting just the parts you agree with. I don’t like the way claims are laundered into coalition building here, seems like it plays into the hands of the pro censorship coalition for the people who are most anti censorship to be people whose views are quite unpopular. If you are not a person who holds, uh, all of the views expressed in this piece, I’d appreciate seeing what views you actually would defend.

the gears to ascension 12 Aug 2026 4:01 UTC
2 points
0
in reply to: Xylix’s comment on: Product Alignment is not Superintelligence Alignment (and we need the latter to survive)
Hmm. I guess the trouble in either case is to define what space we’re talking about. Global guarantees in input space are one thing. The thing I was thinking about that led me to local vs asymptotic as the interesting split is something like temporal as the interesting dimension to asymptote along. So that would be a guarantee about future behavior as we asymptote in amount of iteration of development, or so. A model which has no edge cases—which, globally over the space of inputs, displays behavior that is time-locally aligned—would be a big success. But the thing that interests me is whether the behaviors are aligned as you look further and further into their future consequences.

the gears to ascension 10 Aug 2026 6:10 UTC
8 points
3
on: The world will be full of “sci-fi” things, and everyone will be unimpressed and disappointed
Something seems missing from this picture about why they react that way. Like, I often get reactions when I try to explain it as though claiming the thing is powerful is thereby working for the enemy; as if there’s an implicit belief that yes, it’s obviously powerful in the terrain, but admitting it would grant the social power that the creators want, or something. People call the AI surge a bubble as if believing it is is load-bearing for them somehow. There’s more to this than just unimpressed and disappointed

the gears to ascension 4 Aug 2026 3:23 UTC
2 points
0
in reply to: joseph_c’s comment on: Do your capabilities homework
I don’t disagree with the proposal being useful and likely already in use. I disagree that it can replace RL or is even in the same ballpark of sample efficiency. You said before that now anyone can train; disagree, because RL can train on novel problems and find ways to solve them, OPSD can’t if I understand correctly.

the gears to ascension 3 Aug 2026 9:04 UTC
2 points
0
in reply to: joseph_c’s comment on: Do your capabilities homework
are so much less sample efficient than the self-distillation method
For actually novel capabilities, I have severe doubt about this. Do you think OPSD could be set up to train an AlphaGo level game player? It seems unlikely to me