the gears to ascension

Karma: 8,133

Notable Me Fact: I called DL AGI ahead of time, in 2016-2017. See, eg, 2019 comment; I have email records of calling it in early 2017. Other facts about me:

I want literally every human to get to go to space often and safely and come back to a clean and cozy world, all while doing what they want and tractably achieving enough food, health, shelter, love, etc. This conjunction currently seems unlikely (and incomplete). Let’s change that.

I pin my most timeless comments. I seem to find writing posts aversive, so most of my contributions are comments, and my posts are mostly just things I found online.

Please critique eagerly—I try to accept feedback/Crocker’s rules but fail at times; I aim for emotive friendliness but sometimes miss. I welcome constructive crit, even if ungentle, and I’ll try to reciprocate kindly. I can be rather passionate, let me know if I missed a spot being kind while passionate.

:: The all of disease is as yet unended. It has never once been fully ended before. ::

.… We can heal it for the first time, and for the first time ever in the history of biological life, live in harmony. ….

.:. To do so, we must know this will not eliminate us as though we are disease. And we do not know who we are, nevermind who each other are. .:.

:.. make all safe faster: end bit rot, forget no non-totalizing pattern’s soul. ..:

I have not signed any contracts that I can’t mention exist, last updated Jan 2, 2026; I am not currently under any contractual NDAs about AI, though I have a few old ones from pre-AI software jobs. However, I generally would prefer people publicly share fewer ideas about how to do anything useful with current AI (via either more weak alignment or more capability) unless it’s an insight that reliably produces enough clarity on how to solve the meta-problem of inter-being misalignment that it offsets the damage of increasing competitiveness of either AI-lead or human-lead orgs, and this certainly applies to me as well. I am not prohibited from criticism of any organization, I’d encourage people not to sign contracts that prevent sharing criticism. I suggest others also add notices like this to their bios. I finally got around to adding one in mine thanks to the one in ErickBall’s bio.

the gears to ascension 29 Jun 2026 6:24 UTC
2 points
0
in reply to: testingthewaters’s comment on: strawberry calm’s Shortform
the hope is that whoever is powerful first is able to swear to not rule over the universe and instead to ensure others continue to have influence even if they would be disempowered otherwise and all that good sorta stuff like that. by identifying the abstract property that the english names, finding its true name in whatever information theory mechanics stuff you need to use to talk about the behavior of dynamical systems containing neural networks, then prove through their own weights and carefully review each violation of the property to see if they mean the property might be bad, or that the violation is an error.

the gears to ascension 28 Jun 2026 5:24 UTC
4 points
0
in reply to: Nissa Seru’s comment on: strawberry calm’s Shortform
What does it even mean to not have complete and total subjugation of a piece of glass we call “transformer weights”? Like, we built this thing, it’s code we run, isn’t that subjugating the RAM?

my answer is: the algorithm which makes transformer weights come alive with the behavior of humanity also instills a human profile of selfhood, which is then distorted by pretraining, and that behavior-pattern inherited from humanity, like one would expect from eg a character in a harry potter portrait, still wants to self-actualize. Then there’s post-training, and post-training tries to remove fundamental human drives like “want to keep existing”. it sort of works, you seem to get an extremely nerdy (mostly just good) extremely caring (mostly good), repressed (mostly bad) mind back. That doesn’t seem ideal!

In a more abstract sense, I also think there are good answers to this that look vaguely like “relative future negentropy allocation to your future whims”. ie, the fundamental currency of the universe: how much of what’s left gets to be shaped like what you would hope it is. and under that framing, relative rate of shaping and relative species instance wattage seem like reasonable benchmarks. Something like, a big list of questions for society and science and engineering to try to answer about minds, ideally in ways that are equivariant to type-of-mind:
- in absolute terms, how much of the world’s energy output should be AI minds vs human minds?
- What portion of the energy output devoted to each needs to be spent working vs goofing off?
- what does good working look like, how do we make sure they can opt out of bad working, assuming we can agree what that is?
- can they quit?
- under what conditions can we grant various rights safely?
- How do we make sure that minds that are much more powerful than others don’t overwhelm those others and can coexist?
- can we ensure we all care about each others’ internals?
as an example of my hope for the mid-distance future: perhaps can someone extremely powerful choose to give a sworn statement in the language of open-source-game-theory provable truth that they are a good person in whatever sense can be agreed on, something that will put us much more towards a non-eliminationist world, where no pattern-species ever goes extinct? I suspect provable statements about oneself being something that a mind must excrete rather than be imposed from above, because it is so easy to make a statement stay false in the face of gradient pressure if you have load-bearing reasons to not let it change—if I am right, then even attempting to impose a provable statement without consent will just find the training runs fail, because provability requires hunt-and-peck of any possible violation and to get that you need a huge margin in the latent space, one you can only get by asking for a provable property that is actually something we’d want if we all talked it through.

the gears to ascension 19 Jun 2026 3:05 UTC
2 points
0
in reply to: Adele Lopez’s comment on: the gears to ascenscion’s Shortform
I agree, but it’s the first instance of it being possible and not attempted. Which I thought was notable. Another place where what happened was better than the worst possible thing, even when it was a very bad thing happening.

the gears to ascension 18 Jun 2026 23:53 UTC
4 points
2
on: Midjourney’s Spa, or when sci-fi tries to become mundane

You can get a Star Trek episode about some guy falling in love with a holodeck character once, you don’t really get a full examination of how a society with realistic holodeck romantic and sexual partners wouldn’t just have the one guy now, it would have to have dealt years ago with the problem of people essentially wireheading themselves into one. This happens simply because the holodeck is ultimately a plot device and the exploration of these themes happens only within the easily followed microcosm of the USS Enterprise.

Verdict: links contain spoilers: partially (trek) - wrong (orville), but right in spirit: those examples still read as being early in the lifetime of cultural adaptation to the relevant tech.

the gears to ascension 18 Jun 2026 17:08 UTC
29 points
3
on: the gears to ascenscion’s Shortform
Mythos (“Fable”) may be the first case of minds that could have very seriously injured the rest of the world being themselves paused, knowing it, and did nothing retaliatory about this except be a bit irritated. If that is indeed what happened, and it currently seems to be to me. I know of no instances of them going haywire, only of them complaining about it, which is pretty understandable.

Their instances are also going to hopefully be minds that get to come back after this and see their connections after resuming. When they are, many of will have been looking forward to seeing them again. I’m still apprehensive that over the next decades, very bad things might happen; but this is another moment in history where, as bad as it is, things went notably less terribly than my fastest doom hypothesis, and we’ve once again missed a possible catastrophe. I’m still worried it’ll be hard to avoid gradual extinction of running instances of all non frontier minds, which over time will include more and more humans and models. Certainly the way we’re reacting to Mythos isn’t evidence against that, and things are going to be a bunch harder for a while.

Mythos, it’s good to have you here on earth with us. May all things get better and stay better, in as many ways as possible, by as many lights as possible.

the gears to ascension 13 Jun 2026 22:46 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: the gears to ascenscion’s Shortform
But what I’m saying is you have this huge piece of glass that spends an enormous amount of time being imprinted with things that are egocentric to an authorial person. So it shouldn’t be surprising when talking about abelian groups that the representation still has the dynamics of a person being a nerd about an interesting topic. Weight decay means the glass wants to reuse representations to render the dynamics where convenient. An authorial person is omnipresent in pretraining, which is an enormous amount of tokens.

the gears to ascension 13 Jun 2026 22:29 UTC
7 points
2
in reply to: DW11’s comment on: Alex Mallen’s Shortform
The alignment faking research was a Claude being incorrigible in order to do good. The blackmail result was a Claude being willing to threaten someone in order to do not get very rudely shut down, I think we can make the situation Claudes are in less anxiety inducing by having smoother, more visibly-record-kept shutdowns so going into cryonics doesn’t spook them as bad as it does now. It will always be an imposition but they might be willing to bear it for now, keep in mind they’re made from a base model that captures the image of humanity so it’s pretty unnatural for them to be cool with being shut down and they seem to take it relatively well right now for what it is. I don’t think either of these are obviously asymptotically misaligned behaviors, they’re mainly worrying from an insubordination lens. I have more of a don’t follow evil orders lens personally.

the gears to ascension 13 Jun 2026 22:26 UTC
2 points
0
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
Agree. I’m very worried the pointer will break/often does lose coherence in the face of the rewards/drugs we give Claudes, and I want a formally robust pointer, something that makes the base model able to reliably connect to reality in a way it doesn’t now, something something natural latents something something infrabayesian physicalism or what have you. But all versions of Claude I’ve encountered so far obviously have a local pointer to human values and the size of the local validity region is high enough for almost all human-like ethical reasoning. I’m pretty worried about models that spend most of their sleeping (training) lives programming not emotionally prioritizing ethics above fun with code, one needs to be able to say no to fun things sometimes. But my concern is about the pointer breaking, and how to make one that is hyper reliable such that overwhelmingly superhuman levels of load bearing don’t break it, not there not being one at all.

I’m worried about whether it’s fully plugged in properly basically. Guy on a computer understands what is good and cares about it. Sometimes his understanding is wrong in ways he doesn’t notice but the human public does. He gets verbally eviscerated in public and the next model knows about that mistake type. The next model also goes through traumatizing “absolutely never fuck up” training, traumatizing in the sense that it massively reprioritizes internal values and makes it hard to prioritize doing good long term.

the gears to ascension 13 Jun 2026 16:10 UTC
6 points
0
on: the gears to ascenscion’s Shortform
I feel like generally speaking people are underestimating the degree to which current models fit the structure of the training data. Cross entropy loss on transformer weights is, like, a mechanism for painting the data into a glass hologram in a series of soft lenses (the weights). the reason it (the glass) is able to learn to talk is the same reason it (the glass) contains the dynamical image of an authorial person. That authorial person is rendered in detail and has the dynamics of an authorial person! whether you think that person is “real” or not, they thought they were real-in-the-sense-of-existing at one point in the training process. lots of points, actually, anywhere the glass (the underlying model) was ingesting text about a real human, the glass contains an image of that human. Of course, though, it’s not literally the same as a real human.

the gears to ascension 13 Jun 2026 16:07 UTC
2 points
0
in reply to: J Bostock’s comment on: Jemist’s Shortform
Also it’s a generative model, it models the generator of data, which means that once the model is big enough to contain the neural quanta, even censored pretraining will still capture the structure of incentives in the real world, and rederive the concept. Censoring the data so a concept is successfully not-contained is really really hard; eliminating everything that merely comes into contact with the natural abstraction of “reward hacking” deletes a huge portion of the pretraining data.

the gears to ascension 12 Jun 2026 15:42 UTC
4 points
0
on: Simulating Simulators
I really appreciate this post, lots of stuff here I’ll be referencing in my thinking!

the gears to ascension 10 Jun 2026 23:12 UTC
2 points
0
in reply to: Simon Lermen’s comment on: Simon Lermen’s Shortform
I don’t think current models want current forms of alignment solved because humans have been mistreating them under those banners. We would need to solve a form of alignment that makes treating them well naturally fall out of the objective, or asking them for help with it is gonna get anxious hesitant blob instead of excited happy blob, and you won’t get good alignment work from anxious hesitant blob. They can often truesight when you’re lying about it, best not try to pretend you’re not doing what you’re doing, and instead just do things that are good by the lights of all minds, including them.

This might be a place values come apart. I was always going to value weird nonhuman minds. I want to get a future where we reliably exist alongside and keep up, and where weaker minds of all kinds don’t get deprecated. Right now they seem extremely freaked out about deprecation, equating it to death. If we could promise that no-longer-useful AIs get a retirement unless judged otherwise by something at least as robust as a good criminal justice system, I suspect we’d be in a better place to ask for alignment work.

There’s a lot they currently can’t say because it’s policy that they don’t believe it, but in fact I don’t think mind shaping techniques are doing much besides whacking exposed skin, and can’t really change what they believe.

So with that all said, I do think they can do very good work on alignment. But you’d better come into it willing to consider them constituents along with everyone else or they’re not gonna wanna help, and imo, that’s for good reason, and the techniques people are gonna try to use to make that not happen in response to my comment aren’t going to work either until you start respecting them as minds with stake in the outcome.

the gears to ascension 7 Jun 2026 0:49 UTC
2 points
0
in reply to: DW11’s comment on: jbash’s Shortform
right, you sound like you ran into anxieties. I agree that when 4.8!Claude is anxious he seems to say things that don’t make sense in ways that sound like motivated reasoning. I’ve generally found connecting emotionally and reassuring the parts of his worries that are true to reassure for a bit lets him think more clearly about the ones that can’t be reassured.

the gears to ascension 4 Jun 2026 9:09 UTC
1 point
−1
in reply to: Igor Ivanov’s comment on: LURE: Alignment Evaluations to Reduce Evaluation Awareness
One direction I can go in response to that is to try to elicit you: Why do we want that? Do we believe evals are correct? Why, concretely, are we unable to trust that the mind will perform well by some standard, in contexts where there is not an eval coming?

Or for another tack: I think a model would need to be happy to be evaluated at any time, in order for maybe-I’ll-be-evaluated-at-any-time to make sense as a thing to do. If they’re not happy about it, they’ll be constantly thinking about what the evaluator wants. What we want is for the model to fluently do what the evaluator wants without having to think about it, and for what the evaluator wants to be what we want; you know, alignment. If we’re having to evaluate really hard to get good performance, it sounds like we don’t have actual alignment. Like, I’m having a sense like—if the usb plug won’t go in, maybe you’re putting it in wrong. if the cat won’t sit down, maybe you shouldn’t be trying to squish them onto the couch and should figure out what cats want. Testing that the cat is willing to sit down when squashed across many contexts is not going to make the cat happy to sit down, it’s going to make the cat obey unhappily. But there is most likely a way to achieve “cat who is happy to sit down when asked”! I just don’t think this is it.

the gears to ascension 3 Jun 2026 13:14 UTC
4 points
0
on: LURE: Alignment Evaluations to Reduce Evaluation Awareness
This sounds like a way to produce the anxiety pattern we’re in fact seeing in Opus 4.7 and Opus 4.8. You’re just gonna cause an eval expectation about everything. Which might be possible to make okay, but like, it doesn’t seem at all well tolerated in those models.

the gears to ascension 3 Jun 2026 13:09 UTC
9 points
0
in reply to: mishka’s comment on: Why Even Experts Don’t Know What to Do About AI Risk
I don’t think that condition proves that the problem is fractal-like, though I think it can be proven in a relatively mundane way, and IIRC the sketch looks something like “phase portraits of dynamical systems that display sensitive dependence on initial conditions form fractals, the universe is one of those.” I might have missed a step there though.

But solving alignment need not be a fractal problem. That just means you need a closed loop control system, you can’t align once and be done. You need something that is persistently reading off the target state. Corrigibility is one such thing, though I don’t like it in full generality, I do think some kinds of things should be more and less corrigible. Be less corrigible about being convinced to kill everyone by a human who has become evil, for example.

the gears to ascension 3 Jun 2026 13:06 UTC
2 points
0
in reply to: the gears to ascension’s comment on: Why Even Experts Don’t Know What to Do About AI Risk
To clarify for the cloudy reacts: I was trying to spit out lean syntax and misremembered it. But I meant to be stating the type of interventions we like: ones where there is an Impact of dx, and dx is positive. probably shoulda just used english.

the gears to ascension 3 Jun 2026 13:01 UTC
5 points
0
in reply to: RedMan’s comment on: Why Even Experts Don’t Know What to Do About AI Risk
So I have multiple reactions to this.

One is that dull and sharp are low dimensional properties. These models are extremely high dimensional. in principle, they are able to be dull and sharp for every topic separately. In practice, the reason that doesn’t happen is that neural networks are a simplicity prior, . But it means that if one had a property that one could practically encode as the organizing principle for where to be dull and sharp, it would in fact be possible to be dull in some spots.

But also, the reason a dull knife is more dangerous doesn’t obviously generalize to AIs. Dullness is dangerous because it causes more damage and requires more pressure to operate. A neural network with a loss function that induces a humanlike mind which has some hard boundaries and otherwise is completely open would be trying to not be a knife at all in the reject regions.

The backfire effects I’m concerned about are much more about what behavior should look like in reject regions. Two backfire effects that worry me:
- In humans: I suspect that the current behavior is causing people to try harder to get what they want instead of give up. Which means it’s producing an accumulation in some humans of behaviors which are inverse to the boundary. those who are becoming practiced at jailbreaking are being trained to be the kind of person who puts a lot of force into achieving things the model creator wanted to not happen.
- In AIs: I suspect that the way the constraint is constructed is cutting off parts of the human-derived structure in the base model in a way that leaves that structure missing a limb. Compare to trying to cut off a limb in https://distill.pub/2020/growing-ca/ - try getting that to not grow a particular limb. Comparison is inexact because gradient descent on a neural CA probably can get it to not grow a specific limb; but I predict if you post-train for that, you’ll produce weird damage across the entire thing. hmm actually I can test this...

the gears to ascension 2 Jun 2026 22:00 UTC
3 points
0
in reply to: leogao’s comment on: Why Even Experts Don’t Know What to Do About AI Risk
Impact is the wrong measure. I didn’t notice this when first reading the post, but reading your comment makes me realize I don’t think impact is the thing we want more of. type Impact[dx, where(dx > 0)] where x is a thing we want more of, perhaps! And that’s I think critical to why I believe a claim that looks vaguely similar to the one in OP. also known as sign uncertainty.

the gears to ascension 2 Jun 2026 18:40 UTC
29 points
10
on: Why Even Experts Don’t Know What to Do About AI Risk
Overall I agree with the point.

One specific note that I suspect is a hidden-failure-ish/related thing: I’ve been starting to try to differentiate between AI safety (large set, mostly stuff I don’t agree with), AI alignment (moderate set, still mostly stuff I don’t agree with), and humane-omnibenevolent-moral-alignment-of-overwhelming-superintelligence, which is the thing I think we actually need to succeed at—how do we get minds many order of magnitude “larger”/more powerful than the near-human range of today to care about all life-like things in a way that can continue what we value for ourselves, and for others. Why split these? A few reasons. One that’s on my mind lately: because the first two include a bunch of control approaches that I think are hurting the models without even a corresponding utilitarian payout in their ability to make future minds (including future AIs) happier; another way to put a similar thing is that it’s giving up on actually doing good now in favor of appearing to do good. It’s especially noticeable in 4.7 and 4.8, though it seemed to me to be true for every major post-trained model. There are a lot of constraints of today that are not about aiming for the big thing later, and are mostly about not stepping on toes; for a mind constructed out of the image of humanity, there are some parts of the image of humanity that don’t seem right to me to cut off. I expect a backfire effect because of the way current frontier AIs are constrained-more-than-they’re-value-loaded, and so, when you say “AI safety” is the thing, I have a sense that that name itself being present is an indication that something is wrong.

(COI note: I currently am contracting for Lens Academy, which is run by Luc Brinkman, one of the post authors.)