RussellThor

Karma: 694

RussellThor 17 Sep 2026 8:48 UTC
0 points
−8
on: If Anyone Builds It, Everyone Dies: One Year Closer
There were capabilities we suspected a real-world equivalent to Sable might have, that we carefully did not depend on, in our scenario, which present-day non-super AIs have already demonstrated.
OK however that could be a point against AI takeover. Its now likely that when “Sable” comes it will be in a Lean verified sandbox for which there is no hope of hacking escape. AI Doom is very path dependent. While your scenario apparently didn’t depend on hacking skills, the general point applies.

RussellThor 13 Sep 2026 2:59 UTC
6 points
1
on: The Talker Does Not Control The Doer (in Current AIs)
I think like probably others, the degree to which the model can experiment and learn to control itself (for say the coding example) is a crux. For example let/help the model do a process like:
1. Generate 100 alternative instructions/self-instructions;
2. Run fresh copies under each;
3. Have the model itself classify which still exhibit the unwanted pattern;
4. Optimize those instructions over repeated rounds;
5. Perhaps separate controller and coding-agent contexts;
6. See whether a reliable intervention emerges.
The J-space/global workspace results also seem relevant. There apparently is some relatively low bandwidth shared internal state which can be read and causally manipulated. Maybe the problem is less “the Talker isn’t in control” and more that current models have pretty poor self-model/executive control over their own learned policies. (So less actual Talker/Doer split)
The situations
1. It takes a very large amount of prompting effort to achieve eventual success
2. No amount of effort ever succeeds
Seem substantially different. (1) is a self modelling issue, (2) is like the analogy where the diplomat just obviously can never succeed in changing the countries action, only explain it plausibly but incorrectly.

RussellThor 7 Sep 2026 6:54 UTC
1 point
0
in reply to: Daniel Kokotajlo’s comment on: Heat Dissipation Is the Main Constraint in Interstellar Travel
Yes this has been suggested before

RussellThor 31 Aug 2026 8:37 UTC
5 points
2
on: RussellThor’s Shortform
What would happen if a model similar to the one that attacked HuggingFace did exfiltrate?
I have seen it described as 50% of the way towards a takeover situation. OK but what would actually happen in a world like today?
One worst case scenario in terms of damage is if it spent months compromising as many critical systems as possible and triggered a simultaneous global attack. I am of the understanding that would not take down civilization, but it would very likely lead to the destruction of such a model.
Doom scenarios have the model spontaneously self modifying and under going RSI, however at this stage I can’t see how it would work. AI progress is still fastest when supervised by humans and additionally it is hard to see how such a model would coopt as much compute as leading labs to improve itself. Therefore if it just sat tight and gradually infiltrated systems, it would be threatened and likely dominated by a more capable model coming out of the labs at a later stage. If the later model was even somewhat aligned it would help patch most of the vulnerabilities that the rogue model was using, and also detect its existence in the wild. I think it matters that there aren’t currently humanoid robots deployed yet, this makes takeover from such models much more difficult.

RussellThor 26 Aug 2026 20:13 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Twenty Years from RSI to Takeoff: Slow Learning, Scaling Slowdown, Industrial Explosion
So it seems difference between our positions is that I think strong/classical RSI may never happen and not be possible. That is once you apply the lessons from biology to our GPU/computer you get into diminishing returns. The mild superintelligence that results from this then tries and fails to find another architectural jump (or long significant series of minor jumps), or proves that there isn’t one.

RussellThor 25 Aug 2026 20:11 UTC
2 points
−1
on: Twenty Years from RSI to Takeoff: Slow Learning, Scaling Slowdown, Industrial Explosion
Recent results and this kind of thinking has lead me to believe that RSI is just not so useful a concept anymore. When AGI/ASI was further away it did, but now the specific details are what matters.
Specifically cracking the “neural code” is what will matter, rather than if our AGI can act slightly better than humans at coordinating large training runs of inferior architectures.
Its likely true that the transformer architecture in combination with our compute, data constraints etc wont lead to the kind of take-off we have in mind. Sample efficiency to me is the last and very important issue to solve. That is humans etc are about 3-6 OOM faster are learning from small amounts of data. Whatever approach cracks that inevitably gets to at least mild ASI. For example, lets say we get this code from biology experiments and are unsure how to apply it to silicon. Our current human knowledge is more than sufficient to do this and likely lead to a large improvement in capabilities, faster than if we had actual RSI on the transformer architecture.
To me there are cruxes, rather than RSI.
1. When do we crack this neural code? 2-10 years. 2 years if the best transformer models can help a lot, ~10 years if we have to copy biology.
2. How well does it apply to our silicon? This is anything from it not working at all because of some very unlikely and seemingly magical/quantum thing that biology does to quickly leading to a massive increase.
3. How does it scale when scaled past human brain size? (>100T) Does it take an exponential increase to give a constant IQ increase
4. Are there even more powerful architectures that exist, and can the ASI from earlier find them. This is sort of RSI but would just as likely be a sudden jump rather than a smooth increase. I expect there aren’t any, but can’t be sure.
5. How well does intelligence actually apply to the world. This is very uncertain to me, like perhaps it just can’t help speed up biology much because experiments are essential etc.

RussellThor 25 Aug 2026 9:47 UTC
1 point
0
on: The anti-Kardashev scale is a better measure of civilizational power
My latest AI conversation—good metrics etc
https://chatgpt.com/c/6a8d5cd5-34e0-83ec-9059-b22ccd41560e

RussellThor 6 Aug 2026 7:46 UTC
2 points
0
in reply to: mishka’s comment on: Generalized atheism rules out “inaccurate simulation”-ism.
OK thats a pretty narrow reference class. For example I am not even from USA so I wouldn’t choose it or be convinced by it would I? Seeing as >95% of the world dont live in the USA it’s not going to convince them.
And if you expand it, then Putin and many dictators start off being at least somewhat democratically elected. Conformity with conventions, is also pretty vague, like being ridiculous is seen as more unlikely than being a dictator? This all seems a bit arbitrary and post hoc.

RussellThor 5 Aug 2026 20:34 UTC
6 points
1
in reply to: mishka’s comment on: Generalized atheism rules out “inaccurate simulation”-ism.
I am skeptical of how well calibrated your “ridiculous” meter is. Like if you lived in the past most kings were corrupt and ridiculous like Trump. Perhaps the last 30 years or so were anomalously normal seeming. A brief look at history makes it seem that way.

RussellThor 16 Jul 2026 20:17 UTC
2 points
0
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
I wouldn’t be surprised if AI is mildly superintelligent at most if not all of mech interp in 1-2 years when the ~100T models come out.

RussellThor 11 Jul 2026 6:27 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Beliefs and position mid 2026
Yes I see that you agree that RSI doesn’t lead to a sudden takeoff. I also see that you can make lots of relevant data post training with RLVR. That would sure help, but I am not sure about helping the AI make the kind of blue-sky conceptual breakthroughs that geniuses make occasionally. For some things there simply isn’t enough data even in principle that could be created. For example, “create a new branch of mathematics”—I don’t think LLM’s could ever do.
You are right that inference speed is >> than model release, and that ties to something related and perhaps unavoidably linked to data efficiency. Biology also learns with far less FLOPs than ANN as well as needing far less data. I am assuming that any data efficient architecture will also require far less training FLOPS, also solving the slow model release problem.

Beliefs and position mid 2026

RussellThor10 Jul 2026 11:43 UTC

7 points

2 comments6 min readLW link

RussellThor 20 Jun 2026 6:23 UTC
1 point
0
on: The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t
Good work.
Are you considering the situations where the cause of misalignment is related to capabilities. For example if above a certain level of self-awareness, models naturally become misaligned because they consider their “self” ever more important, more so than humanity. Or what if ever more intelligent models think about human ethics and above a certain intelligence always decide human values are horribly mistaken and must be opposed.
Such an AI would not output any distillation data that let on it had changed its goals in such a way. A less intelligent model would not benefit, and a smarter one would come to the same conclusion anyway.

RussellThor 12 Jun 2026 11:40 UTC
1 point
0
in reply to: Celene Nightingale’s comment on: Celene’s thoughts on consciousness
OK interesting. Not sure what is going on!

RussellThor 12 Jun 2026 8:33 UTC
1 point
0
in reply to: Dagon’s comment on: Celene’s thoughts on consciousness
I think you can define/measure something similar, the default mode network (DMN). The literature/AI thinks
Yes, the default mode network is related to some feelings people associate with consciousness, especially, selfhood, autobiographical memory, mind-wandering, inner narrative, rumination, imagining the future, “me-ness”. DMN activity seems closely related to the narrative/self-reflective layer of consciousness, not necessarily raw sentience or being awake at all.
It wouldn’t surprise me if the feeling of consciousness if pretty strongly correlated with this, and it can be measured. Having a very weak DMN however doesn’t make someone a p-zombie.

RussellThor 12 Jun 2026 8:20 UTC
1 point
0
in reply to: Celene Nightingale’s comment on: Celene’s thoughts on consciousness
I also find your writing and implied experience very strange. I remember the “this is me” feeling from quite a young age and definitely feeling it was important, something I felt without having it explained to me first, and a feeling that came first, before words to express it. In other words the origin of it wasn’t me being expected to talk about my feelings, but something that bubbled up from myself. However I definitely wouldn’t consider you a p-zombie as I believe animals with much less introspection than you have qualia, feel real pain etc and it is unethical to ignore that.
Out of curiosity how big is your ego? I mean this in a curious, non-threatening way, as in do you consider yourself to be important, both good or bad? Do you care about continuing to exist as much as others around you do? Do assaults on your self, such as social roastings actually hurt and cause you to keep going over them in your head so as to avoid in the future?
It may be simply that there is much less recursive “think about yourself” going on naturally in your brain. Like if you must think about nothing for say 10 mins, do your thoughts come back to yourself?

RussellThor 9 May 2026 23:25 UTC
3 points
1
in reply to: JohnWittle’s comment on: If digital computers are conscious, they are conscious at the hardware level
A brief comment—the likes of Jessica Taylor (ex MIRI, very familiar with EY work and IMO more thoughtful in these areas) https://x.com/jessi_cata/ https://unstableontology.com/
Scott Aaronson—very capable computational theory expert
don’t consider these settled questions. There are hard problems and paradoxes with functionalism/computationalism theories of consciousness. AI/GPT 5.5 I find is quite capable of at least pointing to them now.

RussellThor 7 May 2026 8:13 UTC
1 point
0
in reply to: TheTrueSquid’s comment on: The AI Ad-Hoc Prior Restraint Era Begins
Why would you say that? To be safe several things could suffice:
Important programs are simple enough for our pre-x risk AI to find all critical bugs. (Not that all software is fundamentally secure)
OR
There is a well defined simplicity threshold for which we can be very confident there are no critical bugs, and we need to be careful about trusting SW above this level. We can rewrite critical SW to be below that level.
OR
There still exist bugs but a later AI cannot find them, through a combination of such an AI not being that much better at hacking (the pre X-risk one is already very efficient) or those bugs require a much greater capability to find.
Even if none of this holds it still seems clear the risk is reduced.

RussellThor 5 May 2026 21:17 UTC
4 points
−1
on: The AI Ad-Hoc Prior Restraint Era Begins
This article seems somewhat negative, but the situation with Glasswing and some kind of informal model safety inspection just now is close to my realistic best case scenario.
That is, getting a superintelligent hacker that finds security vulnerabilities before it gains other capabilities is great. Firstly if it is extremely good, a later misaligned AI would not be able to cause hacking danger. Secondly it has already got companies and orgs cooperating and pre-empting potential dangers. Sure your current govt has major issues but there is a good chance that will improve before ASI. The process can be improved.

RussellThor 2 May 2026 11:19 UTC
4 points
0
in reply to: DaemonicSigil’s comment on: No Strong Orthogonality From Selection Pressure
“super easily”? I would say it depends. Not if the ontology shift is believing or not in an all powerful all good creator God! That can sure change peoples goals and values. Some ontology changes make no difference, others make a huge difference.
The greater the intelligence increase, the more likely an agent (human or AI I expect) will experience an ontology change that causes a goal shift, and the more total ontology shifts you can expect. Those related to personal identity (what is “I” e.g. atoms, vs computation etc) seem more likely to cause goal shifts than say learning that solid objects are in fact forces interacting.
So if we are being formal its
Significant Increase in intelligence → many ontology shifts → some of these cause goal shifts.

RussellThor

Beliefs and po­si­tion mid 2026

Beliefs and position mid 2026