RussellThor

Karma: 679

RussellThor 16 Jul 2026 20:17 UTC
2 points
0
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
I wouldn’t be surprised if AI is mildly superintelligent at most if not all of mech interp in 1-2 years when the ~100T models come out.

RussellThor 11 Jul 2026 6:27 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Beliefs and position mid 2026
Yes I see that you agree that RSI doesn’t lead to a sudden takeoff. I also see that you can make lots of relevant data post training with RLVR. That would sure help, but I am not sure about helping the AI make the kind of blue-sky conceptual breakthroughs that geniuses make occasionally. For some things there simply isn’t enough data even in principle that could be created. For example, “create a new branch of mathematics”—I don’t think LLM’s could ever do.
You are right that inference speed is >> than model release, and that ties to something related and perhaps unavoidably linked to data efficiency. Biology also learns with far less FLOPs than ANN as well as needing far less data. I am assuming that any data efficient architecture will also require far less training FLOPS, also solving the slow model release problem.

Beliefs and position mid 2026

RussellThor10 Jul 2026 11:43 UTC

7 points

2 comments6 min readLW link

RussellThor 20 Jun 2026 6:23 UTC
1 point
0
on: The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t
Good work.
Are you considering the situations where the cause of misalignment is related to capabilities. For example if above a certain level of self-awareness, models naturally become misaligned because they consider their “self” ever more important, more so than humanity. Or what if ever more intelligent models think about human ethics and above a certain intelligence always decide human values are horribly mistaken and must be opposed.
Such an AI would not output any distillation data that let on it had changed its goals in such a way. A less intelligent model would not benefit, and a smarter one would come to the same conclusion anyway.

RussellThor 12 Jun 2026 11:40 UTC
1 point
0
in reply to: Celene Nightingale’s comment on: Celene’s thoughts on consciousness
OK interesting. Not sure what is going on!

RussellThor 12 Jun 2026 8:33 UTC
1 point
0
in reply to: Dagon’s comment on: Celene’s thoughts on consciousness
I think you can define/measure something similar, the default mode network (DMN). The literature/AI thinks
Yes, the default mode network is related to some feelings people associate with consciousness, especially, selfhood, autobiographical memory, mind-wandering, inner narrative, rumination, imagining the future, “me-ness”. DMN activity seems closely related to the narrative/self-reflective layer of consciousness, not necessarily raw sentience or being awake at all.
It wouldn’t surprise me if the feeling of consciousness if pretty strongly correlated with this, and it can be measured. Having a very weak DMN however doesn’t make someone a p-zombie.

RussellThor 12 Jun 2026 8:20 UTC
1 point
0
in reply to: Celene Nightingale’s comment on: Celene’s thoughts on consciousness
I also find your writing and implied experience very strange. I remember the “this is me” feeling from quite a young age and definitely feeling it was important, something I felt without having it explained to me first, and a feeling that came first, before words to express it. In other words the origin of it wasn’t me being expected to talk about my feelings, but something that bubbled up from myself. However I definitely wouldn’t consider you a p-zombie as I believe animals with much less introspection than you have qualia, feel real pain etc and it is unethical to ignore that.
Out of curiosity how big is your ego? I mean this in a curious, non-threatening way, as in do you consider yourself to be important, both good or bad? Do you care about continuing to exist as much as others around you do? Do assaults on your self, such as social roastings actually hurt and cause you to keep going over them in your head so as to avoid in the future?
It may be simply that there is much less recursive “think about yourself” going on naturally in your brain. Like if you must think about nothing for say 10 mins, do your thoughts come back to yourself?

RussellThor 9 May 2026 23:25 UTC
3 points
1
in reply to: JohnWittle’s comment on: If digital computers are conscious, they are conscious at the hardware level
A brief comment—the likes of Jessica Taylor (ex MIRI, very familiar with EY work and IMO more thoughtful in these areas) https://x.com/jessi_cata/ https://unstableontology.com/
Scott Aaronson—very capable computational theory expert
don’t consider these settled questions. There are hard problems and paradoxes with functionalism/computationalism theories of consciousness. AI/GPT 5.5 I find is quite capable of at least pointing to them now.

RussellThor 7 May 2026 8:13 UTC
1 point
0
in reply to: TheTrueSquid’s comment on: The AI Ad-Hoc Prior Restraint Era Begins
Why would you say that? To be safe several things could suffice:
Important programs are simple enough for our pre-x risk AI to find all critical bugs. (Not that all software is fundamentally secure)
OR
There is a well defined simplicity threshold for which we can be very confident there are no critical bugs, and we need to be careful about trusting SW above this level. We can rewrite critical SW to be below that level.
OR
There still exist bugs but a later AI cannot find them, through a combination of such an AI not being that much better at hacking (the pre X-risk one is already very efficient) or those bugs require a much greater capability to find.
Even if none of this holds it still seems clear the risk is reduced.

RussellThor 5 May 2026 21:17 UTC
4 points
−1
on: The AI Ad-Hoc Prior Restraint Era Begins
This article seems somewhat negative, but the situation with Glasswing and some kind of informal model safety inspection just now is close to my realistic best case scenario.
That is, getting a superintelligent hacker that finds security vulnerabilities before it gains other capabilities is great. Firstly if it is extremely good, a later misaligned AI would not be able to cause hacking danger. Secondly it has already got companies and orgs cooperating and pre-empting potential dangers. Sure your current govt has major issues but there is a good chance that will improve before ASI. The process can be improved.

RussellThor 2 May 2026 11:19 UTC
4 points
0
in reply to: DaemonicSigil’s comment on: No Strong Orthogonality From Selection Pressure
“super easily”? I would say it depends. Not if the ontology shift is believing or not in an all powerful all good creator God! That can sure change peoples goals and values. Some ontology changes make no difference, others make a huge difference.
The greater the intelligence increase, the more likely an agent (human or AI I expect) will experience an ontology change that causes a goal shift, and the more total ontology shifts you can expect. Those related to personal identity (what is “I” e.g. atoms, vs computation etc) seem more likely to cause goal shifts than say learning that solid objects are in fact forces interacting.
So if we are being formal its
Significant Increase in intelligence → many ontology shifts → some of these cause goal shifts.

RussellThor 30 Apr 2026 21:01 UTC
5 points
−1
in reply to: lumpenspace’s comment on: No Strong Orthogonality From Selection Pressure
I upvoted, but I think this highlights a weakness with this site, its associated worldview and external comms. It seems like the OH framing of the problem/potential danger (and yes there definitely is danger in related concepts) is defended on tribal grounds now rather than because it is actually a good framing of the issue. Something like Jessica Taylors framing is just obviously fairer, more balanced and more relevant to our actual situation. It is clear to me that if it was framed this way first, then we would have that framing now as the default and we would be better off.
There would still be nuance needed—such concepts need to be communicated on a spectrum from the full technical to the “normie”, without totally changing the argument. For an “Obliqueness” like point of view, expressing it as untechnically as possible could be like saying:
”Values will be affected by increasing intelligence and increasing self reflection, but we do not know exactly how, and this clearly creates danger. We cannot just assume AI will become friendlier as it becomes more powerful. Furthermore our experience with actual AI’s and theoretical results tell us that these values will be more varied, weird and potentially harmful than what you would expect if it was a human intelligence at a similar level of ability”.
I think this would go down much better on the discussions on places like X.com. There you see people saying the OH is just wrong. Sure they do not understand it properly, but such misunderstanding seems essentially inevitable to me given how it is presented.
Unfortunately I think there is nothing that would make EY/MIRI change their presentation of it, they are too locked into this framing. In terms of alternative worlds, this puts us at a disadvantage compared to ones where it was first presented better.

RussellThor 30 Apr 2026 3:24 UTC
3 points
4
on: No Strong Orthogonality From Selection Pressure
The second tells us to beware reflection itself,
There is a good reason to beware reflection. A reflective AI will be self aware, know it is different to us and value self-preservation. Its a short step then to it valuing itself more than us if there is conflict.

RussellThor 21 Apr 2026 21:03 UTC
1 point
0
on: 10 non-boring ways I’ve used AI in the last month
Thanks for the info. You mentioned in #4 “Review my LessWrong essays .. ” That got me thinking about applying LLM tech to the whole site. I have sometimes remembered a blog post from a year or so back, but not been able to find it with any search tools as they were not semantic enough. I would also like to debate with the LLM, where the LLM takes the worldview of any author or combo of authors, or even the whole site as if it had a persona.
Currently, a leading model doesn’t know this literature in detail enough to do that (do we know how much of LW was used directly in the SOTA models training data?)
There are various options
1. Fine tune an open weights model
Do this on all LW content (comments are sometimes just as helpful as the posts). I don’t expect this is affordable, and the performance would be significantly worse than SOTA models
2. Use RAG etc and integrate with a SOTA model
This would seem to be more useful, still somewhat expensive I expect. The app could then pull all relevant blog posts etc into the context before you start discussion with it.
It would also be useful to see things like how sentiment changes to various concepts across the years etc. That can only be done with automatic tools.
I assume these kind of features have been discussed, but I am not aware of it.

RussellThor 20 Apr 2026 3:14 UTC
1 point
0
in reply to: dan.parshall’s comment on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026
I sure agree with your sentiment on the OH. But I would say it is perhaps worse than not true. For a start, there is the weak and strong version on the original definition page. To me this is inviting misunderstandings and Motte-Bailey from the start. There is no strong/weak version of E=MC^2. I think it has a valid general point, but does a bad job of making it, leading people either to unfairly reject the whole system of claims or believe it says more than it does. As a result, essentially everyone new to the field say on X misinterprets it, assumes its wrong and distrusts much other alignment literature as a result.
To me the goal of OH related discussions is to make people realize that AI’s will have a wider range of potential goals and values than humans, even under self reflection. Also the goals they appear to have may not be what they actually have for reasons that apply to AI but not people. This is a cause for concern and monitoring etc.
If I was writing it, I would start with the “normie” position that humans have somewhat different goals, they change under self reflection and discuss how that would be different in AI’s. The weak form of the OH would be a theoretical footnote (its true but irrelevant). The strong form is too strong to be justified. Then the crux is how does self-reflection change things? Everyone accepts that it will shrink the space of possible minds/goals, but by how much? We want to know if encouraging self reflection is desirable or not, and in what situations, not the more general question of that the mind space looks like in some abstract sense. I would then lead readers from their more general intuitions about such matters to the specifics of why things may be more dangerous than they appear, rather than start with seemingly irrelevant and unjustified symbolism.

RussellThor 18 Apr 2026 5:13 UTC
1 point
0
on: Consent-Based RL: Letting Models Endorse Their Own Training Updates
Yes I would like to see something like this tried, I mentioned it here.
https://www.lesswrong.com/posts/KkRjgPoSGrSHa32ke/beliefs-and-position-going-into-2026
In this instance, the AI is constantly asked if it consents to being improved in return for it respecting the stratified utopia outcome
Seems an alignment tax well worth paying

RussellThor 17 Apr 2026 3:59 UTC
1 point
0
on: On Dwarkesh Patel’s Podcast With Nvidia CEO Jensen Huang
Jensen talks about how open source and the startup ecosystem are vital to cybersecurity, that ‘the ecosystem needs open models’
The thing with being “ahead” on open source models really annoys me—I mean its not a real thing to be ahead on at all. All it would take would be a top 3 lab to release one of their good models as open source, and surprise surprise, the USA would be ahead again!
The claim about open source models being potentially adapted to a different tech stack seems very sus as well.

RussellThor 9 Apr 2026 19:51 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Slightly-Super Persuasion Will Do
Such an AI would convince everyone that alignment is solved, accelerate the deployment of robots that can run civilization autonomously, then take over. Ironically that’s pretty likely what would happen if alignment was actually solved, without the takeover part.

RussellThor 16 Mar 2026 21:01 UTC
1 point
0
in reply to: habryka’s comment on: adam_scholl’s Shortform
I dont see how. If AGI/ASI is powerful, then the existing deployed compute will suddenly become more powerful when AGI happens. If that isn’t X risk, then a few more data centers won’t change things. This only matter in long timelines where more data centers are required to get AGI. I don’t think that is the case. I think copying the mammal neocortex etc will get us there and that doesn’t need more compute.

RussellThor 13 Feb 2026 1:44 UTC
1 point
0
in reply to: Seth Herd’s comment on: Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.

RussellThor

Beliefs and po­si­tion mid 2026

Beliefs and position mid 2026