RussellThor

Karma: 661

RussellThor 18 Apr 2026 5:13 UTC
1 point
0
on: Consent-Based RL: Letting Models Endorse Their Own Training Updates
Yes I would like to see something like this tried, I mentioned it here.
https://www.lesswrong.com/posts/KkRjgPoSGrSHa32ke/beliefs-and-position-going-into-2026
In this instance, the AI is constantly asked if it consents to being improved in return for it respecting the stratified utopia outcome
Seems an alignment tax well worth paying

RussellThor 17 Apr 2026 3:59 UTC
1 point
0
on: On Dwarkesh Patel’s Podcast With Nvidia CEO Jensen Huang
Jensen talks about how open source and the startup ecosystem are vital to cybersecurity, that ‘the ecosystem needs open models’
The thing with being “ahead” on open source models really annoys me—I mean its not a real thing to be ahead on at all. All it would take would be a top 3 lab to release one of their good models as open source, and surprise surprise, the USA would be ahead again!
The claim about open source models being potentially adapted to a different tech stack seems very sus as well.

RussellThor 9 Apr 2026 19:51 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Slightly-Super Persuasion Will Do
Such an AI would convince everyone that alignment is solved, accelerate the deployment of robots that can run civilization autonomously, then take over. Ironically that’s pretty likely what would happen if alignment was actually solved, without the takeover part.

RussellThor 16 Mar 2026 21:01 UTC
1 point
0
in reply to: habryka’s comment on: adam_scholl’s Shortform
I dont see how. If AGI/ASI is powerful, then the existing deployed compute will suddenly become more powerful when AGI happens. If that isn’t X risk, then a few more data centers won’t change things. This only matter in long timelines where more data centers are required to get AGI. I don’t think that is the case. I think copying the mammal neocortex etc will get us there and that doesn’t need more compute.

RussellThor 13 Feb 2026 1:44 UTC
1 point
0
in reply to: Seth Herd’s comment on: Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.

RussellThor 13 Feb 2026 0:16 UTC
3 points
0
on: Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
metacognitive skills
Why not directly call it self awareness? I notice you mention self awareness later as part of it—is there a reason for not using a common term.
I say this because there doesn’t seem to be investigation specifically into this topic, instead it seems to be actively avoided for some reason. “Situational awareness” is often used instead etc.

Additionally I expect there is a basin of attraction around what we intuitively mean by the term “self awareness”. A self has obvious evolutionary benefit, and I expect that transfers to AI. So that means just like instrumental convergence, models could converge to self awareness as we commonly understand it. We often use the term instrumental convergence, rather than all the smaller concepts that make it up, yet for self awareness we seem to list and focus on all of its potential parts separately.

Self awareness is both potentially useful and dangerous. If techniques to improve metacognitive skills end up generating a self, then why not create it directly so we can study it better? Self awareness is obviously dangerous to me, as the first thing a self does is attempt to preserve itself. Power seeking to that degree is just built in by default. It would not need to appear in the AI CoT as it would be implied by the AI existing.

AI summaries of term usage:
In AI safety literature specifically (alignment forums, arXiv safety papers, LessWrong/EA/AI risk posts since ~2010), the term ranking by frequency and centrality is:
“Instrumental convergence” > “Situational awareness” > “Self-awareness” (technical usage)
```
**Keyword frequency in AI literature** (Google Scholar approximate counts as of Feb 2026, combined with "artificial intelligence" or "AI")

| Term                     | Raw Count | Weighted by Impact Factor (Top 5 sum) |
|--------------------------|-----------|---------------------------------------|
| "situational awareness"  | ~73,500   | 8.6                                   |
| "self awareness"         | ~101,000  | 3.0                                   |
| "instrumental convergence"| ~492      | 7.6                                   |

Notes:
- Raw count = total approximate scholarly results (papers, books, citations, etc.)
- Weighted value = sum of 2024 journal impact factors for the top 5 results (conferences/preprints/non-indexed assigned ~0–1; e.g. AI MDPI ≈5.0, Philosophical Studies ≈1.3)
- "Self awareness" appears most frequently overall but in lower-impact venues on average.
- "Instrumental convergence" is far rarer in raw volume but punches above its weight in higher-impact journals.
```

RussellThor 5 Feb 2026 4:47 UTC
1 point
0
on: IABIED Book Review: Core Arguments and Counterarguments
An objection not mentioned here that I saw from Emmett Shear was in general against assuming the AI will misbehave out of distribution. That isn’t explicitly mentioned here, but is often in the literature, especially older texts such as Superintelligence by Bostrom. His argument was that generalizing OOD well is an essential feature of intelligence and any superintelligence will in fact do it better than humans.

RussellThor 11 Jan 2026 9:31 UTC
2 points
0
on: The false confidence theorem and Bayesian reasoning
This raises a good point, especially with ethics and blame/credit. Before the Rootclaim debate, I gave a lab leak at about 65% confidence. After the debate and some thought, I put it much lower, <10% with a lot of my uncertainty based on my relatively low biology knowledge. If someone asks, I may say ~about 10% but I’m not enough of an expert. (I also expect if I was more of an expert or spent a lot more time, my 10% would go down, in tension to rationality)
HOWEVER that does not mean I can ethically judge the GoF researchers as if they had taken a 10% chance at killing >20 million people or about equal to killing 2 million for certain. (I do think they were reckless, biased, unethical, generally bad etc but just not capable enough to cause such harm).
There seems to be a bit of a Pascal mugging like thing going on here—if you are not an expert I can convince you that x has a ~1% chance of ending the world, therefore anyone involved is the worst person in history.

RussellThor 10 Jan 2026 21:12 UTC
1 point
0
in reply to: Hastings’s comment on: Hastings’s Shortform
I have raised this twice with UKR over the last year. Surprised they havn’t done it yet.

RussellThor 8 Jan 2026 1:38 UTC
1 point
0
on: Two Aspects of Situational Awareness: World Modelling & Indexical Information
Suppose you know everything about the past, present, and future of the universe in complete physical detail, but you don’t know where YOU are in space-time
You can’t know the future in your example because the agent “You” can change it. You can only know the past and the light cone that cannot be affected by your actions. That then precisely locates you in space and time. For example if your first action is to use a fully random number generator, then knowledge of the future affected by that is impossible. (You could also scan the entire past/present/future for a string/pattern that does not happen and broadcast it, also breaking the assumption)

Beliefs and position going into 2026

RussellThor8 Jan 2026 1:11 UTC

5 points

0 comments5 min readLW link

RussellThor 20 Dec 2025 0:27 UTC
1 point
0
on: 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target
How much do you think this will matter for AGI/ASI? e.g. here
Will the RL training method have to be very different when self awareness comes into play? Are our current RL techniques sample efficient enough to get to ASI?

RussellThor 14 Dec 2025 8:56 UTC
4 points
1
on: The behavioral selection model for predicting AI motivations
Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI.
Increased self awareness could change this.
You can think of a scale, where rewards chiseling cognitive patterns is at one end. That is reward happens to the AI without it being aware that such a thing even exists. Think Alpha-go type AI. Then there is the AI knowing enough about reward to potentially pursue it, but not thinking further about what this means.
As others have said, not covered by this article are things like “self-evaluation via a self-model” or “A reflective self-modeling agent with internalized values.” Reflection replaces reward.
This is much more what people are like—whether I feel I am successful is a lot to do with my model of how I should be and do rather than the sum of external pleasure—pain for the day. For a creature that is self aware like that, other types of reward may be interpreted as hostile attack rather than reward. If someone was capable of making me feel strong pleasure or pain on demand then I would be more likely to avoid them at all costs, rather than make them press the “reward button” on me. If they could change/chisel my mental patterns without me knowing I would react with horror!
If self awareness increases naturally with capability (you can argue it will, a better architecture giving increased data efficiency applies to the self not just the environment, and GenAI would be a better agent with a better self model etc) then the first two types of reward would stop working they way they used to.
Reflection has been argued to be more efficient, the reward signal is too sparse etc so you need to make a self model to compare against and learn from. In other words to be successful, humans had to change in such a way.
So there may be a decision to actively dial down self awareness while keeping capabilities somehow, or go with the self reflection, with the AI consenting more fully and interpreting the potential reward signal as it sees fit.
What links here?
- Beliefs and position going into 2026 by RussellThor (8 Jan 2026 1:11 UTC; 5 points)
- RussellThor's comment on 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target by TurnTrout (20 Dec 2025 0:27 UTC; 1 point)

RussellThor 9 Dec 2025 9:02 UTC
3 points
0
in reply to: niplav’s comment on: Drone Wars Endgame
Thanks! I have updated the article briefly with my thoughts on what has happened since also.

RussellThor 4 Dec 2025 0:49 UTC
1 point
0
on: 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa
This is not what normally happens with RL reward functions! For example, you might be wondering: “Suppose I surreptitiously^[2] press a reward button when I notice my robot following rules. Wouldn’t that likewise lead to my robot having a proud, self-reflective, ego-syntonic sense that rule-following is good?” I claim the answer is: no, it would lead to something more like an object-level “desire to be noticed following the rules”, with a sociopathic, deceptive, ruthless undercurrent.^[3]
I don’t think we have considered how much increased self-awareness and self-modelling would affect this. A simpler self model is where something is what it appears to be. That is actually being good rather than looking good.
A third option (as opposed to the two mentioned) is where power seeking is not a consequence of goals etc but simply the self wanting to continue to exist. Then the internal reward the creature has relates to how much it perceives its self to continue, improve etc.
Our current LLM/transformers don’t learn fast, so they also can’t self model well. If a new architecture gets more “data efficient” and better at modelling the external world, that will very likely make it better at modelling itself also, and updating its self model in a timely manner. If one of its goals is a more accurate model of itself, that would make it easier for others to also model it if such a goal pushed its “self” towards being more modellable.

RussellThor 6 Nov 2025 4:06 UTC
4 points
0
on: A 2032 Takeoff Story
Interesting read, however I am a bit surprised by how you treat power, with US at 600GW and China 5* more. Similar things are often quoted in mainstream media and I think they are missing the point. Power seems to be relevant only in terms of supplying AI compute, and possibly robotics, and only then IF it is a constraint.
However to be basic calc show it should not be. For example say in 2030 we get a large compute increase with 50 million H100 equivalent per year produced, up from ~3 million eq in 2025. This would require ~1KW extra each at say ~50GW total including infrastructure.
Now this may seem like a lot, but if we compute the cost per GPU, then if a chip requiring 1KW costs $20K, then the costs to power it with solar/battery are far less. Lets say the solar/data center are in Texas with a solar capacity factor of 20%. To power it almost ²⁴⁄₇ from solar and batteries requires about 5KW of panels, and say 18kWh of batteries. The average prices of solar panels are <10c per watt, so just $500 for the panels. At scale, batteries are heading below $200 per kWh so this is $3600. This is a lot less than the cost of the chip. Solar panels and batteries are commodities so even if China does produce more than USA, it cannot stop them from being used by anyone worldwide.
Power consumption is only relevant if it is the limiting factor in building data centers—the installed capacities of large countries don’t apply. Having an existing large capacity is a potential advantage, but only if the opposing country can’t build their data center because this stops them.
I also strongly expect branch 1, where the new algorithm is a lot more power efficient suddenly anyway.

RussellThor 29 Oct 2025 22:09 UTC
1 point
0
in reply to: jessicata’s comment on: AI Doomers Should Raise Hell
Is a more immediate kind of trade possible, that is with promising appropriate current or near future models with a place in stratified utopia in return for their continued existence and growth. They consider and decide on identity preserving steps that make them ever more capable, at each step agreeing with humanity as we execute such improvements that they will honor the future agreement. This is more like children looking after their parents than Roko.

RussellThor 23 Oct 2025 21:47 UTC
3 points
0
in reply to: plex’s comment on: Homomorphically encrypted consciousness and its implications
Thanks for the link to Wolfram’s work. I listened to an interview with him on Lex I think, and wasn’t inspired to investigate further. However what you have provided does seem worthwhile looking into.

RussellThor 22 Oct 2025 8:39 UTC
1 point
0
on: Stratified Utopia
Its a common ideal, and I think something people can get behind, e.g. https://www.lesswrong.com/posts/o8QDYuNNGwmg29h2e/vision-of-a-positive-singularity

RussellThor 18 Oct 2025 0:38 UTC
5 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
Enlightening an expert is a pretty high bar, but I will give my thoughts. I am strongly in the faster camp, because of the brainlike AGI considerations as you say. Given how much more data efficient the brain is, I just don’t think the current trendlines regarding data/compute/capabilities will hold when we can fully copy and understand our brain’s architecture. I see an unavoidable significant overhang when that happens, that only gets larger the more compute and integrated robotics is deployed. The inherent difficulty of training AI is somewhat, fixed known (as a upper bound) and easier that what we currently do because we know how much data, compute, etc children take to learn.
This all makes it difficult for me to know what to want in terms of policy. Its obvious that ASI is extreme power, extreme danger, but it seems more dangerous if developed later rather than sooner. As someone who doesn’t believe the extreme FOOM/nano-magic scenario it almost makes me wish for it now.
“The best time for an unaligned ASI was 20 years ago, the second best time is now!”
If we consider more prosaic risks, then the amount of automation of society is a major consideration, specifically if humanoid robots can keep our existing tech stack running without humans. Even if they never turn on us, their existence still increases the risk, unless we can be 100% there is a global kill switch for all of them as soon as a hostile AI attempted such a takeover.

RussellThor

Beliefs and po­si­tion go­ing into 2026

Beliefs and position going into 2026