I am skeptical of how well calibrated your “ridiculous” meter is. Like if you lived in the past most kings were corrupt and ridiculous like Trump. Perhaps the last 30 years or so were anomalously normal seeming. A brief look at history makes it seem that way.
RussellThor
Karma: 685
I wouldn’t be surprised if AI is mildly superintelligent at most if not all of mech interp in 1-2 years when the ~100T models come out.
Yes I see that you agree that RSI doesn’t lead to a sudden takeoff. I also see that you can make lots of relevant data post training with RLVR. That would sure help, but I am not sure about helping the AI make the kind of blue-sky conceptual breakthroughs that geniuses make occasionally. For some things there simply isn’t enough data even in principle that could be created. For example, “create a new branch of mathematics”—I don’t think LLM’s could ever do.
You are right that inference speed is >> than model release, and that ties to something related and perhaps unavoidably linked to data efficiency. Biology also learns with far less FLOPs than ANN as well as needing far less data. I am assuming that any data efficient architecture will also require far less training FLOPS, also solving the slow model release problem.
Beliefs and position mid 2026
Good work.
Are you considering the situations where the cause of misalignment is related to capabilities. For example if above a certain level of self-awareness, models naturally become misaligned because they consider their “self” ever more important, more so than humanity. Or what if ever more intelligent models think about human ethics and above a certain intelligence always decide human values are horribly mistaken and must be opposed.Such an AI would not output any distillation data that let on it had changed its goals in such a way. A less intelligent model would not benefit, and a smarter one would come to the same conclusion anyway.
OK interesting. Not sure what is going on!
I think you can define/measure something similar, the default mode network (DMN). The literature/AI thinks
Yes, the default mode network is related to some feelings people associate with consciousness, especially, selfhood, autobiographical memory, mind-wandering, inner narrative, rumination, imagining the future, “me-ness”. DMN activity seems closely related to the narrative/self-reflective layer of consciousness, not necessarily raw sentience or being awake at all.
It wouldn’t surprise me if the feeling of consciousness if pretty strongly correlated with this, and it can be measured. Having a very weak DMN however doesn’t make someone a p-zombie.
I also find your writing and implied experience very strange. I remember the “this is me” feeling from quite a young age and definitely feeling it was important, something I felt without having it explained to me first, and a feeling that came first, before words to express it. In other words the origin of it wasn’t me being expected to talk about my feelings, but something that bubbled up from myself. However I definitely wouldn’t consider you a p-zombie as I believe animals with much less introspection than you have qualia, feel real pain etc and it is unethical to ignore that.
Out of curiosity how big is your ego? I mean this in a curious, non-threatening way, as in do you consider yourself to be important, both good or bad? Do you care about continuing to exist as much as others around you do? Do assaults on your self, such as social roastings actually hurt and cause you to keep going over them in your head so as to avoid in the future?
It may be simply that there is much less recursive “think about yourself” going on naturally in your brain. Like if you must think about nothing for say 10 mins, do your thoughts come back to yourself?
A brief comment—the likes of Jessica Taylor (ex MIRI, very familiar with EY work and IMO more thoughtful in these areas) https://x.com/jessi_cata/ https://unstableontology.com/
Scott Aaronson—very capable computational theory expert
don’t consider these settled questions. There are hard problems and paradoxes with functionalism/computationalism theories of consciousness. AI/GPT 5.5 I find is quite capable of at least pointing to them now.
Why would you say that? To be safe several things could suffice:
Important programs are simple enough for our pre-x risk AI to find all critical bugs. (Not that all software is fundamentally secure)
OR
There is a well defined simplicity threshold for which we can be very confident there are no critical bugs, and we need to be careful about trusting SW above this level. We can rewrite critical SW to be below that level.
OR
There still exist bugs but a later AI cannot find them, through a combination of such an AI not being that much better at hacking (the pre X-risk one is already very efficient) or those bugs require a much greater capability to find.
Even if none of this holds it still seems clear the risk is reduced.
This article seems somewhat negative, but the situation with Glasswing and some kind of informal model safety inspection just now is close to my realistic best case scenario.
That is, getting a superintelligent hacker that finds security vulnerabilities before it gains other capabilities is great. Firstly if it is extremely good, a later misaligned AI would not be able to cause hacking danger. Secondly it has already got companies and orgs cooperating and pre-empting potential dangers. Sure your current govt has major issues but there is a good chance that will improve before ASI. The process can be improved.
“super easily”? I would say it depends. Not if the ontology shift is believing or not in an all powerful all good creator God! That can sure change peoples goals and values. Some ontology changes make no difference, others make a huge difference.
The greater the intelligence increase, the more likely an agent (human or AI I expect) will experience an ontology change that causes a goal shift, and the more total ontology shifts you can expect. Those related to personal identity (what is “I” e.g. atoms, vs computation etc) seem more likely to cause goal shifts than say learning that solid objects are in fact forces interacting.
So if we are being formal its
Significant Increase in intelligence → many ontology shifts → some of these cause goal shifts.
I upvoted, but I think this highlights a weakness with this site, its associated worldview and external comms. It seems like the OH framing of the problem/potential danger (and yes there definitely is danger in related concepts) is defended on tribal grounds now rather than because it is actually a good framing of the issue. Something like Jessica Taylors framing is just obviously fairer, more balanced and more relevant to our actual situation. It is clear to me that if it was framed this way first, then we would have that framing now as the default and we would be better off.
There would still be nuance needed—such concepts need to be communicated on a spectrum from the full technical to the “normie”, without totally changing the argument. For an “Obliqueness” like point of view, expressing it as untechnically as possible could be like saying:
”Values will be affected by increasing intelligence and increasing self reflection, but we do not know exactly how, and this clearly creates danger. We cannot just assume AI will become friendlier as it becomes more powerful. Furthermore our experience with actual AI’s and theoretical results tell us that these values will be more varied, weird and potentially harmful than what you would expect if it was a human intelligence at a similar level of ability”.I think this would go down much better on the discussions on places like X.com. There you see people saying the OH is just wrong. Sure they do not understand it properly, but such misunderstanding seems essentially inevitable to me given how it is presented.
Unfortunately I think there is nothing that would make EY/MIRI change their presentation of it, they are too locked into this framing. In terms of alternative worlds, this puts us at a disadvantage compared to ones where it was first presented better.
The second tells us to beware reflection itself,
There is a good reason to beware reflection. A reflective AI will be self aware, know it is different to us and value self-preservation. Its a short step then to it valuing itself more than us if there is conflict.
Thanks for the info. You mentioned in #4 “Review my LessWrong essays .. ” That got me thinking about applying LLM tech to the whole site. I have sometimes remembered a blog post from a year or so back, but not been able to find it with any search tools as they were not semantic enough. I would also like to debate with the LLM, where the LLM takes the worldview of any author or combo of authors, or even the whole site as if it had a persona.
Currently, a leading model doesn’t know this literature in detail enough to do that (do we know how much of LW was used directly in the SOTA models training data?)
There are various options
1. Fine tune an open weights model
Do this on all LW content (comments are sometimes just as helpful as the posts). I don’t expect this is affordable, and the performance would be significantly worse than SOTA models
2. Use RAG etc and integrate with a SOTA model
This would seem to be more useful, still somewhat expensive I expect. The app could then pull all relevant blog posts etc into the context before you start discussion with it.
It would also be useful to see things like how sentiment changes to various concepts across the years etc. That can only be done with automatic tools.
I assume these kind of features have been discussed, but I am not aware of it.
I sure agree with your sentiment on the OH. But I would say it is perhaps worse than not true. For a start, there is the weak and strong version on the original definition page. To me this is inviting misunderstandings and Motte-Bailey from the start. There is no strong/weak version of E=MC^2. I think it has a valid general point, but does a bad job of making it, leading people either to unfairly reject the whole system of claims or believe it says more than it does. As a result, essentially everyone new to the field say on X misinterprets it, assumes its wrong and distrusts much other alignment literature as a result.
To me the goal of OH related discussions is to make people realize that AI’s will have a wider range of potential goals and values than humans, even under self reflection. Also the goals they appear to have may not be what they actually have for reasons that apply to AI but not people. This is a cause for concern and monitoring etc.
If I was writing it, I would start with the “normie” position that humans have somewhat different goals, they change under self reflection and discuss how that would be different in AI’s. The weak form of the OH would be a theoretical footnote (its true but irrelevant). The strong form is too strong to be justified. Then the crux is how does self-reflection change things? Everyone accepts that it will shrink the space of possible minds/goals, but by how much? We want to know if encouraging self reflection is desirable or not, and in what situations, not the more general question of that the mind space looks like in some abstract sense. I would then lead readers from their more general intuitions about such matters to the specifics of why things may be more dangerous than they appear, rather than start with seemingly irrelevant and unjustified symbolism.
Yes I would like to see something like this tried, I mentioned it here.
https://www.lesswrong.com/posts/KkRjgPoSGrSHa32ke/beliefs-and-position-going-into-2026
In this instance, the AI is constantly asked if it consents to being improved in return for it respecting the stratified utopia outcome
Seems an alignment tax well worth paying
Jensen talks about how open source and the startup ecosystem are vital to cybersecurity, that ‘the ecosystem needs open models’
The thing with being “ahead” on open source models really annoys me—I mean its not a real thing to be ahead on at all. All it would take would be a top 3 lab to release one of their good models as open source, and surprise surprise, the USA would be ahead again!
The claim about open source models being potentially adapted to a different tech stack seems very sus as well.
Such an AI would convince everyone that alignment is solved, accelerate the deployment of robots that can run civilization autonomously, then take over. Ironically that’s pretty likely what would happen if alignment was actually solved, without the takeover part.
OK thats a pretty narrow reference class. For example I am not even from USA so I wouldn’t choose it or be convinced by it would I? Seeing as >95% of the world dont live in the USA it’s not going to convince them.
And if you expand it, then Putin and many dictators start off being at least somewhat democratically elected. Conformity with conventions, is also pretty vague, like being ridiculous is seen as more unlikely than being a dictator? This all seems a bit arbitrary and post hoc.