Personally, I was never a fan of the FAE term because it seems to privilege environmental causes over dispositional ones without justification.
I get where you’re coming from, but I think most people need a nudge towards an environmental direction rather than towards the dispositional direction. FAE is I guess just a ‘reminder’ that environmental causes are even possible.
Fundamental attribution error applies to arguments as well. We often think of people who are quick to anger, standoffish, unwilling to accept hypotheticals, or deny “obvious” claims as intrinsically close-minded, foolish, or otherwise unreasonable. Instead those behaviours are often manifested due to other sources of persistent stress in their lives, feeling vulnerable/exposed in the moment, or even just a shitty mood.
Just to be clear, while I “vibe very hard” with what the author says on a conceptual level, I’m not directly calling for you to shut down those projects. I’m trying to explain what I think the author sees as a problem within the AI safety movement. Because I am talking to you specifically, I am using the immediate context of your work, but only as a frame not as a target. I found AI 2027 engaging, a good representation of a model of how takeoff will happen, and I thought it was designed and written well (tbh my biggest quibble is “why isn’t it called AI 2028″). The author is very very light on actual positive “what we should do” policy recommendations, so if I talked about that I would be filling in with my own takes, which probably differ from the author’s in several places. I am happy to do that if you want, though probably not publicly in a LW thread.
@Daniel Kokotajlo Addendum:
Finally, my interpretation of “Chapter 18: What Is to Be Done?” (and the closest I will come to answering your question based on the author’s theory/frame) is something like “the AGI-birthing dynamic is not a rational dynamic, therefore it cannot be defeated by policies or strategies that are focused around rational action”. Furthermore, since each actor wants to believe that their contribution to the dynamic is locally rational (if I don’t do it someone else will/I’m counterfactually helping/this intervention will be net positive/I can use my influence for good at a pivotal moment [...] pick your argument), further arguments about optimally rational policies only encourages the delusion that everyone is acting rationally, making them dig in their heels further.
The core emotions the author points to that motivate the AGI dynamic are: thrill of novelty/innovation/discovery, paranoia and fear about “others” (other labs/other countries/other people) achieving immense power, distrust of institutions, philosophies, and systems that underpin the world, and a sense of self importance/destiny. All of these can be justified with intellectual arguments but are often the bottom line that comes before such arguments are written. On the other hand the author also shows how poor emotional understanding and estrangement from one’s emotions and intuitions lead to people getting trapped by faulty but extremely sophisticated logic. Basically, emotions and intuitions offer first order heuristics in the massively high dimensional space of possible actions/policies, and when you cut off the heuristic system you are vulnerable to high dimensional traps/false leads that your logic or deductive abilities are insufficient to extract you from.
Therefore, the answer the author is pointing at is something like an emotional or frame realignment challenge. You don’t start arguing with a suicidal person about why the logical reasons they have offered for jumping don’t make sense (at least, you don’t do this if you want them to stay alive), you try to point them to a different emotional frame or state (i.e. calming them down and showing them there is a way out). Though he leaves it very vague, it seems that he believes the world will also need such a fundamental frame shift or belief-reinterpretation to actually exit this destructive dynamic, the magnitude of which he likens to a religious revelation and compares to the redemptive power of love. Beyond this point I would be filling in my own interpretation and I will stop there, but I have a lot more thoughts about this (especially the idea of love/coordination/ends to moloch).
To be honest, I wasn’t really pointing at you when I made the comment, more at the practice of the hedges and the qualifiers. I want to emphasise that (from the evidence available to me publicly) I think that you have internalised your beliefs a lot more than those the author collects into the “uniparty”. I think that you have acted bravely and with courage in support of your convictions, especially in face of the NDA situation, for which I hold immense respect. It could not have been easy to leave when you did.
However, my interpretation of what the author is saying is that beliefs like “I think what these people are doing might seriously end the world” are in a sense fundamentally difficult to square with measured reasoning and careful qualifiers. The end of the world and existential risk are by their nature so totalising and awful ideas that any “sane” interaction with them (as in, trying to set measured bounds and make sensible models) is extremely epistemically unsound, the equivalent of arguing whether 1e8 + 14 people or 1e8 + 17 people (3 extra lives!) will be the true number of casualties in some kind of planetary extinction event when the error bars are themselves +- 1e5 or 1e6. (We are, after all, dealing with never-seen-before black swan events.)
In this sense, detailed debates about which metrics to include in a takeoff model and the precise slope of the METR exponential curve and which combination of chip trade and export policies increases tail risk the most/least is itself a kind of deception. This is because the arguing over details implies that our world and risk models have more accuracy and precision than they actually do, and in turn that we have more control over events than we actually do. “Directionally correct” is in fact the most accuracy we’re going to get, because (per the author) Silicon Valley isn’t actually doing some kind of carefully calculated compute-optimal RSI takeoff launch sequence with a well understood theory of learning. The AGI “industry” is more like a group of people pulling the lever of a slot machine over and over and over again, egged on by a crowd of eager onlookers, spending down the world’s collective savings accounts until one of them wins big. By “win big”, of course, I mean “unleashes a fundamentally new kind of intelligence into the world”. And each of them may do it for different reasons, and some of them may in their heads actually have some kind of master plan, but all it looks like from the outside is ka-ching, ka-ching, ka-ching, ka-ching...
The ideal market-making move is to introduce a new necessity for continued existence, like water.
well, with nuance. Like, it’s not my ideal policy package, I think if I were in charge of the whole world we’d stop AI development temporarily and then figure out a new, safer, less power-concentrating way to proceed with it. But it’s significantly better by my lights than what most people in the industry and on twitter and in DC are advocating for. I guess I should say I approximately believe all those things, and/or I think they are all directionally correct
With all due respect, I’m pretty sure that the existence of this very long string of qualifiers and very carefully reasoned hedges is precisely what the author means when he talks about intellectualised but not internalised beliefs.
Information warfare and psychological warfare are well known terms. However, I would suggest that any well-intentioned outsider trying to figure out “what’s going on with AI right now” (especially in a governance context) is effectively being subject to the equivalent of an information state of nature (a la Hobbes). There are masses of opinions being shouted furiously, most of the public experts have giant glowing signs marked “I have serious conflicts of interest”, and the number of self-proclaimed insiders trying to get power/influence/money/a job at a lab/a slice of the lightcone by marketing insider takes is kind of deafening. And of course the companies are running targeted influencing/lobbying campaigns on top of all this, trying to position themselves as the sole reliable actors.
@Caleb Biddulph For future reference, what I meant by “set up other systems” is classical RL systems like vanilla Q-learning: https://www.geeksforgeeks.org/machine-learning/q-learning-in-python/ . Today we know Q-learning primarily as deep Q-learning (which was one of Deepmind’s original Big Papers), but it is entirely possible to do Q-learning with no neural networks to learn state representations or Q-values, instead just using a lookup table that matches state and action. This is pretty inefficient, for somewhat obvious reasons.
Like this post a lot—might have more detailed thoughts later but just wanted to park this here
I remember that you posted some variant of this idea as a short form or a post some time ago. I can see that you feel the idea is very important, and I want to respond to it on its terms. My quick answer is that even under the “same” morals, people can undertake quite destructive actions to others, because most choices are made under a combination of perceived values (moral beliefs) and perceived circumstances (factual beliefs). Longer answer follows:
C. S. Lewis once tried to create a rough taxonomy of worldwide moral standards, showing that ideas such as the golden rule (do unto others what you would have others do unto you) and variants like the inverse golden rule (do not do unto others what you would not have others do unto you) were surprisingly popular across cultures. This was part of a broader project which is actually quite relevant to discussions of transhumanism. He was arguing that what we would call eugenics and transformative technology would annihilate “moral progress”, but we can set aside the transhumanist argument for now and just focus on the shared principles—things like “family is good” or “killing is bad”.
First of all, it should be a bad sign for your plan that such common principles can be identified at all, since it suggests that people might already have similar morals but still come to different conclusions about what is to be done. Second, it becomes quickly clear that some shared moral principles might lead to quite strong conflicts: I’m thinking about morals like “me/my family/my ethnic group is superior and should come first/have the priority in situations of scarcity and danger”. If four different nations are led by governments with that same belief, and the pool of resources becomes limited, fighting will almost certainly break out. This is true even if cooperation would prevent the destruction of valuable limited resources, increasing total resources available to the nations as a whole!
From a broader perspective, what I see in the steelman of your idea is something like “if we get people to discuss, they will quickly realise that their ideas about what is moral are insufficient, and a better set of morals will emerge”. So they might converge on something like “we are all one global family, and we should all stick together even when we disagree because conflict is terrible”. However, this is where the circumstances part of the choice comes in. I can agree in principle that unity is good and that life is sacred. However, if I believe that someone else does not share those ethics, and is (for example) about to kill me or rob me, I might act in self defence. Most of us would call that justified, even though it violates my stated values. Today many leaders make lip service about respecting human rights and international norms… but it’s just that those evil evildoers are so evil that we need to do something serious to stop them. My values are pure, but circumstances forced my hand. And so on, and so forth.
Now, if you can truly convince everyone that everyone else is also a reasonable and nice human being, then maybe some progress can be made, but this is a very, very difficult thing, especially when there are centuries of conflict to deal with and legacies of complex and multilayered traumas in many parts of the world. So all in all I think this proposal is very unlikely to succeed. I hope this makes sense.
Some people in fact do such round trips (“road trips”) for personal enjoyment xD
Testingthewaters: “I believe that within 6 months this line of research [online in-sequence learning] will produce a small natural-language capable model that will perform at the level of a model like GPT-4, but with improved persistence and effectively no “context limit” since it is constantly learning and updating weights.”
Hey, just to clarify that this prediction has not expired yet. Thanks for collecting these predictions though! From the original post:
Concretely, these are the developments I am predicting within the next six months (i.e. before Feb 1st 2026) with ~75% probability:
And for those interested these are recent papers that continue to chart the trend:
[test time training family]
https://test-time-training.github.io/e2e.pdf (probably closest to the prediction but unsure how it scales to GPT-4 levels)
[tiny recursive model/hierarchical reasoning model family (which should really be called the Universal Transformers family)]
https://arxiv.org/pdf/2510.25741
https://arxiv.org/pdf/2512.14693
[other similar things]
The worked examples of how to measure/implement this policy make up the rest of the essay xD
From a play I probably will never finish:
I say, the lights are dim, and winking fast
Were that my heart were so alike and still!
Around me the rictus grins tell all:
Time herself diminishes, and the fates divide
And all of us, however dimly, recognise
Down one path, the broad and high way
By which destruction is serv’d to all
This path seems foretold, and star-divin’d
Yet my heart refuses to be still, and damn Hope to die
For down another, much less travel’d road
Where bravery and ill-reason take the place
Of logical, rationable cowardice
I spy some strange escape
If only we possess the high spirit to demand it
Reject the calculus of momentary gain and loss
And turn our eyes to the greater safety
Afforded by universal brotherhood
And is brotherhood not the natural state of every man?
I believe, by our deeds, we may yet not prove it otherwise.
Thank you for the reply and for your sincerity. I think my response as to “why would you not work at a capabilities lab” is something like “I worry about both the pragmatic and the existential risks quite a lot”, but that is more of a personal thought.
I have wavered a bit about whether to post this comment, or maybe make it a DM, or maybe not at all. I hope this does not feel like I’m doing some kind of personal attack. But tbh (as someone else pretty young who feels quite adrift right now) I find this post somewhat baffling. It is of course much easier to feel like “you will be okay” when you are a professor at Harvard who also has a well paid job at one of the companies riding the peak of the AI wave. You probably have more savings right now than I would accumulate with a decade more of “things as normal”, and you’re also attached to organisations that either already have a lot of institutional power or stand to gain much more by leading the development and deployment of a radical transformative technology.
By choosing not to work for AI capabilities labs (if we have the capability to get hired there, which I do not claim to be true for me), people who have relatively little career or financial capital are not only loosing out on prestige or fame. They are also losing out on security and power in a terrible job market and a world that seems increasingly both politically and socially dysfunctional. In this position and on this forum, for someone who has instead accepted that bargain and the accompanying risk of harm (whether you think your contribution is net-positive or not) to then tell us that “we will be fine” feels like being told by someone on a hill that we will be fine as a tsunami bears down on our seaside village. Perhaps the tsunami will be stronger than expected and drown everyone on the hill as well. But either way I would not want to be on the beach right now.
P.S. I do however endorse not acting based on panic, nihilism, or despair, and cultivating an attitude towards chance/randomness that allows for unexpected good outcomes as well as unexpected bad outcomes. Also, I understand why people would decide to work for a lab, given the circumstances surrounding capital, the emergent myth of the technology being crafted, and the clearly important and non-replaceable role powerful AI systems have in our information ecosystem already. Still, that doesn’t change my analysis regarding the feelings of powerlessness and helplessness.
Basically completely agreed, and I really like that last line.
Related essay: https://vitalik.eth.limo/general/2025/11/07/galaxybrain.html
My general opinion about “staying afloat” is that trying to secure only your own immediate slice of the pie/capital allocation/lightcone is a bit like trying to install tsunami proof fencing around your seaside property. Trying to secure your own slice in such a chaotic and high-variance time period is really really hard.
With that in mind the choice is clear: Either we pitch in to society’s collective defences (i.e. seawalls in this metaphor) or you are one of the lucky ones who had the privilege or foresight to be shielded from the turbulence (i.e. being born rich and living inland). Either way, not much hope trying to become a kingpin at the last minute.
Basically totally agreed, except I’d also argue that we’re not doing mostly okay together, and that we should work quite hard to make sure that we’re cooperating better before we advance any more down the mind tech tree.