Wei Dai

Karma: 44,507

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 26 Jan 2026 2:07 UTC
12 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

And so more thoughtful utilitarians will defend justice as an instrumental moral good, albeit not as a terminal moral good. Unfortunately, it seems very hard to actually hold this position without in practice deprioritizing justice (e.g. it’s rare to see effective altruists reasoning themselves into trying to make society more just).

There’s an extensive literature in economics on optimal punishment. Does that count, as far as utilitarians working on justice as an instrumental good?

For example, before trying to plan for the future, you need to have a sense of personal identity whereby your future self will feel a sense of continuity with and loyalty to your plans.

I think we just need our terminal values to not change too much over time, so if I ever feel like I need to rethink my plans, I’ll come up with a similar or even better plan. Is your thinking that this is impossible or infeasible for most humans, due to things like “power corrupts”? If so, I think consequentialism is still good as it lets us manage or mitigate such value drift, e.g., if I can foresee power (or other circumstances) corrupting my values, I can take precautions like avoiding getting into those situations?

Linking this to your other recent shortform, how could Paul have avoided other people misusing his work, except by doing better consequentialism (i.e., foreseeing this consequence and doing something ahead of time to mitigate it)? Are you not applying consequentialism in predicting the possible downside of one research/communications approach and adopting a different approach based on this prediction?

Wei Dai 22 Jan 2026 2:30 UTC
20 points
2
on: How (and why) to read Drexler on AI

There are a lot of important questions that Eric doesn’t say much about.

How to explain this? (I note that he not only fails to proactively address these questions, but also ignores them when others raise them, which seems totally inexplicable to me. Or at least this was my experience when I participated in the CAIS discussion when that came out.)

Wei Dai 22 Jan 2026 2:14 UTC
LW: 4 AF: 4
2
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
How does the Shareholder Value Revolution fit into your picture? From an AI overview:
1. The Intellectual Origins (The 1970s)

The revolution was born out of economic stagnation in the 1970s. As U.S. corporate profits dipped and competition from Japan and Germany rose, economists and theorists argued that American managers had become “fat and happy,” running companies for their own comfort rather than efficiency.

Two key intellectual pillars drove the change:
- Milton Friedman (The Moral Argument): In a famous 1970 New York Times essay, Friedman argued, “The social responsibility of business is to increase its profits.” He posited that executives spending money on “social causes” (like keeping inefficient plants open to save jobs) were essentially stealing from the owners (shareholders).
- Jensen and Meckling (The Economic Argument—“Agency Theory”): In 1976, these economists published a paper describing the “Principal-Agent Problem.” They argued that managers (agents) were not aligned with shareholders (principals). Managers wanted perks (corporate jets, large empires), while shareholders wanted profit. The solution? Align their interests by paying executives in stock.
It seems to better fit my normative picture of human values: terminal values come from philosophy, and subservience of instrumental values to terminal values improves over time as we get better at it, without need to permanently raise instrumental values to terminal status or irreversibly commingle the two.

Wei Dai 21 Jan 2026 1:52 UTC
2 points
0
on: Money Can’t Buy the Smile on a Child’s Face As They Look at A Beautiful Sunset… but it also can’t buy a malaria free world: my current understanding of how Effective Altruism has failed
Thank you, this is helpful to clarify and remind for me what Ben was trying to say.

Through a mix of ambient cultural pressures silencing or warping the clarity of good meaning folk

Do you or Ben have a more detailed explanation of what happened here? What can/should “good meaning folk” do to prevent this? Should I personally be worried about something like this?

Wei Dai 11 Jan 2026 0:24 UTC
4 points
0
in reply to: TsviBT’s comment on: Wei Dai’s Shortform
I’m pretty confused by your comment. Surely there are arguments other than wastefulness for not having cycles in one’s terminal/intrinsic values? Like if I prefer to tile the universe with qualia A more than qualia B, and prefer B to C, and C to A, how do I actually make the decision of what qualia to tile the universe with?

Wei Dai 11 Jan 2026 0:11 UTC
LW: 16 AF: 7
1
AF
on: Wei Dai’s Shortform
Possible root causes if we don’t end up having a good long term future (i.e., realize most of the potential value of the universe), with illustrative examples:
1. Technical incompetence
  - We fail to correctly solve technical problems in AI alignment.
  - We fail to build or become any kind of superintelligence.
  - We fail to colonize the universe.
2. Philosophical incompetence
  - We fail to solve philosophical problems in AI alignment
  - We end up optimizing the universe for wrong values.
3. Strategic incompetence
  - It is not impossible to cooperate/coordinate, but we fail to figure out how.
  - We fail to have other important strategic insights
    E.g., related to whether it’s better in the long run to build AGI first, or enhance human intelligence first
  - We have the insights but fail to make use of them correctly.
4. Actual impossibility of cooperation/coordination
  - It is actually rational or right in some sense to underspend on AI safety while racing for AGI/ASI, and nothing can be done about this.
Is this missing anything, or perhaps not a good way to break down the root causes? The goal for this includes:
1. Having a high-level schema for what someone could work on if they care about having a good long term future.
2. Talking about how likely a good long term future is, e.g., how competent is humanity in each of these areas relative to the likely challenge.

Wei Dai 8 Jan 2026 23:22 UTC
67 points
9
on: Wei Dai’s Shortform
“Utility” literally means usefulness, in other words instrumental value, but in decision theory and related fields like economics and AI alignment, it (as part of “utility function”) is now associated with terminal/intrinsic value, almost the opposite thing (apparently through some quite convoluted history). Somehow this irony only occurred to me ~3 decades after learning about utility functions.

Wei Dai 8 Jan 2026 21:49 UTC
3 points
−1
in reply to: RogerDearnaley’s comment on: My 2003 Post on the Evolutionary Argument for AI Misalignment
Isn’t living in cities itself driven at least in part by memetics (e.g., glamour/appeal of city living shown on TV/movies)? Certainly memes can cause people to not live in cities, e.g., the Amish or the meme of moving out to the suburbs to raise kids.

Wei Dai 8 Jan 2026 19:23 UTC
5 points
0
in reply to: gwern’s comment on: HIA and X-risk part 2: Why it hurts
Oops, thought I could trust “reasoning” AI (Gemini 3 Pro) for such a simple seeming question. Had it redo the estimate taking your comment into account, and it came up with 1m assuming N(90,15) globally, which still felt wrong, so I had it redo the estimate using country-level data, and it ended up with 7.5m with 6.1m in East Asia, 1.1m in the West, and .3m in RoW. This assumed N(105,15) for East Asia (so not quite using country-level data), which Opus and GPT point out might be an overestimate due to China being a bit lower than this. Had them redo the EA estimate using country-level data and they came up with 4.5m and 5.5m for EA (using N(103,15) and N(104,15) for China) respectively.

This is actually a significant update for my mental world model, as I didn’t previously realize that China had more than half of the world’s population of IQ>145 people.

Wei Dai 8 Jan 2026 8:22 UTC
6 points
0
on: HIA and X-risk part 2: Why it hurts
It looks like Part 1 was never cross-posted to LW. Please fix this?

Wei Dai 8 Jan 2026 8:19 UTC
24 points
10
on: HIA and X-risk part 2: Why it hurts
Assuming I have an IQ of 145, there are ~11 million people on Earth smarter (have a higher IQ) than me, but almost none of them, including e.g. Terence Tao, are trying to do something about AI x-risk even at this late date. Updating on this has to move one away from HIA directionally, right, versus the prior 10 years ago?
10 years ago you could say that those ~11m have just never thought about AI, but today the conclusion seemingly has to be that strategic competence is surprisingly little correlated with, or not much scaled by, intelligence, which if true would mean that HIA wouldn’t do much for the key bottleneck of humanity’s strategic incompetence^[1], but could easily make things worse by creating more and harder strategic problems.
Yeah, so I think this is probably my most salient crux at this point: what does the “strategic competence landscape” look like after significant HIA has occurred?
1. ^
  As evidenced by, aside from the neglect of AI safety, how much time it has taken for very obvious strategic arguments/considerations to be written down by anyone, and how late we’re having a proper AI-first vs IA-first debate.

Wei Dai 6 Jan 2026 22:10 UTC
LW: 10 AF: 5
3
AF
on: 1. The CAST Strategy

A less-spooky solution might involve the principal simply asking the agent to write a comprehensive guide to building a truly friendly AGI which would be aligned with human values in a way that was robustly good, then follow that guide (with the corrigible agent’s help) to produce an aligned, superintelligent sovereign.

Please take a look at A Conflict Between AI Alignment and Philosophical Competence (especially the last paragraph, about corrigibility), which is in part a reaction to this.

My 2003 Post on the Evolutionary Argument for AI Misalignment

Wei Dai6 Jan 2026 20:45 UTC

37 points

7 comments2 min readLW link

A Conflict Between AI Alignment and Philosophical Competence

Wei Dai27 Dec 2025 21:32 UTC

68 points

13 comments2 min readLW link

Wei Dai 13 Dec 2025 22:57 UTC
LW: 3 AF: 3
0
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I’m not surprised by this, my sense is that it’s usually young people and outsiders who pioneer new fields. Older people are just so much more shaped by existing paradigms, and also have so much more to lose, that it outweighs the benefits of their expertise and resources.
All of the fields that come to my mind (cryptography, theory of computation, algorithmic information theory, decision theory, game theory) were founded by much more established researchers. (But on reflection these all differ from AI safety by being fairly narrow and technical/mathematical, at least at their founding.) Which fields are you thinking of, that were founded by younger people and outsiders?
Perplexity AI Pro (with GPT-5.1-Thinking)’s answer to “Who were the founders of academic cryptography research as a field and what where their jobs at the time?”
There isn’t a single universally agreed-on “founder” of academic cryptography. Instead, a small group of researchers in the 1940s–1970s are usually credited with turning cryptography into an open, university-based research field.
No single founder
Histories of the subject generally describe a progression: Claude Shannon’s mathematical theory of secrecy in the 1940s, followed by the public‑key revolution of the 1970s and early 1980s that created today’s academic cryptography community. Shannon’s work was foundational, but it did not yet create an academic field in the modern sense; that came later with Whitfield Diffie, Martin Hellman, Ralph Merkle, and the inventors of RSA, whose work is often described as pioneering “modern” cryptography and has been recognized by ACM Turing Awards for cryptography pioneers.wikipedia+1
Early mathematical groundwork
Claude Shannon is widely regarded as the founder of mathematical cryptography; in the 1940s he worked at Bell Labs as a researcher, where he developed the information‑theoretic framework for secrecy systems that later influenced public‑key cryptography. At roughly the same time and into the 1960s, cryptography research also existed in industry—most notably at IBM, where Horst Feistel headed an internal cryptography research group that designed ciphers such as Lucifer, which evolved into the Data Encryption Standard (DES), but this work was largely not yet an open academic discipline.research.ibm+1
Founders of modern academic cryptography
Most accounts of “academic cryptography as a field” point first to the group around Stanford in the 1970s, whose work on public‑key ideas made cryptography a mainstream research topic in universities. In that period, the key people and their roles were approximately:sandilands+2
Whitfield Diffie – A researcher working with Martin Hellman at Stanford when they introduced public‑key cryptography in their 1976 work; he had come from earlier industry and research positions and was not yet a long‑tenured professor at that time.
Martin Hellman – A faculty member (electrical engineering professor) at Stanford University, supervising cryptography research and collaborating with Diffie on the new public‑key paradigm.sandilands
Ralph Merkle – A graduate student whose “Merkle puzzles” were developed first as an undergraduate project and then as part of his PhD work under Hellman at Stanford, making him one of the student‑level founders of modern academic cryptography.sandilands
These three are often singled out as the core founders of modern, open, university‑based cryptography research because their work shifted cryptography from a mostly classified or industrial activity into a widely studied academic topic.awards.acm+2
RSA and later theoretical founders
Immediately after the Diffie–Hellman–Merkle work, Ronald Rivest, Adi Shamir, and Leonard Adleman introduced RSA while they were all at MIT, with Rivest and Shamir as faculty members and Adleman as a researcher/graduate student; this solidified public‑key cryptography as a central academic topic and led to sustained university research groups in cryptography. Their roles at MIT and the impact of RSA meant that, alongside the Stanford group, they are commonly treated as co‑founders of modern academic cryptography.
In the 1980s, researchers such as Manuel Blum, Shafi Goldwasser, and Silvio Micali (then faculty and graduate students at Berkeley and MIT) developed the complexity‑theoretic foundations of cryptography—zero‑knowledge proofs, formal security definitions, and provable security—which are why they are frequently listed among the leading theoreticians and foundational figures of the field in modern cryptographer lists.wikipedia
https://en.wikipedia.org/wiki/History_of_cryptography
https://en.wikipedia.org/wiki/List_of_cryptographers
https://en.wikipedia.org/wiki/International_Cryptology_Conference
https://www.reddit.com/r/crypto/comments/6loygx/who_is_your_favorite_cryptography_researcher_and/
https://cryptography.isg.rhul.ac.uk/history.html
https://sandilands.info/crypto/OrganisationsandPeopleinCryptography.html
https://research.ibm.com/topics/cryptography
https://www.ibm.com/think/topics/cryptography-history
https://cryptoanarchy.wiki/people-and-organisations/pioneers-of-modern-cryptography
https://en.wikipedia.org/wiki/Cryptologist
https://www.reddit.com/r/cybersecurity/comments/1oo653i/ama_were_cryptography_and_security_researchers/
https://awards.acm.org/about/2015-turing
https://www.reddit.com/r/crypto/comments/393r8g/where_is_the_best_research_in_cryptography/
“What about theory of computation, algorithmic information theory, decision theory, game theory? You can be more concise with these.”
Each of these fields has several widely recognized “founders,” mostly mathematicians, logicians, and economists working in universities or research institutes. Below is a concise mapping from field → key founders → what they were professionally doing at the time.
Theory of computation
Modern theory of computation is usually traced to Alonzo Church and Alan Turing in the 1930s, along with related work by Kurt Gödel. Church was a logician and professor of mathematics at Princeton University when he introduced the lambda calculus and the Church–Turing thesis–style view of effective calculability. Turing was a young mathematician at Cambridge University when he published his 1936 paper defining Turing machines and formalizing computation, later becoming widely regarded as a founder of theoretical computer science.wikipedia+1
Algorithmic information theory
Algorithmic information theory was founded by Ray Solomonoff, an American mathematician who, working largely as an independent researcher on algorithmic probability and inductive inference, first presented the core ideas at a Caltech conference and in a 1960 report on a “General Theory of Inductive Inference.” It was then independently and almost simultaneously redeveloped by Andrey Kolmogorov, a leading Soviet mathematician, and Gregory Chaitin, a young mathematician working in industrial research, both of whom formalized similar notions of algorithmic complexity.wikipedia+2
Game theory
Modern game theory is generally credited to John von Neumann and Oskar Morgenstern. Von Neumann, a Hungarian mathematician working in pure and applied mathematics (including at Princeton and later the Institute for Advanced Study), introduced the foundational minimax theorem for zero‑sum games and then co‑authored Theory of Games and Economic Behavior in 1944 with Morgenstern, who was at the time an economist and professor (later at Princeton), thereby establishing game theory as a major branch of economics and social science.cmu+2
Decision theory
Normative decision theory in economics is typically traced to the von Neumann–Morgenstern expected‑utility framework, developed in the same 1944 book that founded game theory. In that work, von Neumann (mathematician) and Morgenstern (economist) provided an axiomatic treatment of rational choice under uncertainty, which is widely treated as the foundational formulation of modern decision theory in economics and statistics.wikipedia+2
https://en.wikipedia.org/wiki/Alan_Turing
https://en.wikipedia.org/wiki/Theory_of_computation
https://en.wikipedia.org/wiki/Ray_Solomonoff
https://en.wikipedia.org/wiki/Algorithmic_information_theory
https://en.wikipedia.org/wiki/Algorithmic_information_theory?oldformat=true
https://www.cmu.edu/dietrich/philosophy/research/areas/science-methodology/rational-choice.html
https://en.wikipedia.org/wiki/Oskar_Morgenstern
https://en.wikipedia.org/wiki/Game_theory
https://thebrooklyninstitute.com/items/courses/new-york/alan-turing-algorithms-computation-machines/
https://www.reddit.com/r/compsci/comments/4vhxnk/why_is_turing_considered_the_father_of_computer/
https://people.idsia.ch/~juergen/goedel-1931-founder-theoretical-computer-science-AI.html
https://www.cs.cmu.edu/~lblum/PAPERS/AlanTuring_and_the_Other_Theory_of_Computation.pdf
https://principlesofcomputation.wordpress.com/what-is-theory-of-computation/
https://christosaioannou.com/History%20of%20Game%20Theory.pdf
https://www.youtube.com/watch?v=CVw1j6pFQ5o
http://www.scholarpedia.org/article/Algorithmic_information_theory
https://en.wikipedia.org/wiki/Theoretical_computer_science
https://www.cs.bu.edu/fac/lnd/research/al-i.htm
https://plato.stanford.edu/entries/game-theory/
https://bgibhopal.com/what-is-the-theory-of-computation-and-why-is-it-important-in-computer-science/
Though I guess the thing I’m pointing at could also be summarized as “why hasn’t someone created a new paradigm of AI safety in the last decade?”
Creating a new paradigm within an existing field seems different enough from creating a new field that the important factors might differ a lot. Also, by asking this question it seems like you’re assuming that someone should have created a new paradigm of AI safety in the last decade, which a lot of people would presumably disagree with (because they either think the existing paradigms are good enough, or this is just too hard technically). (Basically I’m suggesting it may be hard to interest people in this question, until someone has created such a paradigm, and then you can go back and say “why didn’t someone do this earlier?”)

Wei Dai 9 Dec 2025 0:03 UTC
7 points
1
in reply to: Cole Wyeth’s comment on: Eliezer’s Unteachable Methods of Sanity
Do you believe in a quantum multiverse, or a spatially infinite universe (beyond the observable universe)? You can get a similar conclusion with either of these (which are Tegmark Levels 3 and 1, respectively).

Wei Dai 8 Dec 2025 2:07 UTC
5 points
−8
in reply to: Cole Wyeth’s comment on: Eliezer’s Unteachable Methods of Sanity
Try applying:
- Is the potential astronomical waste in our universe too small to care about?
- Shut Up and Divide?
Also recall that we’re in a tiny tiny corner of Reality (whatever Tegmark level it is, it’s probably much larger than what we can see), and it’s pretty unclear how to update EU(Reality | human history).

Wei Dai 7 Dec 2025 11:55 UTC
33 points
1
in reply to: Kaj_Sotala’s comment on: Eliezer’s Unteachable Methods of Sanity
After reading this article by a human historian (Bill Black), I think there’s a number of inaccuracies in Claude’s account above, but the key point I wanted to verify is that Truman’s reaction happened after just that one sentence by Oppenheimer (which in my mind seems like an appropriate expression of reflection/remorse, not being a drama queen, if he didn’t do or say anything else “dramatic”), and that does seem to be true.
The author’s conclusions, which seems right to me:
He, the president, dropped the bomb, not Oppenheimer. How dare this scientist — this government employee — assume the guilt for the greatest weapon ever used in human history? How dare he make himself the hero, albeit a tragic one?
I think Nolan got this right — this was what really annoyed Truman about Oppenheimer’s comment. By assuming guilt for the bomb, Oppenheimer was taking credit for it. And Truman resented this. He wanted the credit for dropping the bomb and saving American lives, whatever bloodguilt that may have entailed.

Wei Dai 7 Dec 2025 10:53 UTC
LW: 8 AF: 7
3
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Given that Vernor Vinge wrote The Coming Technological Singularity: How to Survive in the Post-Human Era in 1993, which single-handedly established much of the memeplex, including the still ongoing AI-first vs IA-first debate, another interesting question is why didn’t anyone found the AI safety field until around 2000.

For me, I’m not sure when I read this essay, but I did read Vinge’s A Fire Upon the Deep in 1994 as a college freshman, which made me worried about a future AI takeover, but (as I wrote previously) I thought there would be plenty of smarter people working in AI safety so I went into applied cryptography instead (as a form of d/acc). Eliezer after reading Vinge (as a teen) didn’t immediately heed the implicit or explicit safety warnings and instead wanted to accelerate the arrival of the Singularity as much as possible. It took him until around 2000 to pivot to safety. Nick Bostrom I think was concerned from the beginning or very early, but he was a PhD student when he got interested and I guess it took him a while to work through the academic system until he could found FHI in 2005.

Maybe the real question is why didn’t anyone else, i.e., someone with established credentials and social capital, found the field. Why did the task fall to a bunch of kids/students? The fact that nobody did it earlier does seem to suggest that it takes a very rare confluence of factors/circumstances for someone to do it.

(Another tangential puzzle is why Vinge himself didn’t get involved, as he was a professor of computer science in addition to science fiction writer. AFAIK he stayed completely off the early mailing lists as well as OB/LW nor had any contacts with anyone in AI safety.)

Wei Dai 7 Dec 2025 5:58 UTC

LW: 5 AF: 4

in reply to: TsviBT’s comment on: Relitigating the Race to Build Friendly AI

Let me rephrase. I already believed that there had been a plan originally, like 2004 +/- 3 years, to make sovereign AI. When I was entering the field, I don’t recall thinking too hard about “what do you do with it”, other than thinking about a Task-ish thing, but with Sovereign AI as a good test case for thought experiments. I don’t know when Yudkowsky and others updated (and still don’t).

I got a free subscription to Perplexity Pro, and used it to generate a report about this. The result seems pretty good. The short answer to “when Yudkowsky updated (from Sovereign to Task AGI)” is very likely 2014 or 2015 (see second collapsed section below).

Perplexity AI research report on “using this post and comment and linked resources as a starting point, can you build a comprehensive picture of SIAI/MIRI’s strategy over time, especially pinpointing the specific dates that its strategy changed/pivoted, backed up by the best available evidence
https://www.lesswrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai″

SIAI/MIRI’s strategy over time: phases and pivot points

The Machine Intelligence Research Institute (MIRI), originally the Singularity Institute for Artificial Intelligence (SIAI), has changed strategy multiple times.

Over 25 years, the strategy has moved through roughly these phases:

2000–2003: “Build the Seed AI ourselves, as fast as possible” (accelerationist + Friendly AI).
~2003–2010: “Friendly AI and existential risk” + heavy outreach, still with a strong self‑image as the eventual FAI builder.
2011–2013: Professionalisation and pivot to technical FAI research; formal mission broadened from “build FAI” to “ensure AI turns out well,” though key people still expected MIRI (or a tight equivalent) to build a world‑altering Friendly AI.
2013–2016: Mature “world‑altering Friendly AI” / CEV vision plus growing emphasis on Task‑AGI and “pivotal acts.”
2017–2020: New engineering‑heavy, largely confidential research program aimed at new foundations for alignable AI, with a clear plan around “minimal aligned AGI” performing a pivotal act.
2020–2022: Public admission that the 2017 program had “largely failed” and a period of strategic regrouping; increasingly pessimistic public tone (e.g. AGI Ruin).
2023–present: Official pivot to policy and communications first, with research now framed as secondary support for an “international halt” to progress toward smarter‑than‑human AI.

Below is a more detailed, evidence‑based timeline, with particular attention to when strategy clearly shifted.

1. Founding: “Create a Friendly, self‑improving AI” (2000–2003)

Founding mission (July 2000).
The organization is founded on 27 July 2000 as the Singularity Institute for Artificial Intelligence by Eliezer Yudkowsky and Brian & Sabine Atkins. The mission statement on the early IRS forms and timeline is explicit:

“Create a Friendly, self‑improving Artificial Intelligence.”timelines.issarice

Early technical writings like Coding a Transhuman AI and Creating Friendly AI 1.0 (2001) describe specific architectures for a transhuman AI that self‑improves while remaining benevolent. This is not just “study risks”; it is a plan to design the Seed AI.intelligence+1

Early accelerationism.
MIRI’s own 2024 retrospective states that at founding, the goal was:

“to try to accelerate to smarter‑than‑human AI as quickly as possible, on the assumption that greater‑than‑human intelligence entails greater‑than‑human morality.”intelligence

So the earliest SIAI strategy can be summarized as:

Build a self‑improving Seed AI (with Friendly properties).
Speeding up AGI is seen as good, provided it is Friendly.

Approximate pivot:
By MIRI’s own later account, Eliezer’s “naturalistic awakening” — realizing that superintelligence would not automatically be moral — occurs between 2000–2003, and:

“MIRI shifted its focus to the alignment problem around 2003.”intelligence+1

So around 2003 there is a conceptual pivot from “accelerate AGI, assume it’s benevolent” to “AI alignment is a hard, central problem.”

2. Mid‑2000s: Friendly AI + outreach, still as would‑be FAI builders (2003–2010)

Mission reformulation (2007 and 2010).
By 2007 the formal mission is updated to:

“developing safe, stable, and self‑modifying Artificial General Intelligence.”timelines.issarice

In 2010 it further becomes:

“To develop the theory and particulars of safe self‑improving Artificial Intelligence… and otherwise improve the probability of humanity surviving future technological advances.”timelines.issarice

This is already less “we will build it” and more “develop theory and improve odds,” but still assumes SIAI as central technical actor.

Strategic plan 2011: three‑pronged strategy.
The 2011 Strategic Plan lays out three “core strategies”:

“Solve open research problems in Friendly AI theory.”
“Mobilize additional human capital and other resources” (Singularity Summit, LessWrong, volunteers, outreach).
“Improve the function and capabilities of the organization.”intelligence

Near‑term (2011–2012) priorities emphasise:

Public‑facing research on “creating a positive singularity.”
Outreach/education/fundraising via the Summit and rationality community.
Organizational maturity.intelligence

So in this period, strategy is mixed:

Technical aim: make progress on FAI theory.
Meta‑aim: grow a rationalist / EA / donor ecosystem and mainstream attention.

Evidence they still saw themselves as the eventual FAI builders.

Several pieces of evidence support the picture that, in the 2000s and up to at least ~2011, SIAI’s central plan was that it (or a very close “functional equivalent”) would actually build the first Friendly AGI:

The MIRI timeline notes the original mission (2000–2006) as “Create a Friendly, self‑improving Artificial Intelligence.”timelines.issarice
In a video first hosted in 2011, Eliezer says:
“As a research fellow of the Singularity Institute, I’m supposed to first figure out how to build a Friendly AI, and then once I’ve done that go and actually build one…
The Singularity Institute was founded on the theory that in order to get a Friendly artificial intelligence someone’s got to build one. So there. We’re just going to have an organization whose mission is ‘build a Friendly AI’. That’s us. … At the core, the reasoning is: ‘Someone has to do it. “Someone” is us.’”lesswrong
Luke Muehlhauser’s 2013 post Friendly AI Research as Effective Altruism (looking back on the founding) explicitly confirms:
“MIRI was founded in 2000 on the premise that creating Friendly AI is the best way to ensure a positive Singularity…
MIRI co‑founder Eliezer Yudkowsky usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI.”intelligence

Wei Dai’s 2025 LessWrong post also recalls that he was “arguing against SIAI/MIRI’s plan to build FAI” as early as 2004. That is consistent with the explicit “someone is us” framing above.lesswrong+1

So for roughly 2000–2010, the strategic picture is:

Goal: Design and eventually build a Friendly, self‑improving AGI.
Means: Foundational theory + community‑building + outreach (Summits, LessWrong, rationality training).
Role: SIAI is not just an advocacy group; it is intended Seed AI shop.

3. 2011–2013: professionalisation and pivot to FAI math research

Here we see the first clearly documented organizational pivot with a date.

3.1. 2013: “MIRI’s Strategy for 2013” – research‑only, FAI math‑first

In April 2013, MIRI (new name) publishes MIRI’s Strategy for 2013. Key elements:intelligence

They have:
“once [been] doing three things — research, rationality training, and the Singularity Summit. Now we’re doing one thing: research. Rationality training was spun out to CFAR, and the Summit was acquired by Singularity University.”intelligence
After strategic planning with 20+ advisors in early 2013, they decide to:
“(1) put less effort into public outreach, and
(2) shift our research priorities to Friendly AI math research.”intelligence
They distinguish expository, strategic, and Friendly AI research and explicitly say:
“Strategic research will consume a minority of our research budget in 2013…”intelligence

In other words, as of early 2013:

Outreach and rationality training are explicitly deprioritised.
The core strategy is: focus limited staff and budget on math‑style Friendly AI research (logical uncertainty, self‑modification, decision theory) with some high‑quality strategy work, but little expository or broad public outreach.

3.2. 2013: bylaw / mission statement change

In the same period, MIRI’s formal mission statement is softened from “create Friendly AI” to a more conditional role. Luke writes in June 2013:

“When updating MIRI’s bylaws in early 2013, Yudkowsky and I came to a compromise on the language of MIRI’s mission statement, which now reads:
‘[MIRI] exists to ensure that the creation of smarter‑than‑human intelligence has a positive impact… Thus, the charitable purpose… is to: (a) perform research… (b) raise awareness… (c) advise… and (d) as necessary, implement a smarter‑than‑human intelligence with humane, stable goals’ (emphasis added).”intelligence

Luke adds:

“My own hope is that it will not be necessary for MIRI (or a functional equivalent) to attempt to build Friendly AI itself. But… I must remain open to the possibility that this will be the wisest course of action as the first creation of AI draws nearer.”intelligence

So early 2013 marks a formal pivot:

In documents: from “we exist to build FAI” to “we exist to ensure AGI goes well; we might build AGI if necessary.”
In practice: from “outreach + community + research” to “almost pure technical FAI math research.”

4. Circa 2013: the “world‑altering Friendly AI” plan

Wei Dai’s 2025 LessWrong post, which you linked, is primarily about this era. He characterizes the “circa 2013” strategy as:

“MIRI’s circa 2013 plan, to build a world‑altering Friendly AI, was a good one?”alignmentforum+1

His argument is that this plan was strategically bad even ex ante, because a small team attempting to directly build a Friendly sovereign AI faces “illegible safety problems” — traps they cannot even see.lesswrong

Does the evidence support that this really was the plan?

Luke’s 2013 post explicitly notes that Eliezer usually talks about MIRI specifically (or a functional equivalent) creating Friendly AI.intelligence
In the 2013 strategy conversation with Holden Karnofsky and others, Eliezer says:
“Civilization doesn’t seem to be handling AI friendliness on its own … so our mission: just go do it…
I have never held out much hope for approaching a random AGI project 30 years from now… and showing them a bunch of results… So it doesn’t necessarily have to be in MIRI’s basement. It’s going to be some project that shows what I would consider a reasonable level of concern… The present community that we’re trying to gather is not so much a community of people who are going to be influential on someone else. It’s more like a community of people who do research work.”openphilanthropy

This is consistent with: MIRI expects either itself, or a close “strategically adequate” project built around its ideas and people, to be the group that actually builds the first aligned AGI.
The LessWrong discussion notes that Yudkowsky “usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI,” and this needed explicit textual evidence by 2025 because it was no longer “common knowledge.”alignmentforum+1

Put together, the best reading is:

Around 2013, MIRI’s operational strategy is “do deep foundational technical work now so that MIRI or a tightly aligned group can someday build a world‑altering Friendly AI (often conceived as CEV‑like), thereby securing the future.”
Critics like Wei Dai were indeed pushing alternative strategies (e.g. invest in more geniuses via embryo selection / cloning and buy time by delaying AI efforts) as early as 2004.lesswrong

This is almost exactly the “race to build Friendly AI” picture the LessWrong post is relitigating, although there is internal disagreement over how much this should be described as “take over the world yourselves” versus “someone legitimate should, and we might be that someone.”lesswrong

5. Emergence of Task‑AGI and “pivotal acts” (≈2014–2017)

Over time, there is a shift in how MIRI talks about what the first aligned system should do.

Early concepts such as Coherent Extrapolated Volition (CEV) envisaged a very powerful sovereign AI that, roughly, does what humanity would want if we were smarter, more informed, and more coherent. That is a maximally ambitious alignment target.
Later, Yudkowsky distinguishes between:
- A Sovereign (a CEV‑like system running the long‑term future), and
- A Task AGI, a powerful but limited system intended to perform a pivotal act — some one‑time intervention that “upsets the gameboard” in our favor (e.g. uploading researchers, preventing hostile superintelligences, enabling nanotech), after which further decisions can be made under safer conditions.lesswrong+1

The LessWrong “pivotal act” and “Task AGI” pages (compiled later, but summarizing this thinking) explicitly give examples of a Task AGI that can:

Upload humans and run uploads at high speed.
Prevent the origin of hostile superintelligences, possibly temporarily.
Deploy nanotechnology that gives a route to intelligence enhancement or strong defense.lesswrong+1

These are very close to the examples quoted in the comment thread you provided.

When does this show up as strategy?

The clearest institutional articulation is MIRI’s 2017 Fundraiser and Strategy Update (Nov 30, 2017). That document lays out a multi‑step high‑level plan:effectivealtruism

Humanity needs a “stable period” to think clearly about long‑run futures and avoid locking in bad outcomes.
To get there, we must first end an “acute risk period” in which AGI could cause extinction.
MIRI expects this to require a “risk‑mitigating technology” developed via:
“AGI‑empowered technological development carried out by task‑based AGI systems.”effectivealtruism
That, in turn, requires:
“Construction of minimal aligned AGI… aligned AGI that has the minimal necessary capabilities for ending the acute risk period.”effectivealtruism
They stress the need for a “strategically adequate” project with strong opsec, research closure, commitment to the common good, and “white‑boxed system understanding.”effectivealtruism

This is, in effect, the Task‑AGI / pivotal‑act plan written down in strategic language.

So while Task‑AGI ideas were developing earlier, 2017‑11‑30 is the first strong documented pivot point where MIRI’s official strategic picture becomes:

Don’t aim first for a full CEV‑style sovereign.
Aim to enable a strategically adequate team to build a minimal aligned AGI capable of executing a pivotal act that ends the acute risk period.

Eli Tyre’s later comment, which you saw in the LessWrong thread, describes an “intentional switch, around 2016 (though I’m not confident in the date), from aiming to design a Friendly CEV‑optimizing sovereign AI to aiming to design a corrigible minimal Science‑And‑Engineering‑AI to stabilize the world.” That matches what shows up publicly in 2017.lesswrong

6. 2017–2018: new engineering‑heavy, nondisclosed‑by‑default research program

In parallel, there was a major research‑tactics pivot.

6.1. New research directions (2017–2018)

The 2017 fundraiser already notes that, with new funding and changed views, MIRI is “starting to explore new research directions while also continuing to push on our agent foundations agenda,” and is hiring software engineers to “experiment and explore some ideas in implementation.”effectivealtruism

The 2018 post 2018 Update: Our New Research Directions gives a detailed picture:intelligence

MIRI still has the Agent Foundations agenda (logical uncertainty, embedded agency, decision theory).openphilanthropy+1
“Recently, some MIRI researchers developed new research directions that seem to enable more scalable progress” by building software systems to test intuitions and iterate quickly.intelligence
High‑level themes include:
- “Seeking entirely new low‑level foundations for optimization, designed for transparency and alignability from the get‑go, as an alternative to gradient‑descent‑style machine learning foundations.”intelligence
- Figuring out parts of cognition that can be “very transparent as cognition.”intelligence

These directions are explicitly engineering‑heavy, Haskell‑centric, and aimed at deconfusion while building real code, not just pen‑and‑paper math.intelligence

6.2. “Nondisclosed‑by‑default” policy

The same 2018 post announces a major change in publication policy:

“MIRI recently decided to make most of its research ‘nondisclosed‑by‑default’, by which we mean that going forward, most results discovered within MIRI will remain internal‑only unless there is an explicit decision to release those results…”intelligence

Reasons include:

Worry that deconfusion results might also lead to capabilities advances.
Desire to let researchers think without constant self‑censorship about dual‑use risks.
Focus on speed of progress rather than exposition.intelligence

So 2018‑11‑22 marks a clear tactical pivot:

From publishing most core work to doing a large fraction of the new engineering‑heavy research under confidentiality.
From a more outreach‑oriented identity to “we really genuinely don’t mean outreach; we’re simply and only aiming to directly make research progress on the core problems of alignment.”intelligence

This is almost certainly the “4–5 years of confidential‑by‑default research” that Habryka references in the comment thread.lesswrong

7. 2020: admitting the 2017 research push “largely failed” and regrouping

By late 2020, MIRI states that the 2017–2019 “new directions” have not panned out as hoped.

In 2020 Updates and Strategy (Dec 20, 2020), Nate Soares writes:intelligence

2020 saw “limited progress in the research MIRI’s leadership had previously been most excited about: the new research directions we started in 2017.”intelligence
Senior staff have become “more pessimistic about this approach”:
“The non‑public‑facing research I (Nate) was most excited about had a flavor of attempting to develop new pragmatically‑feasible foundations for alignable AI… After pushing more on these lines of research, MIRI senior staff have become more pessimistic about this approach.”intelligence
As a result:
“We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working.”intelligence

Some researchers leave; others attempt to “rescue” the 2017‑initiated lines; others pivot to new exploratory projects.intelligence

This is a clear strategic pivot dated 2020‑12‑20:

Official recognition that the previous core engineering program is unlikely to succeed.
An explicit search phase for new strategy, with no replacement yet fully specified.

8. 2022: “AGI Ruin” and strong public pessimism

Eliezer’s widely read 2022 essay AGI Ruin: A List of Lethalities further clarifies how MIRI leadership now see the situation.intelligence+1

Key points relevant to strategy:

Alignment, in the sense “get to less than certain doom on the first dangerous try,” is portrayed as extremely difficult; the problem is lethal and must be solved under both a time limit and a “first critical try” constraint.intelligence
He argues a pivotal act is still needed:
“We need to align the performance of some large task, a ‘pivotal act’ that prevents other people from building an unaligned AGI that destroys the world.”intelligence
But he also says he sees no realistic path to doing this under current conditions, and regards doom as the default outcome.

Although AGI Ruin is not itself a strategy document, it marks a public rhetorical pivot:

The earlier plan “MIRI (or a strategically adequate project we seed) devises and executes a pivotal act using aligned AGI” is treated as effectively unachievable.
This viewpoint sets the stage for the next major institutional shift: toward halting or pausing AGI rather than expecting to align and deploy it.

9. 2023–2024: pivot to policy and communications first

MIRI’s 2024 Mission and Strategy Update (Jan 3–4, 2024) is the most explicit top‑level strategy document and also provides a retrospective on earlier phases.lesswrong+2

After reviewing its history (founding accelerationism, 2003 alignment shift, 2006–2012 community‑building, 2013 Agent Foundations, 2017 engineering push, 2020 regrouping), it states:

“In 2023, MIRI shifted its strategy to pursue three objectives:
Policy: Increase the probability that the major governments of the world end up coming to some international agreement to halt progress toward smarter‑than‑human AI, until humanity’s state of knowledge… has drastically changed; and until we are able to secure these systems…
Communications: Share our models of the situation with a broad audience…
Research: Continue to invest in a portfolio of research… including technical alignment research (though we’ve become more pessimistic that such work will have time to bear fruit if policy interventions fail to buy the research field more time).”effectivealtruism+2

They add:

“Although we plan to pursue all three of these priorities, it’s likely that policy and communications will be a higher priority for MIRI than research going forward.”lesswrong+1

Thus, in 2023 (per the 2024 retrospective) MIRI’s top‑level strategy pivots to:

Primary: Push for an international halt / moratorium on smarter‑than‑human AI (policy) and make their models legible enough to shift the Overton window (communications).
Secondary: Continue alignment research, but now framed as conditional on policy success buying time.

This is a very sharp contrast with:

2013 (“less outreach, more technical research”).intelligence
2018 (“we really don’t mean outreach; we’re just trying to solve core alignment problems”).intelligence

Now, outreach and policy work are explicitly prioritized over new technical work.

10. Consolidated list of major strategic pivots with dates and evidence

Below is a concise list of the clearest, evidence‑backed pivot points.

Date / period	Pivot	Evidence
2000‑07‑27	Founding as SIAI with mission “Create a Friendly, self‑improving AI” and early accelerationist assumption “greater intelligence ⇒ greater morality.”	Founding mission on 2000 Form 990 and timeline; MIRI 2024 retrospective.timelines.issarice+1
~2003	Conceptual shift from “accelerate AGI, assume morality” to “alignment is hard and central”; focus moves to the Friendly AI problem and x‑risk.	MIRI 2024 history: “shifted its focus to the alignment problem around 2003.”intelligence+1
2007–2010	Mission reworded to “developing safe, stable, and self‑modifying AGI” and then “develop the theory… of safe self‑improving AI,” still with self‑image as eventual FAI builder.	Mission entries on timeline; early FAI texts.timelines.issarice+2
2011 strategic plan (Aug 2011)	First formal organisational strategy: three pillars (FAI research, outreach/human capital, organisational improvements); heavy emphasis on Summit and rationality outreach.	Strategic Plan, updated Aug 2011.intelligence
By ~2011 (video)	Eliezer states explicitly that SI’s mission is “first figure out how to build a Friendly AI, and then… actually build one… ‘Someone has to do it. “Someone” is us’,” confirming the self‑builder strategy.	2011 video quoted by Eli Tyre.lesswrong
Early 2013	Formal bylaw/mission change: from “build FAI” to “ensure that the creation of smarter‑than‑human intelligence has a positive impact,” with building AGI only “as necessary.”	Luke’s Friendly AI Research as Effective Altruism quoting the new bylaws.intelligence
2013‑04‑12	MIRI’s Strategy for 2013: pivot from “research + outreach + Summit + rationality” to “one thing: research,” with outreach and expository work sharply deprioritized and FAI math research prioritized.	MIRI’s Strategy for 2013.intelligence
Circa 2013 (world‑altering FAI plan)	Operational plan: MIRI or a closely allied small team aims to build a world‑altering Friendly AI (often sovereign / CEV‑like) to secure the future; this is the central “race to build Friendly AI” debated by Wei Dai et al.	Wei Dai’s retrospective; Luke’s confirmation that Eliezer talks of MIRI in particular creating FAI; 2013 strategy conversation stating “mission: just go do it,” possibly not “in MIRI’s basement” but via a tightly concerned project.lesswrong+3
2017‑11‑30	2017 Fundraiser and Strategy Update: explicit strategic model of success via “AGI‑empowered technological development” by task‑based AGI, construction of “minimal aligned AGI”, and need for a strategically adequate project; this is the Task‑AGI / pivotal‑act plan in official form.	2017 strategy/fundraiser post.effectivealtruism
2017–2018	Launch of “new research directions” — engineering‑heavy, software‑based deconfusion work — alongside Agent Foundations, enabled by new funding.	2017 fundraiser and 2018 update.effectivealtruism+1
2018‑11‑22	Adoption of “nondisclosed‑by‑default” policy: most new technical results kept internal unless clear safety upside in release; MIRI explicitly positions itself as doing non‑outreach, straight‑shot alignment research.	2018 Update: Our New Research Directions.intelligence
2020‑12‑20	2020 Updates and Strategy: senior staff judge that the 2017–2019 non‑public‑facing research “has, at this point, largely failed”; MIRI enters a state of regrouping and strategy search.	2020 Updates and Strategy.intelligence
2022‑06‑10	AGI Ruin: A List of Lethalities: public articulation that alignment is extremely difficult and likely to fail; still endorses need for a “pivotal act” but with very little hope of achieving it under current dynamics.	AGI Ruin.intelligence+1
2023 (described 2024‑01‑03)	Major strategy shift to policy and communications as higher priority than research: goal is to increase odds of an international agreement to halt progress toward smarter‑than‑human AI, while still doing some technical and supporting research.	MIRI 2024 Mission and Strategy Update.intelligence+2

How this lines up with the LessWrong post and comments

Starting from the post and comments you linked:

The claim that “MIRI’s circa‑2013 plan [was] to build a world‑altering Friendly AI” is well supported by:
- The original mission (“Create a Friendly, self‑improving AI”).timelines.issarice
- Eliezer’s 2011 statements that SI’s mission is literally to build Friendly AI and that “someone has to do it… ‘Someone’ is us.”lesswrong
- Luke’s 2013 remark that Eliezer usually talks about MIRI in particular (or a functional equivalent) creating FAI.intelligence
- The 2013 strategy conversation, which explicitly dismisses the idea of merely influencing “a random AGI project 30 years from now,” in favor of gathering a community that “just [goes and does] it” with safety baked in from the ground up.openphilanthropy
The “pivotal act” and “Task AGI” thinking that comes up in the comments corresponds closely to what MIRI formalizes in 2017 as “minimal aligned AGI” used for “AGI‑empowered technological development” to end the acute risk period.lesswrong+2
The comment‑thread reference to a 4–5‑year confidential research period matches the 2017–2018 launch of nondisclosed‑by‑default, engineering‑heavy research and the 2020 admission that this push had largely failed.intelligence+1
The observation that MIRI later pivoted away from “build FAI ourselves” because alignment turned out to be too hard is consistent with:
- The 2020 regrouping post.intelligence
- The strongly pessimistic tone of AGI Ruin (2022).intelligence
- The 2023 pivot toward policy and communication, treating large‑scale AI pause as central and technical alignment research as increasingly unlikely to arrive in time absent that pause.effectivealtruism+2

The main point of disagreement in the LessWrong discussion is about phrasing like “MIRI’s plan, to build a Friendly AI to take over the world,” which some participants (e.g. Habryka) see as overstating how explicitly “take over the world ourselves” was ever adopted as official organizational strategy, versus “someone should do something world‑altering, and maybe that someone is us or a very similar team.”lesswrong

On the factual side, however, the evidence is fairly clear that:

For roughly its first 10–15 years, SIAI/MIRI saw itself as a small group that might directly build the first Friendly superintelligence (initially a sovereign CEV‑like system, later a minimal Task‑AGI for a pivotal act).
Around 2013 it professionalized and narrowed to technical FAI work while still effectively planning to be (or seed) that project.
From 2017 onward it concentrated on an internal, engineering‑heavy attempt to find new foundations for alignable AI.
By 2020–2023 it concluded that this had failed and pivoted to policy and communication, with technical work now secondary and survival odds seen as hinging largely on halting or drastically slowing frontier AI.

That is the best evidence‑backed reconstruction of SIAI/MIRI’s strategic trajectory given the post, its links, and MIRI’s own later retrospectives.

Follow-up answer to “Can you dig deeper, for example into LessWrong posts/comments, to find out when Eliezer or MIRI first started thinking or talking about task AGI?”

Based on a deeper analysis of LessWrong posts, Arbital edit histories, and MIRI archives, the concept of Task AGI (and the associated shift to “pivotal acts” rather than immediate “Sovereign/CEV”) emerged earlier than the 2017 strategy update.

The specific “origin date” for the term and formalized concept is June 9, 2015.

Key Dates and Evidence

June 9, 2015 (The “Smoking Gun”): Eliezer Yudkowsky created the page “Task-directed AGI” on Arbital on this exact date.
- Evidence: Metadata from Arbital scrapes explicitly list the page as “by Eliezer Yudkowsky Jun 9 2015 updated Mar 25 2017.”arbital.obormot
- Significance: This proves the concept was being actively formalized in mid-2015, fully two years before it became the centerpiece of the 2017 public strategy.
May 24, 2016: The Arbital page for the “Do-What-I-Mean hierarchy” was created, which integrated “Task AGI” into a broader safety framework. This confirms that by 2016, MIRI had a developed internal language around “Task AGI” vs “Sovereign AGI” that was distinct from their earlier “Tool AI” discussions.arbital
2014 (The Intellectual Catalyst): The term likely crystallized as a response to Nick Bostrom’s book Superintelligence (published July 2014).
- Context: Bostrom popularized the typology of “Oracle, Genie, Sovereign, Tool.”
- Connection: The Arbital definition of Task AGI explicitly references Bostrom: “An advanced AI that’s meant to pursue a series of limited-scope goals… In Bostrom’s terminology, a Genie.”arbital.obormot
- Pivot Logic: MIRI had previously been skeptical of “Tool AI” (arguing that passive tools are unsafe). “Task AGI” appears to be their 2015 attempt to construct a safe version of a Genie/Tool—one that is agentic enough to be useful (perform a pivotal act) but limited in scope (“corrigible” and “mildly optimizing”) to avoid taking over the future.

Summary of the Evolution

The timeline of MIRI’s internal pivot to “Task AGI” can be reconstructed as:

2013–2014: Debate over “Tool AI” vs. “Oracle AI.” Eliezer is critical of the idea that AI can just be a “passive tool” (see Holden Karnofsky’s Singularity Institute Objection, 2012–2014).
July 2014: Bostrom’s Superintelligence provides new vocabulary (Genie vs. Sovereign).
June 2015: Eliezer formally documents “Task-directed AGI” on Arbital. The strategic hope shifts from “Build CEV immediately” to “Build a Task AGI to perform a pivotal act.”
2016: Internal consensus solidifies. MIRI researchers (like Eli Tyre and Nate Soares) begin treating “Task AGI for a pivotal act” as the standard plan. This is the “intentional switch around 2016″ mentioned in the comments.
Nov 2017: The strategy is officially announced to the public in the 2017 Fundraiser and Strategy Update.

In short: While the public saw the pivot in 2017, the intellectual shift happened in mid-2015, likely triggered by the digestion of Bostrom’s 2014 book and the need for a more pragmatic “minimum viable” target than full CEV.

Wei Dai

1. The Intellectual Origins (The 1970s)

My 2003 Post on the Evolu­tion­ary Ar­gu­ment for AI Misalignment

A Con­flict Between AI Align­ment and Philo­soph­i­cal Competence

No single founder

Early mathematical groundwork

Founders of modern academic cryptography

RSA and later theoretical founders

Theory of computation

Algorithmic information theory

Game theory

Decision theory

SIAI/​MIRI’s strategy over time: phases and pivot points