Inspired by Daniel and Romeo https://x.com/DKokotajlo/status/1986231876852527270, I’m posting a memo on Takeoff, which I plan to expand into a proper article.
This memo comes from notes I took from a talk from Daniel, which have been synthesized by Claude for breviy:
Transformative AI timeline: Late 2028
Key prediction: All existing benchmarks solved by 2030
p(doom): 10%
Reasoning capabilities (mostly solved)
Agency and long-horizon skills:
Error correction
Goal pursuit
Hierarchical planning
Data-efficient learning
New models and architectures (e.g., recurrence)
Language Model Programs (LMP)
Scaffolding systems
AI bureaucracies
Increased inference compute
“Just train it to have those skills” approach
Research Engineer for AI Research (we are entering here with Kosmos)
“Devin actually works”
Significant speedup in AI research
Research Scientist
“CEO talks to the cluster”
Autonomous white paper production
Genius
Qualitatively better than human researchers
Artificial Superintelligence
Capable of anything
Equivalent to 100k human teams
10^14 text predictions/rollouts on agentic paths
10^23 floating point operations for transformative capabilities
GPT-4 pre-training: ~2.15 × 10^25 FLOPS for comparison
t-AGI (10 second AGI to 10 day AGI)
Year-AGI scaling
Annual 1 OOM (order of magnitude) improvements expected
Robotics gap: “If TAI is Industrial Revolution level, what about physical automation?”
Job stickiness: Institutional and social lags in job displacement
“Wet robots”: Humans as interim solution
“Paperclips are a metaphor for money”
“Capitalism is the ultimate Turing test”
AI systems are “incentivized to be constrained by human values”
Early prediction market odds:
1⁄3 Futurama scenario
1⁄4 Fizzle out
1⁄5 Dystopia
Cyberpunk 2077-style hacking risks
43% of Americans believe civil war is somewhat likely
Democracy at its peak by voter count (2024)
Unprecedented political deepfakes
Cultural homogenization
Human disempowerment (sci-fi scenarios)
Potential for singleton control
“OpenAI coming for everybody’s jobs is a solace” -
Logarithmic growth in atoms, exponential growth in bits
“Folk theorem—bigger (model) is better”
“We’re very different from current machines”
“Brains are way more complex than even the most sophisticated NNs”
Adaptation and time remain key differentiators
Virtual societies and digital twins
AI safety as cancer immune response
Automated social science research
a deceptively simple solution to alignment: “Train them to be honest”
A memo on Takeoff
Inspired by Daniel and Romeo https://x.com/DKokotajlo/status/1986231876852527270, I’m posting a memo on Takeoff, which I plan to expand into a proper article.
This memo comes from notes I took from a talk from Daniel, which have been synthesized by Claude for breviy:
Key Predictions
Transformative AI timeline: Late 2028
Key prediction: All existing benchmarks solved by 2030
p(doom): 10%
Current Gaps
Reasoning capabilities (mostly solved)
Agency and long-horizon skills:
Error correction
Goal pursuit
Hierarchical planning
Data-efficient learning
Proposed Solutions
New models and architectures (e.g., recurrence)
Language Model Programs (LMP)
Scaffolding systems
AI bureaucracies
Increased inference compute
“Just train it to have those skills” approach
Stages of AI Development
The RE→RS→G→ASI Framework
Research Engineer for AI Research (we are entering here with Kosmos)
“Devin actually works”
Significant speedup in AI research
Research Scientist
“CEO talks to the cluster”
Autonomous white paper production
Genius
Qualitatively better than human researchers
Artificial Superintelligence
Capable of anything
Equivalent to 100k human teams
Technical Specifications:
Compute Requirements
10^14 text predictions/rollouts on agentic paths
10^23 floating point operations for transformative capabilities
GPT-4 pre-training: ~2.15 × 10^25 FLOPS for comparison
Time Horizons
t-AGI (10 second AGI to 10 day AGI)
Year-AGI scaling
Annual 1 OOM (order of magnitude) improvements expected
Critical Questions and Concerns
On Full Automation
Robotics gap: “If TAI is Industrial Revolution level, what about physical automation?”
Job stickiness: Institutional and social lags in job displacement
“Wet robots”: Humans as interim solution
The Capitalism-AI Nexus
“Paperclips are a metaphor for money”
“Capitalism is the ultimate Turing test”
AI systems are “incentivized to be constrained by human values”
Societal Predictions
Early prediction market odds:
1⁄3 Futurama scenario
1⁄4 Fizzle out
1⁄5 Dystopia
Risk Scenarios
Cyberpunk 2077-style hacking risks
43% of Americans believe civil war is somewhat likely
Democracy at its peak by voter count (2024)
Unprecedented political deepfakes
Cultural homogenization
Human disempowerment (sci-fi scenarios)
Potential for singleton control
“OpenAI coming for everybody’s jobs is a solace” -
Basic Critiques
Logarithmic growth in atoms, exponential growth in bits
“Folk theorem—bigger (model) is better”
“We’re very different from current machines”
“Brains are way more complex than even the most sophisticated NNs”
Adaptation and time remain key differentiators
Alternative Approaches
Virtual societies and digital twins
AI safety as cancer immune response
Automated social science research
a deceptively simple solution to alignment: “Train them to be honest”