I’ve had several conversations with people over the last few weeks that have highlighted how far apart my view of the near future is from many people I talk to. Here are some things I might tweet if that was the kind of thing I did:
AI has a very real chance of getting us all killed. I think it probably won’t because I expect a lot of people to work very hard to avoid that outcome.
AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be planning for 1-3 more years in which they can productively contribute. Use the time well!
But also don’t live your life in a way where if it takes longer than that you’re destitute; there’s still a lot of uncertainty in how quickly this plays out.
We are already seeing AI speeding up the development of AI, as it substitutes for human expertise. As the remaining human contribution gets smaller I expect this to compound dramatically, and we’ll see rapid improvement even compared to today.
I don’t know if the metaphor is “goalpost moving” or “frog boiling” but if you dropped a top model into 2006 it would very clearly be “AGI”.
If you think AI isn’t improving rapidly you’re doing some combination of not applying it to hard enough problems, not giving it enough context/tooling/tokens, or not using top models.
I’m worried about the environmental impacts of data centers, in that I see plausible near futures in which the world produces vastly more energy than it does today and almost all of it goes into powering computers.
Many paths to appropriately serious action run through warning shots, and we’ve been getting some. So far they look like hacks, and there will be more. Aside: information that should not become public is increasingly dangerous to keep.
Lots of people have plans to make the world better by improving X to improve Y to improve Z, where it’s Z that matters. If Z is more than 3y out, consider finding a different plan.
If AI goes well I think the world could be extremely good. The bottom 10% living far better than the top 10% do now. This is the promise that gets people working so hard to build something so dangerous.
I’m being a bit lazy here because the tweet format means I can make bold claims without backing any of them up. If you think I’m wrong on any of these, happy to give arguments in the comments.
It seems clearly true that many people would have a very strong first and second and third impression of it being AGI. Is there an implication you’re drawing from that, such as “it really is AGI” in some relevant sense, or that it’s strong evidence of that? I think we actually have strong evidence for “this is some weird thing which few people anticipated specifically, which is jagged, where the point of saying it’s jagged is not to say it is or isn’t scary, but to say that it doesn’t fit into the implied low-dimensional manifold of our prior intuitions”.
Agree that how he answers would change the view he stated—but if he’s right that current AI systems and methods and continuing AI research is enough to lead to takeoff and ASI in a few more years, I’m not sure the difference matters as much.
His tweet about that is
Which I think is significantly less strong than your summary. Regarding takeoff, the questions of
How close is the whole AI capabilities research community (including AIs) already to knowing enough to build AGI?
How much do current / near-future AIs contribute to the hardest parts of AI capabilities research?
would be informed a lot by “is the current thing already basically weak AGI”.
1-3 years seems plausible to me, but my median estimate of the date when most people in the work force today will no longer be able to productively contribute is significantly more than 3 years. I’d probably say sometime in the 2030s.
I want to acknowledge that I strongly agree with this takeaway despite my median being >3 years.
The next 1-3 years are indeed the most important years left to have an impact in expectation, and so it’s unusually important that we use this time well.
I’m guessing the difference here is that I’m expecting more RSI, and that I expect AI to get heavily applied to finding all the places where it can displace humans. Labor is such a big cost to companies.
I think considering the US workforce vs. the broader global workforce could change the median, but certainly for the US workforce I’m on board with the 3 year timeline.
I agree that one’s median date for the US workforce should be sooner than for the global workforce (largely because it takes time to build billions of robots to do physical tasks and the US is expected to get such robots before many other countries), but I still think my median is >5 years for the US.
Yes, this seems plausible. Both RSI scenarios where we survive and where we don’t survive:
If all humans are dead by 2030, then humans are no longer productively contributing in the workforce, so your ~20% on us being dead by 2030 due to AI (from your May post) is contributing to your forecast on humans being no longer able to contribute in 3 years being higher than mine, I think.
I’m guessing another difference is that I’m expecting it’s more likely that we “pace the frontier” and slow down AI development and deployment in the next 3 years, such that even if it was possible for AI to automate AI research and the full AI production supply chain to build robot factories and billions of robots that could outcompete most people globally for all jobs within 3 years, that won’t actually happen that quickly thanks to society’s “pacing” efforts.
This really hits on an emotional level, but I ultimately take a different perspective:
I think solving the relevant safety problems for any AI systems that could disrupt the world at this scale will be extremely difficult, and that most surviving worlds are ones in which AI progress has been ~halted via a long-term globally coordinated pause. That means the only futures I should expect to be alive to experience are ones that are fairly “normal” in many respects, which means normal futures are the ones I should plan for.
How that cashes out for me:
I don’t have any major long-term ambitions, since I’m so focused on pausing AI, but my partner for instance is working on a mathematics education PhD and wants to help reform math education in the US. I think there’s a very good chance that that will still be a valuable thing for her to pursue, despite how long-term it is.
Work aiming to make an impact in worlds in which we successfully arrange a pause does make sense, and I do think some people should be working on it. But:
In those worlds we have time, so there’s often not a good reason to take those actions now when the opportunity cost is high and the benefit is discounted by the risk we don’t get there.
On the margin, relative to today, I think many more people should be working on trying to bring about such a world.
So much work people describe to me as being valuable in these worlds isn’t even accounting for the disruption and change that will be downstream from AI we already have.
I feel you—this does hit hard emotionally.
I think someone pursuing a mathematics education PhD is a perfect example for Jeff’s X/Y/Z tweet (which I agree with).
But how valuable? (It’s easy to focus on how it’s very likely more than zero value, but that would be missing the point.) Is the marginal expected value really anywhere close to the marginal EV of raising the probability of a pause in the next couple years?
Sadly, I think not.
This can be really hard to admit for emotional reasons, especially if she is passionately focused on this path and reluctant to consider radical changes to unusual paths like working to pause AI.
And of course, since pausing AI is the path you’re pursuing, then even if you admitted it (or took the view—I don’t want to use too much language that presumes I’m right), might this just lead to unnecessery tension in the relationship? “Hey romantic parter, I think that your expected social impact would be much higher if you switched from your current path which you’re very passionate about to working on what I’m working on instead” seems pretty awkward TBH. I’d expect most people in that situation to feel like “Why is my partner trying to discourage me from doing what I’m passionate about?” and “Aren’t you biased to think that what you’re doing is most valuable?” and feel bad like they aren’t being supported. If they aren’t unusually open to pivoting, they probably won’t pivot, and so will end up having the same impact that they would have had if you hadn’t said anything, but will just think that you don’t think what they’re doing is important. And that sucks for both of you. It would be much nicer if it could just be true that continuing with her current path is the most valuable thing she can do.
Yep, that’s the situation.
Like most people, she isn’t strictly focused on maximizing expected value. (Which is fine! It’s okay to be human in ordinary human ways! That’s part of what I’m fighting to protect.) She seems to accept the arguments for extinction risk, but then compartmentalizes hard, and commits to being agnostic, saying “I don’t understand those claims” about anything at the existential level.
She listens to me all the time and has been very supportive of my activism, so I haven’t pushed for her to have more involvement, since I’m fortunate to have this level of stability and support in the first place.
As a side note, her knowledge of constructivism and other math education topics means that a lot of what we have learned about artificial minds resonates with what she has learned about human minds. Math education research was established by mathematicians moreso than teachers, so it is extremely technical and more like cognitive neuroscience than pedagogy. I’d bring her the latest in my understandings of deep learning theory and interpretability, rambling on about the nature of understanding and intelligence, then say, “I probably sound crazy.” And she’d say, “No, you sound like a constructivist.”
I expect a lot of people to work very hard to avoid that outcome as well, but I think the chance that they / we all succeed is somewhat less than 50%.
Precise forecasting here probably isn’t that important, but I want to flag that just because a lot of people work very hard to avoid an outcome doesn’t mean that the outcome will necessarily be successfully avoided.
“AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be planning for 1-3 more years in which they can productively contribute. Use the time well!”
Do you have any interest in making a bet (for charity) on this?
Possibly! How would you structure a bet like this?
Bet structure candidates:
(1) Bet on the unemployment rate. I have an active bet of this kind:
(2) If a significant amount of your probability mass on you being right comes from scenarios where RSI kills us within 3 years, such that Andy wouldn’t be able to pay out in those scenarios, then you could structure the bet as a loan in the form of Andy pays you now, and you pay Andy back in 3 years if most people are still able to productively contribute (however exactly you operationalize that). Choose your bet odds like usual, then do time discounting by either staking to an investment you’d both be happy to make over the next three years, or to an explicit interest rate you’re both agreeable to. For example, for a 20% interest rate and an even odds bet, for each $1 Andy pays you now, you pay him
In terms of operationalization, if you literally mean most people can’t get a job because employers would rather hire AI and robots to do the job better/faster/cheaper, that’s easy enough to judge. But if you mean something more conservative, like “opportunities to have a significant altruistic impact for most people are gone” then I think you should propose the operationalization since Andy and others probably don’t know exactly what you mean.
@jefftk What William said! Thanks William.
I’m only looking for smaller bets, around $50-100, to a charity of the others’s choosing. To me it’s less about the money and more about trying to operationalize possibly differing probability distributions over future outcomes. I’m game to bet on unemployment (probably higher than 6.4% based on your statement, but perhaps by August 2029?), or something else if you think it is more about opportunities for significant altruistic impact being gone.
We need to be careful here because many definitions of unemployment wouldn’t capture this kind of change. For example, if someone drops out of the labor force entirely because they can’t get a job, they are not normally counted as unemployed. I think the cleanest version is to look at the fraction of the core working age population that is employed. How about https://fred.stlouisfed.org/series/LNS12300060 below 50% for 2029-08? My $50 to charity against Andy’s $75, to cover xrisk/collapse/etc?
Sounds good to me! How about this?
If the seasonally adjusted August 2029 value of FRED series LNS12300060 is below 50.0%, I will donate $75 to a charity of your choice. Otherwise, you will donate $50 to a charity of mine. We will use the first published value for August 2029. If no August 2029 value has been published by December 31, 2029, the bet is void.
There’s a tension between the future environmental concerns and the idea that most people won’t be able to contribute much after 3 years or so. Valar Atomics and Terraform Industries are two companies that will substantially address most clean energy concerns if they succeed; there are likely many others that I don’t know about. It’s very hard for me to believe that those efforts will fail in a world where they gain access to unlimited scientific brainpower. I think the post-AGI world will have many problems, but the nature of those problems is extremely hard to predict in advance.
The environmental bullet is a bit tongue in cheek, prompted by the ACX survey asking how concerned one is about resource use and environmental degradation from data centers, where I expect Scott is thinking of this as an indication that people have scope insensitive worries about AI. But I’m pointing at futures with 100,000x increases in how much energy is going into computation, which would unavoidably directly heat the planet. In these futures there’s widespread environmental damage, but we also have much larger problems including probably already being extinct.
This seems like a conservative understatement to me. I think the bottom 0.0001% could be living far better than all humans do now.
Maybe at this point we’re mostly arguing about what it means to live well? Material improvements matter to the extent that they make us happier and more satisfied. Perhaps I should have said 1% or something a bit smaller, but “0.0001%” and “all” seem more wrong in the other direction.
When I say the extreme “0.0001%” and “all” I’m mostly thinking about how I think near-optimal futures are nontrivially likely, how “AI going well” is what could enable those futures, and how near-optimal futures are not only somewhat better than the world today, but literally “extremely good” / much, much better.
That is, in these extremely good futures, I see us surviving to what Bostrom calls “technological maturity,” and I also see the population of humans and/or conscious minds we care about ballooning to astronomically large numbers as we engage in space colonization and put to use of much of the matter in the reachable universe (~400 billion stars in our galaxy, ~2 trillion galaxies in the reachable universe). The vast number of beings in these futures means that ensuring that 99.9999+% of them have better lives than all of us in 2026 is just a matter of being intentional about making sure that the minds we create in other galaxies are actually ~all awesome. I think technologically we’d easily have the capability and it’d just be a question of are we actually universally optimizing/trying for that or not. (In Letter from Utopia, I think Bostrom does a good job conveying that humans today have lives nowhere close to the best possible lives or experiences permitted by physics.) And again, in futures where AI goes well, I think there’s a significant chance (and it definitely “could” happen, to use the word you used when saying 10%/90%) that we would manage to actually optimize for such a near-optimal, extremely good future.
It seems to me that AI developers are not taking the problem remotely as seriously as they should (I wrote more about my beliefs here), and I do not expect this to change. Where do you (primarily) disagree?
Do you think AI devs’ current efforts will turn out to be adequate?
Do you think AI devs will start taking the problem more seriously?
Do you think governments will enforce better safety measures?
(My best hope is #3 but I’m not optimistic about it.)
I think 3 and 2. We’re already seeing some of each.
Where? This is not at all obvious to me.
Insofar as that last sentence is true, I think it’d be true mostly because it provides cover (to oneself and others) for other motivations like working on cool problems with smart people at high profile companies and getting large compensation.
I don’t know how we could check who’s right but I don’t believe this is true at all, much less clearly true.
All depiction of AGI in fiction I know of is stronger than today’s LLMs are and I think that is the central example people would compare LLMs to. I can provide some examples.
My favorite example is the sci-fi roleplaying system Eclipse Phase from 2009, as it has both AGI and non-general AI called Muses, with Muses very similar to today’s LLMs. The people writing the rules tought clearly Muses are not AGI and Muses are stronger than todays LLMs, though not by much.
And just generally, if you showed todays LLMs to people in 2006 and asked it “do you think this machine can do all/most intellectual tasks about as good or better than humans” people would, for the most part, probably disagree.
There’s still many intellectual tasks LLMs do not perform well at and people in 2006 were smart enough to notice. Also probably motived to find those so they can dismiss LLMs being AGI, which is something I think people don’t actually want to believe.
This seems very hard to argue about, because if I point at one of them you’ll just say “but that’s not a depiction of AGI”.
I suppose you are right. Fiction rarely labels whether a given AI system is meant to be general or not.
There’s definitely AI stronger then LLMs in fiction and AI weaker than LLMs in fiction and I cannot show that when people think of AGI they think of the examples stronger than LLMs.
So let me make another argument:
Even careful and thoughtful people are generally bad at predicting what other people believe outside of trivial examples. See for example thoughtful Democrats being often unable to articulate accurately what thoughtful Republicans believe and vice versa.
Whether or not other people believe LLMs are AGI is not a trivial example
We’re generally worse at predicting what people 20 years in the past have thought about some issue unless we have specific evidence (e.g. a written record) than we are at predicting what our contemporaries believe
We have no specific evidence (e.g. a written record) about what people in 2006 thought AGI would look like exactly and if it matches LLMs
So we cannot accurately predict whether or not people in 2006 would’ve generally considered LLMs to be AGI
And it’s especially not “clearly” or “obivously” the case they would’ve seen 2026 LLMs as AGI
The two bars I’ve been using for physical AI tasks that represent (to me) a certain level of real-world physical unknown-unknowns and trust are (1) residential tree trimming and (2) changing a diaper on a human 11 month old. Are you envisioning these both being human-out-of-the-loop within 1-3 years? From what I’ve seen the physical world manipulation capabilities just aren’t there yet, but I absolutely understand I could be wrong.