Daniel Kokotajlo
The date of AI Takeover is not the day the AI takes over
Using Steve’s analogy would make for much shorter timeline estimates. Steve guesses 10-100 runs of online-learning needed, i.e. 10-100 iterations to find the right hyperparameters before you get a training run that produces something actually smart like a human. This is only 1-2 orders of magnitude more compute than the human-brain-human-lifetime anchor, which is the nearest anchor (and which Ajeya assigns only 5% credence to!) Eyeballing the charts it looks like you’d end up with something like 50% probability by 2035, holding fixed all of Ajeya’s other assumptions.
Persuasion tools: What they are, how they might get really good prior to TAI, how that might change the world in important ways (e.g. it’s an x-risk factor and possibly a point of no return) and what we can do about it now.
I promised a followup to my Soft Takeoff can Still Lead to DSA post. Well, maybe it’s about time I delivered...
Against GDP as a metric for timelines and takeoff speeds: I think that world GDP growth increasing significantly from its current rate is something which could happen years before, OR YEARS AFTER, transformative AI. Or anything in between. I think it is a poor proxy for what we care about and that people currently go astray on several occasions when they rely on it too heavily. I think this goes for timelines, but also for takeoff speeds: GDP growth doubling in one year before it doubles in four years is a bad proxy for fast vs. slow takeoff.
Ships as precedent for AI: Lots of the arguments against fast takeoff, against AGI, against discontinuous takeoff, against local takeoff and decisive strategic advantage, are somewhat mirrored by arguments that could have been made in the middle ages about ships. I think that history turned out to mostly support the fast/AGI/discontinuous/local/DSA side of those arguments.
Explanation of how what we really care about when forecasting timelines is not the point when the last human is killed, nor the point where AGI is created, but the point where it’s too late for us to prevent the future from going wrong. And, importantly, this point could come before AGI, or even before TAI. It certainly can come well before the world economy is growing at 10%+ per year. (I give some examples of how this might happen)
Argument that AIs are reasonably likely to be irrational, tribal, polarized, etc. as much or more than humans are. More broadly an investigation of the reasons for and against that claim.
A response and critique of Ajeya Cotra’s awesome timelines report.
OK, sure. I think I misread you.
Shouldn’t the singleton outcome be in the bottom right quadrant? If attack is easy but so is coordination, the only stable solution is one where there is only one entity (and thus no one to attack or be attacked.) If by contrast defense is easier, we could end up in a stable multipolar outcome… at least until coordination between those parties happen. Maybe singleton outcome happens in both coordination-is-easy scenarios.
Several of the discontinuities in the AI Impacts investigation were the result of discontinuities in resource investment, IIRC.
I think Ajeya’s report mostly assumes, rather than argues, that there won’t be a discontinuity of resource investment. Maybe I’m forgetting something but I don’t remember her analyzing the different major actors to see if any of them has shown signs of secretly running a Manhattan project or being open to doing so in the future.
Also, discontinuous progress is systematically easier than both of you in this conversation make it sound: The process is not “Choose a particular advancement (GPT-3), identify the unique task or dimension which it is making progress on, and then see whether or not it was a discontinuity on the historical trend for that task/dimension.” There is no one task or dimension that matters; rather, any “strategically significant” dimension matters. Maybe GPT-3 isn’t a discontinuity in perplexity, but is still a discontinuity in reasoning ability or common-sense understanding or wordsmithing or code-writing.
(To be clear, I agree with you that GPT-3 probably isn’t a discontinuity in any strategically significant dimension, for exactly the reasons you give: GPT-3 seems to be just continuing a trend set by the earlier GPTs, including the resource-investment trend.)
I feel like you should have said “irrational” instead of “stupid.” It would sound less funny, but it would be more accurate.
I think that AI capable of being nerd-sniped by these landmines will probably be nerd-sniped by them (or other ones we haven’t thought of) on its own without our help. The kind of AI that I find more worrying (and more plausible) is the kind that isn’t significantly impeded by these landmines.
I think the algorithmic progress isn’t as fast as you say, at least not in the sense relevant for progress towards TAI/AGI.
Algorithmic improvements can be divided into two types, I think: Doing what we already know how to do more efficiently, and exploring new stuff more efficiently. I think lots of the improvements we’ve seen so far are of the first category, but the second category is what’s relevant for TAI/AGI. I’m not sure what the ratio is but my guess is it’s 50⁄50 or so. I’d love to see someone tackle this question and come up with a number.
That said, once we get close to TAI/AGI we should expect the first category to kick in and make it much cheaper very quickly.
When I first read the now-classic arguments for slow takeoff—e.g. from Paul and Katja—I was excited; I thought they described a serious alternative scenario to the classic FOOM scenarios. However I never thought, and still do not think, that the classic FOOM scenarios were very unlikely; I feel that the slow takeoff and fast takeoff scenarios are probably within a factor of 2 of each other in probability.
Yet more and more nowadays I get the impression that people think slow takeoff is the only serious possibility. For example, Ajeya and Rohin seem very confident that if TAI was coming in the next five to ten years we would see loads more economic applications of AI now, therefore TAI isn’t coming in the next five to ten years...
I need to process my thoughts more on this, and reread their claims; maybe they aren’t as confident as they sound to me. But I worry that I need to go back to doing AI forecasting work after all (I left AI Impacts for CLR because I thought AI forecasting was less neglected) since so many people seem to have wrong views. ;)
This random rant/musing probably isn’t valuable to anyone besides me, but hey, it’s just a shortform. If you are reading this and you have thoughts or advice for me I’d love to hear it.
Is the link for the 6-byte Code Golf solution correct? It takes me to something that appears to be 32 bytes.
Thanks! Just as a heads up, I now have read it thoroughly enough that I’ve collected quite a few thoughts about it, and so I intend to make a post sometime in the next week or so giving my various points of disagreement and confusion, including my response to your response here. If you’d rather me do this sooner, I can hustle, and if you’d rather me wait till after the report is out, I can do that too.
I’m happy to bet on my timelines: https://elicit.org/builder/zKNSxZIhn
However, since I don’t expect money to matter much in the event that I win the bet, I ask that the bets be small. I furthermore am worried that by betting on longer timelines people will make themselves psychologically more resistant to updating towards shorter timelines… The same goes for me, of course, making me extra averse to doing this. Idk.