Philosophy PhD student, worked at AI Impacts, then Center on Long-Term Risk, now OpenAI Futures/Governance team. Views are my own & do not represent those of my employer. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html
Daniel Kokotajlo
Karma: 13,170
To build intuition about what takeoff might look like, I highly encourage everyone to read this report and then play around with the model—or at least, just go play around with the model (it’s user-friendly & does not assume familiarity with the report). Try different settings and look at what’s happening around AGI (the 100% automation line). In particular look at the green line.
It’s gonna take me a while to digest this post, but in the meantime, thank you! This is the sort of content I love to see. (ETA: I strong-upvoted this post)
To elaborate on what Jacob said:
A long time ago I spent a few months reading and thinking about Ajeya’s bio anchors report. I played around with the spreadsheet version of it, trying out all sorts of different settings, and in particular changing the various settings to values that I thought were more plausible.
As a result I figured out what the biggest cruxes were between me and Ajeya—the differences in variable-settings that led to the largest differences in our timelines.
The biggest one was (unsurprisingly, in retrospect) the difference in where we put our probability mass for the training requirements distribution. That in turn broke down into several sub-cruxes.
I wrote Fun with +12 OOMs to draw everyone’s attention to that big uber-crux. In addition to just pointing out that uber-crux, my post also operationalized it and explained it so that people didn’t have to be super familiar with Ajeya’s report to understand what the debate was about. Also, I gave five examples of things you could do with +12 OOMs, very concrete examples, which people could then argue about, in the service of answering the uber-crux.
So, what I would like to see now is the same thing I wanted to see after writing the post, i.e. what I hoped to inspire with the post: A vigorous debate over questions like “What are the reasons to think OmegaStar would constitute AGI/TAI/etc.? What are the reasons to think it wouldn’t?” and “What about Crystal Nights?” and “What about a smaller version of OmegaStar, that was only +6 OOMs instead of +12? Is that significantly less likely to work, or is the list of reasons why it might or might not work basically the same?” All in the service of answering the Big Crux, i.e. probability that +12 OOMs would be enough / more generally, what the probability distribution over OOMs should be.
Just chiming in to say huge +1 to the idea of rewarding people for doing reviews, it’s an awesome and very pro-social thing to do and I’m honored that so many people chose my post to review. I endorse rewarding Shimi et al, and Nostalgebraist, in particular.
Also: I happen to be having a related conversation that also gives some context on how I conceived of the OP at least & what I hoped to accomplish with it.
Warning: Long rant incoming, one you probably won’t benefit from reading unless you are Raemon, and in fact I’m a bit embarrassed to have written it:
I admit I feel some dismay at seeing Nostalgebraist’s review and especially Shimi/Collman/Gyrodiot’s reviews appear on this list. I respect all of these people as thinkers and upvoted their reviews, IIRC, and also I am genuinely honored and flattered that they not only read my post but took the time to review it. I won’t object if you pay them money for their reviews; I wish them well. In fact I’ll feel guilty if this comment of mine gets in the way of their reward, and I hope that it doesn’t.
But am having to do some serious soul-searching upon receiving the evidence that their reviews have stood the test of time and helped you understand my original post—because I think they both miss the point of the original post. Now I’m wondering what I did wrong, how I could have been so unclear in the OP, that so many people misunderstood...
Quoting from the original post:I describe a hypothetical scenario that concretizes the question “what could be built with 2020’s algorithms/ideas/etc. but a trillion times more compute?” Then I give some answers to that question. Then I ask: How likely is it that some sort of TAI would happen in this scenario? This second question is a useful operationalization of the (IMO) most important, most-commonly-discussed timelines crux: “Can we get TAI just by throwing more compute at the problem?” I consider this operationalization to be the main contribution of this post; it directly plugs into Ajeya’s timelines model and is quantitatively more cruxy than anything else I know of. The secondary contribution of this post is my set of answers to the first question: They serve as intuition pumps for my answer to the second, which strongly supports my views on timelines.
I literally said right at the front (admittedly behind spoiler screen) what the main and secondary points of the post were. And the subtitle said it too: “Big Timelines Crux Operationalized.”
the Shimi/Collman/Gyrodiot review most seriously misunderstands the OP; see this quote from the review:The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame. If not, then the arguments made by Daniel wouldn’t have the consequence of making people have shorter timelines.
The main point of the post was to focus the discussion on the big crux, not to argue for short timelines. The secondary point was an intuition pump for short timelines—but it does NOT depend on it being at all plausible for us to achieve +12 OOMs in the real world anytime soon! I said very clearly that the +12 OOMs thing was a hypothetical, involving magic! I brought this up in the comments; see discussion. You quote a passage that seems to be making the same mistake:
Another issue with this hypothesis is that it assumes, under the hood, exactly the kind of breakthrough that Daniel is trying so hard to remove from the software side. Our cursory look at Ajeya’s report (focused on the speed-up instead of the cost reduction) showed that almost all the hardware improvement forecasted came from breakthrough into currently not working (or not scalable) hardware. Even without mentioning the issue that none of these technologies look like they can provide anywhere near the improvement expected, there is still the fact that getting these orders of magnitude of compute requires many hardware breakthroughs, which contradicts Daniel’s stance on not needing new technology or ideas, just scaling.
To be fair to the authors, I didn’t spell out as much as I could have why it doesn’t matter if we ever achieve +12 OOMs in real life anytime soon. I mean I did spell it out, but I didn’t spell it out in as much detail as I could have—I relied on the readers being somewhat familiar with Ajeya’s model I guess. In response to a conversation with Adam Shimi after the review went up, I wrote the “Master Argument” google doc which you may have seen by now. It explains Ajeya’s model and then explains how having 80% by +12 gets you t much shorter timelines than just 50%. The key, I guess, is that if you move 30% of your mass from above 12 to below 12, unless you are crazy you will move a bunch of it to the 0-6 OOM range. You won’t pile it all up in the 6-12 OOM range. In retrospect I should have said more about that in the OP.
Anyhow. On to Nostalgebraist’s review:
...to be honest I’m not sure I understand it. The part of it where it’s talking about what the main point of Fun With +12 OOMs is… well, maybe it’s something interesting that I said, and maybe it’s equivalent to the main point under some transformation, but it’s certainly not how I think of the main point. I think the main point is “here’s this big timelines crux we all should be debating: What is the probability that +12 OOMs would be enough?” and the secondary point is “Here are some intuition pumps that +12 OOMs would be enough.”
Part of Nostalgebraist’s review was a critique of my secondary point. That part I agree with; there’s a LOT more that needs to be said (and a lot more I could have said, believe me!) about why +12 OOMs is probably enough, than just the 5 intuition pumps I gave. There’s a lot more I could do to make those 5 pumps pump harder, too. I hope someone one day finds the time to write all that stuff.
Side note: Zach Stein-Perlman’s review of Fun with +12 OOMs is great, I think he understood the original post quite well. The others… again, I appreciate them, they said some interesting things and some useful things, but it annoys me that they don’t seem to have understood the main point. And, as I said at the beginning, it makes me a bit defensive and soul-searchy. What did I do wrong? I thought I was being so clear, signposting everything, etc.!?! Yet multiple smart people I respect read it closely enough that they were motivated to review it, and came away with a different impression!
I think Nostalgebraist’s review might not deserve this reaction from me, actually. Like I said, maybe what they think the main takeaway is, is also what I thought it was, just described differently. And anyhow it’s possible that they understood perfectly what I thought the main takeaway was, and just disagreed with me about it—maybe they think that the most interesting and novel contribution isn’t what I thought it was! Fair enough. I may be making a mistake by dragging them into this. I probably shouldn’t be wasting time writing this anyway. But their review of Ajeya’s Bio Anchors report also rankled me in the same way, but more so—I think it misunderstood the whole point of the report, and I feel more confident in this claim than in the claims I made above.
There’s definitely a bias/selection effect pushing this community towards having shorter timelines. However, there’s also definitely a bias/selection effect pushing the world in general towards having longer timelines—the anti-weirdness heuristic, wanting-to-not-sound-like-a-crackpot heuristic, wanting-to-sound-like-a-sober-skeptic bias, and probably lots of others that I’m not thinking of. Oh yeah, and just general ignorance of history and the topic of tech progress in particular. I suspect that on the whole, the biases pushing people towards longer timelines are stronger than the biases pushing people towards shorter timelines. (Obviously it differs case by case; in some people the biases are stronger one way, in other people the biases are stronger in the other way. And in a few rare individuals the biases mostly cancel out or are not strong in the first place.)
I generally prefer to make up my mind about important questions by reasoning them through on the object level, rather than by trying to guess which biases are strongest and then guess how much I should adjust to correct for them. And I especially recommend doing that in this case.
If “outside view” was a natural category that was useful to use, AND people generally had a shared understanding of what it meant, then it would slow things down unnecessarily to be more specific, at least often (sometimes even then you’d want to be more specific.) My original post cast doubt on both the naturalness/usefulness of the concept (not saying there’s absolutely nothing tying the things in the Big Lists together, just saying that there isn’t really any good evidence that whatever it is that ties them together is epistemically important) and the shared understanding (ho boy do different people seem to have different ideas of what it means and how it should be used and what evidential status it confers)
Also, they might let the AIs proceed with the research anyway even though they don’t trust that they are aligned, or they might erroneously trust that they are aligned due to deception. If this sounds irresponsible to you, well, welcome to Earth.
Oh, also, a citation about my contribution to this post (Tom was going to make this a footnote but ran into technical difficulties): The extremely janky graph/diagram was made by me in may 2021, to help explain Ajeya’s Bio Anchors model. The graph that forms the bottom left corner came from some ARK Invest webpage which I can’t find now.
I’m so excited to see this go live! I’ve learned a lot from it & consider it to do for takeoff speeds what Ajeya’s report did for timelines, i.e. it’s an actual fucking serious-ass gears-level model, the best that exists in the world for now. Future work will critique it and build off it rather than start from scratch, I say. Thanks Tom and Epoch and everyone else who contributed!
I strongly encourage everyone reading this to spend 10min playing around with the model, trying out different settings, etc. For example: Try to get it to match what you intuitively felt like timelines and takeoff would look like, and see how hard it is to get it to do so. Or: Go through the top 5-10 variables one by one and change them to what you think they should be (leaving unchanged the ones about which you have no opinion) and then see what effect each change has.
Almost two years ago I wrote this story of what the next five years would look like on my median timeline. At the time I had the bio anchors framework in mind with a median training requirements of 3e29. So, you can use this takeoff model as a nice complement to that story:
Go to takeoffspeeds.com and load the preset: best guess scenario.
Set AGI training requirements to 3e29 instead of 1e36
(Optional) Set software returns to 2.5 instead of 1.25 (I endorse this change in general, because it’s more consistent with the empirical evidence. See Tom’s report for details & decide whether his justification for cutting it in half, to 1.25, is convincing.)
(Optional) Set FLOP gap to 1e2 instead of 1e4 (In general, as Tom discusses in the report, if training requirements are smaller then probably the FLOP gap is smaller too. So if we are starting with Tom’s best guess scenario and lowering the training requirements we should also lower the FLOP gap.)
The result:
In 2024, 4% of AI R&D tasks are automated; then 32% in 2026, and then singularity happens around when I expected, in mid 2028. This is close enough to what I had expected when I wrote the story that I’m tentatively making it canon.
Thanks for this thoughtful review! I think the more moderate version that you recommend is very sensible, even though I still stand by the more aggressive original version.
I think a crux for me is just how these things go in practise. Some people have reported that the meme “taboo outside view” has been helpful in conversations, and now you are reporting that it was harmful. I’m interested to hear that & would be interested to hear more details/examples if you remember any. I suppose ideally we’d do a poll of the community to find out whether the taboo has been overzealously applied or not.
Update: Russian fake news / disinfo / astroturfing seems to have been a somewhat smaller deal in 2016 than I thought. (I didn’t think it was a big effect, but “no evidence of a meaningful relationship” is still mildly surprising.)
Very interesting point! I think it’s a good one, but I’ll give a little counterpoint here since it’s on my mind:
The heuristic of “AIs being used to do X won’t have unrelated abilities Y and Z, since that would be unnecessarily complicated” might work fine today but it’ll work decreasingly well over time as we get closer to AGI. For example, ChatGPT is currently being used by lots of people as a coding assistant, or a therapist, or a role-play fiction narrator—yet it can do all of those things at once, and more. For each particular purpose, most of its abilities are unnecessary. Yet here it is.
I expect things to become more like this as we approach AGI. Eventually as Sam Altman once said, “If we need money, we’ll ask it to figure out how to make money for us.” (Paraphrase, I don’t remember the exact quote. It was in some interview years ago).
Thanks for this thoughtful review! Below are my thoughts:
--I agree that this post contributes to the forecasting discussion in the way you mention. However, that’s not the main way I think it contributes. I think the main way it contributes is that it operationalizes a big timelines crux & forcefully draws people’s attention to it. I wrote this post after reading Ajeya’s Bio Anchors report carefully many times, annotating it, etc. and starting several gdocs with various disagreements. I found in doing so that some disagreements didn’t change the bottom line much, while others were huge cruxes. This one was the biggest crux of all, so I discarded the rest and focused on getting this out there. And I didn’t even have the energy/time to really properly argue for my side of the crux—there’s so much more I could say!--so I contented myself with having the conclusion be “here’s the crux, y’all should think and argue about this instead of the other stuff.
”—I agree that there’s a lot more I could have done to argue that OmegaStar, Amp(GPT-7), etc. would be transformative. I could have talked about scaling laws, about how AlphaStar is superhuman at Starcraft and therefore OmegaStar should be superhuman at all games, etc. I could have talked about how Amp(GPT-7) combines the strengths of neural nets and language models with the strengths of traditional software. Instead I just described how they were trained, and left it up to the reader to draw conclusions. This was mainly because of space/time constraints (it’s a long post already; I figured I could always follow up later, or in the comments. I had hoped that people would reply with objections to specific designs, e.g. “OmegaStar won’t work because X” and then I could have a conversation in the comments about it. A secondary reason was infohazard stuff—it was already a bit iffy for me to be sketching AGI designs on the internet, even though I was careful to target +12 OOMs instead of +6; it would have been worse if I had also forcefully argued that the designs would succeed in creating something super powerful. (This is also a partial response to your critique about the numbers being too big, too fun—I could have made much the same point with +6 OOMs instead of +12 (though not with, say, +3 OOMs, those numbers would be too small) but I wanted to put an extra smidgen of distance between the post and ‘here’s a bunch of ideas for how to build AGI soon.’)
Anyhow, so the post you say you would love to read, I too would love to read. I’d love to write it as well. It could be a followup to this one. That said, to be honest I probably don’t have time to devote to making it, so I hope someone else does instead! (Or, equally good IMO, would be someone writing a post explaining why none of the 5 designs I sketched would work. Heck I think I’d like that even more, since it would tell me something I don’t already think I know.)
Just registering my own disagreement here—I don’t think it’s a key aspect, because I don’t think it’s necessary; the bulk of the problem IS about agency & this post encourages us to focus on the wrong problems.
I do agree that this post is well written and that it successfully gives proofs of concept for the structural perspective, for there being important problems that don’t have to do with agency, etc. I just think that the biggest problems do have to do with agency & this post is a distraction from them.
(My opinion is similar to what Kosoy and Christiano said in their comments)
I think this exchange between Paul Christiano (author) and Wei Dai (commenter) is pretty important food for thought, for anyone interested in achieving a good future in the long run, and for anyone interested in how morality and society evolve more generally.
I think this post makes an important point—or rather, raises a very important question, with some vivid examples to get you started. On the other hand, I feel like it doesn’t go further, and probably should have—I wish it e.g. sketched a concrete scenario in which the future is dystopian not because we failed to make our AGIs “moral” but because we succeeded, or e.g. got a bit more formal and complemented the quotes with a toy model (inspired by the quotes) of how moral deliberation in a society might work, under post-AGI-alignment conditions, and how that could systematically lead to dystopia unless we manage to be foresightful and set up the social conditions just right.
I recommend not including this post, and instead including this one and Wei Dai’s exchange in the comments.
I’m torn about this one. On the one hand, it’s basically a linkpost; Katja adds some useful commentary but it’s not nearly as important/valuable as the quotes from Lewis IMO. On the other hand, the things Lewis said really need to be heard by most people at some point in their life, and especially by anyone interested in rationality, and Katja did LessWrong a service by noticing this & sharing it with the community. I tentatively recommend inclusion.
The comments have some good discussion too.
My updated thoughts are: Still a great post, not as polished as it should be though. That’s OK. The important thing is that it compiles a big list of problems and alleged problems for RLHF, with links.