Philosophy PhD student, worked at AI Impacts, now works at Center on Long-Term Risk. Research interests include acausal trade, timelines, takeoff speeds & scenarios, decision theory, history, and a bunch of other stuff. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html
Daniel Kokotajlo
Karma: 7,491
TMM IIRC has an a priori argument that we should expect horizontal transmission to beat vertical transmission often: It happens faster. Replication rate of memes is orders of magnitude faster than replication rate of genes. It is surprisingly analogous to viruses/parasites in that regard. Genes do evolve over time to be more resistant to viruses/parasites; this is why the native Americans were disproportionately wiped out by disease compared to Europeans, who were disproportionately wiped out compared to Africans. However, despite this, viruses/parasites remain prevalent in the world, even (especially) in Africa. The solution to the puzzle is that the diseases are evolving too.
Culture survives by reproducing itself. Ultimately it can only do this by aiding its bearers in reproducing themselves.
Not so. Part of the point of The Meme Machine was that this isn’t true; memes (which are a kind of culture?) can be harmful to their bearers in every way and yet still successful.
Consider viruses and parasites. They typically harm their bearer’s reproductive chances, and harm their bearers in other ways as well. Just not enough that they die before passing on the virus/parasite.
This is not very important, but: What was your thought process behind the acronym PASTA? It sounds kinda silly, and while I don’t mind that myself I feel like that makes it harder to pitch to other people new to the topic. You could have said something like “R&D Automation.”
I agree that humans would do poorly in the experiment you outline. I think this shows that, like the language model, humans-with-one-second do not “understand” the code.
Haha, good point—yes. I guess what I should say is: Since humans would have performed just as poorly on this experiment, it doesn’t count as evidence that e.g. “current methods are fundamentally limited” or “artificial neural nets can’t truly understand concepts in the ways humans can” or “what goes on inside ANN’s is fundamentally a different kind of cognition from what goes on inside biological neural nets” or whatnot.
Thanks again for these newsletters and summaries! I’m excited about the flagship paper.
First comment: I don’t think their experiment about code execution is much evidence re “true understanding.”
Recall that GPT-3 has 96 layers and the biggest model used in this paper was smaller than GPT-3. Each pass through the network is therefore loosely equivalent to less than one second of subjective time, by comparison to the human brain which typically goes through something like 100 serial operations per second I think? Could be a lot more, I’m not sure. https://aiimpacts.org/rate-of-neuron-firing/#Maximum_neural_firing_rates
So, the relevant comparison should be: Give a human the same test. Show them some code and give them 1 second to respond with an answer (or the first token of an answer, and then 1 second for the second token, and so forth). See how well they do at predicting the code output. I predict that they’d also do poorly, probably <50% accuracy. In claim that this passage from the paper inadvertently supports my hypothesis:
Including test cases and natural language descriptions in the prompt lead to the highest overall performance—higher than using the code itself. Because the code unambiguously describes the semantics, whereas test cases do not, this suggests that models are in some sense not really “reading” the source code and using it to execute. Models trained on general text corpora may be better at inducing patterns from as few as two input-output examples than they are at predicting the execution of code.
Second comment: Speculation about scaling trends:
Extrapolating from Figure 3, it seems that an AI which can solve (via at least one sample) approximately 100% of coding tasks in this set, without even needing fine-tuning, will require +2 OOMs of parameters, which would probably cost about $5B to train when you factor in the extra data required but also the lower prices and algorithmic improvements since GPT-3. Being almost 2 OOMs bigger than GPT-3, it might be expected to cost $6 per 1000 tokens, which would make it pretty expensive to use (especially if you wanted to use it at full-strength where it makes multiple samples and then picks the best one) though I think it might still find an economic niche; you could have a system where first a smaller model attempts a solution and you only call up the big model if that fails, and then you keep generating samples till you get one that works so on average the number of samples you need to generate will be small, and only cost you multiple dollars for a the toughest few percentile of cases. Then this service could be used by well-paid programmers for whom the time savings are worth it.
Does this extrapolation/speculation seem right?
IDA stands for iterated distillation and amplification. The idea is to start with a system M which is aligned (such as a literal human), amplify it into a smarter system Amp(M) (such as by letting it think longer or spin off copies of itself), and then distilling the amplified system into a new system M+ which is smarter than M but dumber than Amp(M), and then repeat indefinitely to scale up the capabilities of the system while preserving alignment.
The important thing is to ensure that the amplification and distillation steps both preserve alignment. That way, we start with an aligned system and continue having aligned systems every step of the way even as they get arbitrarily more powerful. How does the amplification step preserve alignment? Well, it depends on the details of the proposal, but intuitively this shouldn’t be too hard—letting an aligned agent think longer shouldn’t make it cease being aligned. How does the distillation step preserve alignment? Well, it depends on the details of the proposal, but intuitively this should be possible—the distilled agent M+ is dumber than Amp(M) and Amp(M) is aligned, so hopefully Amp(M) can “oversee” the training/creation of M+ in a way that results in M+ being aligned also. Intuitively, M+ shouldn’t be able to fool or deceive Amp(M) because it’s not as smart as Amp(M).
Great review! One comment:
But the collection bogs down with a series of essays (about 1⁄3 of the book) on Paul Christano’s research on Iterated Amplification. This technique seems to be that the AI has to tell you at regular intervals Why it is doing what it is doing, to avoid the AI gaming the system. That sounds interesting! But then the essays seem to debate whether its a good idea or not. I kept shouting at the book JUST TRY IT OUT AND SEE IF IT WORKS! Did Paul Christano DO this? If so what were the results? We never find out.
First of all, that’s not how I would describe the core idea of IDA. Secondly, and much more importantly, we can’t try it out yet because our AIs aren’t smart enough. For example, we could ask GPT-3 to tell us why it is doing what it is doing… but as far as we can tell it doesn’t even know! And if it did know, maybe it could be trained to tell us… but maybe that only works because it’s too dumb to know how to convincingly lie to us, and if it were smarter, the training methods would stop working.
Our situation is analogous to someone in medieval europe who has a dragon egg and is trying to figure out how to train and domesticate the dragon so it doesn’t hurt anyone. You can’t “just try out” ideas because your egg hasn’t hatched yet. The best you can do is (a) practice on non-dragons (things like chickens, lizards, horses...) and hope that the lessons generalize, (b) theorize about domestication in general, so that you have a firm foundation on which to stand when “crunch time” happens and you are actually dealing with a live dragon and trying to figure out how to train it, (c) theorize about dragon domestication in particular by imagining what dragons might be like, e.g. “We probably won’t be able to put it in a cage like we do with chickens and lizards and horses because it will be able to melt steel with its fiery breath...”
Currently people in the AI risk community are pursuing the analogues of a, b, and c. Did I leave out any option d? I probably did, I’d be interested to hear it!
It’s been a year, what do my timelines look like now?
My median has shifted to the left a bit, it’s now 2030. However, I have somewhat less probability in the 2020-2025 range I think, because I’ve become more aware of the difficulties in scaling up compute. You can’t just spend more money. You have to do lots of software engineering and for 4+ OOMs you literally need to build more chip fabs to produce more chips. (Also because 2020 has passed without TAI/AGI/etc., so obviously I won’t put as much mass there...)
So if I were to draw a distribution it would look pretty similar, just a bit more extreme of a spike and the tip of the spike might be a bit to the right.
Even in the simple case no. 1, I don’t quite see why Evan isn’t right yet.
It’s true that deterministically failing will create a sort of wall in the landscape that the ball will bounce off of and then roll right back into as you said. However, wouldn’t it also perhaps roll in other directions, such as perpendicular to the wall? Instead of getting stuck bouncing into the wall forever, the ball would bounce against the wall while also rolling in some other direction along it. (Maybe the analogy to balls and walls is leading me astray here?)
I disagree about compressibility; Elon said “AI is summoning the demon” and that’s a five-word phrase that seems to have been somewhat memorable and memetically fit. I think if we had a good longer piece of content that expressed the idea that lots of people could read/watch/play then that would probably be enough.
How about “AI is like summoning a series of increasingly powerful demons/aliens and trying to make them do what we want by giving them various punishments and rewards?”
Thanks!!! Why is the pre-event more relevant?
[Question] Growth of prediction markets over time?
I don’t know, I haven’t thought much about it. I’d love it if people realized it’s more dangerous than nukes and treated it accordingly.
Excellent questions & pushback, thanks! Hmm, let me think...
I think that if we had anything close to an adequate government, AI research would be heavily regulated already. So I’m not sure what you mean by “functional government.” I guess you are saying that probably by 2026 if things go according to my story, the US government would be provoked by all the AI advancements to do more regulation of AI, a lot more, that would change the picture in some way and thus be worthy of mention?
I guess my expectation is:
(a) The government will have “woken up” to AI by 2026 more than it has by 2021. It will be attempting to pass more regulations as a result.
(b) However, from the perspective of the government and mainstream USA, things in 2026 aren’t that different from how they were in 2021. There’s more effective censorship/propaganda and now there’s all these nifty chatbot things, and there’s more hype about the impending automation of loads of jobs but whatever hype cycles come and go and large numbers of jobs aren’t actually being automated away yet.
(c) Everything will be hyperpartisan and polarized, so that in order to pass any regulation about AI the government will need to have a big political fight between Right and Left and whoever gets more votes wins.
(d) What regulations the government does pass will probably be ineffective or directed at the wrong goals. For example, when one party finally gets enough votes to pass stuff, they’ll focus on whatever issues were most memetically fit in the latest news cycle rather than on the issues that actually matter long-term. On the issues that actually matter, meanwhile, they’ll be listening to the wrong experts. (those with political clout, fame, and the right credentials and demographics, rather than e.g. those with lots of alignment forum karma)
Thus, I expect that no regulations of note relating to AI safety or alignment will have been passed. For censorship and stuff, I expect that the left will be trying to undo right-wing censorship and propaganda while strengthening their own, and the right will be trying to undo left-wing censorship and propaganda while strengthening their own. I cynically expect the result to be that both sides get stronger censorship and propaganda within their own territory, whatever that turns out to be. (There’ll be battlegrounds, contested internet territories where maybe no censorship can thrive. Or maybe the Left will conquer all the territory. Idk. I guess in this story they both get some territory.)
Yep, epistemic dysfunction affects AI companies too. It hasn’t progressed to the point where it is more than a 0.5X slowdown on their capabilities research, however. It’s more like a 0.9X slowdown I’d say. This is a thing worth thinking more about and modelling in more detail for sure.
I’m a bit skeptical of that global deaths in conflict dataset. Eyeballing it it looks like almost all of the labelled data points are from Europe. What about China? Africa? India? the Americas? I worry that there may be missing data / lack of records from those other parts of the world, or at least that it’s not appearing in this dataset.
Fun stuff by 2026: (i.e. aspects of the canonical What 2026 Looks Like storyline that probably aren’t relevant to anything important, but I’m adding them in anyway because it was fun to think about.)
Disclaimer: I made these numbers up from memory, could get much better numbers by looking up stats. Also, I suspect that these predictions err a bit on the optimistic side.
VR
10x more VR headsets sold per year. 3x more games made primarily for VR, 3x higher average budget. Headsets are noticeably higher quality without being more expensive; the cheapest ones are $200.
3D printing
10x more metal stuff gets 3D printed these days, though it’s hard for ordinary consumers to notice this. Prices have dropped, etc. Fans of miniatures and tabletop games now routinely use 3D printed resin models and the like.
SpaceX
Starship basically works now. Artemis and DearMoon have happened. The public is getting excited about space again. It’s amazing and glorious.
Starlink is fully operational, and raking in cash for SpaceX. Quality of internet connection in rural areas, on planes, etc. is going way up. Thanks to the pandemic, lots of people are living in beautiful cheap places and working remotely, and Starlink makes it possible to do this in the backwoods instead of just away from major cities.
Asteroid mining is now something lots of people take seriously. Everyone can see a path to $20/kg LEO launch costs by 2030, and are designing missions accordingly.
Cyborg stuff:
Neuralink has had a successful human trial. They are working on artificial limbs.
Robots
Boston Dynamics Spot robot and copycats are now at 100,000 units sold. (400 by end of 2020 Boston Dynamics’ Spot adds self-charging to live at remote sites forever—The Verge). The price has dropped 5x, to $15,000 per robot, and they have a lot more skills and features. Longer battery life, too.
Tesla’s prototype humanoid robot has caught up to Boston Dynamics. Still isn’t available to the public, but there is a lot of hype now as they try to figure out how to mass-produce the thing. Unclear how big the market will be. It certainly won’t be automating away millions of jobs anytime soon though.
Self-driving cars still aren’t a thing. However, Tesla’s Level 2 is pretty good.
Cars
Tesla at 3.5 million vehicles sold per year. Total EV market is about 10M, 10% of total.
Energy
Solar power costs half as much as it did in 2019. It’s really starting to overtake coal, natural gas, etc. in a big way. Lots of people are talking about the “solar revolution.” The price of solar electricity has dropped 89% in 10 years (fastcompany.com)
Crypto
It’s being heavily regulated now, like the Internet, to the point where much of its original purpose has been subverted.
On the bright side, crypto investments have outperformed tech stocks (which themselves outperformed the stock market, which has been doing great these past five years).
Various dapps and DeFi things are more mainstream now. In particular, prediction markets are 100x bigger than they were in 2020!
My own answer would be the EDT answer: how much does your decision correlate with theirs? Modulated by ad-hoc updatelessness: how much does that correlation change if we forget “some” relevant information? (It usually increases a lot.)
I found this part particularly interesting and would love to see a fleshed-out example of this reasoning so I can understand it better.
Thanks! Have you used Codex?
What are the main benefits people seek when they buy the more expensive plans? I don’t understand the stuff on the page, but it looks like it’s storage space + more features that make it easier to work in teams. I’m not sure how to compare that stuff to Codex but intuitively I feel like Codex is more valuable, because more people could benefit from Codex than are working in teams. I don’t know what I’m talking about though, which is why I’m asking. :)
If the charge is per token… let me think… suppose Codex gets called up to write something 10 times per programmer work-hour (it would come in clumps probably, not evenly spaced. Sometimes it would not give you what you want and you’d retry a couple times). That’s maybe 1000 tokens per work-hour, which (if it were GPT-3) would cost $0.06, so that’s like $0.50 a day, which comes out to $15.00 a month… I swear I didn’t plan that calculation to come out that way! (But of course it’s just a fermi estimate, could be off by orders of magnitude. Also, the current version of Codex is the 12B param version which probably costs an OOM less than GPT-3)
I think this sequence of posts is underrated/underappreciated. I think this is because (A) it’s super long and dense and (B) mostly a summary/distillation/textbook thingy rather than advancing new claims or arguing for something controversial. As a result of (a) and (b) perhaps it struggles to hold people’s attention all the way through.
But that’s a shame because this sort of project seems pretty useful to me. It seems like the community should be throwing money and interns at you, so that you can build a slick interactive website that displays the full graph (perhaps with expandable/collapsible subsections) and has the relevant paragraphs of text appear when you hover over each node, and that would just be Phase 1, Phase 2 would be adding mathematical relationships to the links and numbers to the nodes, such that people can input their own numbers and the whole system updates itself to spit out answers about nodes at the end. The end result would be like a wiki + textbook only better organized and more fun, while simultaneously being a the most detailed and comprehensive quantitative model of AI stuff ever.