gwern
“Blessings of scale” observations aside, it seems like right now, environments are not the bottleneck to DL/DRL work. No one failed to solve Go because gosh darn it, they just lacked a good Go simulator which correctly implemented the rules of the game; the limits to solving ALE-57 (like Montezuma’s Revenge) in general or as a single multi-task agent do not seem to be lack of Atari games where what we really need is ALE-526*; Procgen performance is not weak because of insufficient variation in levels; OpenAI Universe failed not for lack of tasks, to say the least; the challenge in creating or replicating GPT-3 is not in scraping the text (and GPT-3 didn’t even run 1 epoch!). Datasets/environments sometimes unlock new performance, like ImageNet, but even when one saturates, there’s typically more datasets which are not yet solved and cannot be solved simultaneously (JFT-300M, for example), and in the case of RL of course compute=data. If you went to any DRL researcher, I don’t think many of them would name “we’ve solved all the existing environments to superhuman level and have unemployed ourselves!” as their biggest bottleneck.
Is it really the case that at some point we will be drowning in so many GPUs and petaflops that our main problem will become coming up with ever more difficult tasks to give them something useful to train on? Or is this specifically a claim about friendly AGI, where we lack any kind of environment which would seem to force alignment for maximum score?
* Apparently the existing ALE suite was chosen pretty haphazardly:
Our testing set was constructed by choosing semi-randomly from the 381 games listed on Wikipedia at the time of writing. Of these games, 123 games have their own Wikipedia page, have a single player mode, are not adult-themed or prototypes, and can be emulated in ALE. From this list, 50 games were chosen at random to form the test set.
I wonder how the history of DRL would’ve changed if they had happened to select from the other 73, or if Pitfall & Montezuma’s Revenge had been omitted? I don’t however, think it would’ve been a good use of their time in 2013 to work on adding more ALE games rather than, say, debugging GPU libraries to make it easier to run NNs at all...
Communication is hard and – importantly – contextual. Most of your readers will be reasonable people
You think this partially because you are not famous or a popular writer.
By the 1% rule of Internet participation, you hear mostly from an extremely self-selected group of critics. You don’t hear from the reasonable people, you hear from the unreasonable people. The more popular you get, the more this is true. And there is a lizardman constant going on: there is a fringe of crazy, stubborn readers who will fail to read the most plain and straightforward writing, misinterpret it in the wackiest way, hate you more the better you write, and amplify the craziest things they can find. (At my level of relative obscurity, it’s petty stuff: sneers, doxing, death/swatting threats, ML researchers trying to get me fired, FBI visits, that sort of thing. Scott seems to have similar issues, just more so. But by the time you reach Tim Ferriss numbers of readers, this will have escalated to ‘attempted kidnappings by organized crime’ levels of risk, and he notes that it escalates still further to attempted murder of popular YouTubers etc.)
Combine this with the asymmetry of loss and reward, where criticism hurts a lot more than praise helps, and the more popular you get, the worse you will feel about everything you write or do, regardless of quality.
...Unless you constantly keep in mind: “haters gonna hate”. If a criticism doesn’t immediately make sense to you or you felt you dealt with it adequately, and it comes from someone you don’t already know or trust, then oh well—haters gonna hate. If you’re genuinely unsure, run a poll or A/B test or something to hear from a less self-selected sample—but do anything other than naively listening to and believing your critics! That’s a luxury permitted only the most obscure or heavily filter-bubbled.
What about your “I am a contract-drafting em” poem?
It’s in the figure.
Might as well finish out this forecasting exercise...
If we assume compute follows the current trend of peak AI project compute doubling every 3.4 months, then 2.2e6× more compute would be log2(2.2e6) = 22 doublings away—or 22*(3.4/12) = 6.3 years, or 2027. (Seems a little unlikely.)
Going the other direction, Hernandez & Brown 2020′s estimate is that, net of hardware & algorithmic progress, the cost of a fixed level of performance halves every 16 months; so if GPT-3 cost ~$5m in early 2020, then it’ll cost $2.5m around mid-2021, and so on. Similarly, a GPT-human requiring 2.2e6× more compute would presumably cost on the order of $10 trillion in 2020, but after 14 halvings (18 years) would cost $1b in 2038.
Metaculus currently seems to be roughly in between 2027 and 2038 right now, incidentally.
It still is, it’s just that beam search (or other search strategies) seem to be mostly useful for closed-end short text generation; translating a sentence apparently is a task with enough of a right-or-wrong-ness to it that beam search apparently taps into no pathologies. But they get exposed for open-ended longform generation.
It’s probably a lower bound. These datasets tend to be fairly narrow by design. I’d guess it’s more than 2x across all domains globally. And cutting the absolute loss by 50% will be quite difficult. Even increasing the compute by 1000x only gets you about half that under the best-case scenario… Let’s see, to continue my WebText crossentropy example, 1000x reduces the loss by about a third, so if you want to halve it (we’ll assume that’s about the distance to human performance on WebText) from 1.73 to 0.86, you’d need
(2.57 * (3.64 * (10^3 * x))^(-0.048)) = 0.86where x = 2.2e6 or 2,200,000x the compute of GPT-3. Getting 2.2 million times more compute than GPT-3 is quite an ask over the next decade or two.
Looking more into reported perplexities, the only benchmark which seems to allow direct comparison of human vs GPT-2 vs GPT-3 is LAMBADA.
LAMBADA was benchmarked at a GPT-2 perplexity of 8.6, and a GPT-3 perplexity of 3.0 (zero-shot) & 1.92 (few-shot). OA claims in their GPT-2 blog post (but not the paper) that human perplexity is 1-2, but provides no sources and I couldn’t find any. (The authors might be guessing based on how LAMBADA was constructed: examples were filtered by whether two independent human raters provided the same right answer.) Since LAMBADA is a fairly restricted dialogue dataset, although constructed to be difficult, I’d suggest that humans are much closer to 1 than 2 on it.
So overall, it looks like the best guess is that GPT-3 continues to have somewhere around twice the absolute error of a human.
and error and hyperparameter tuning that would probably increase the cost several-fold.
All of which was done on much smaller models and GPT-3 just scaled up existing settings/equations—they did their homework. That was the whole point of the scaling papers, to tell you how to train the largest cost-effective model without having to brute force it! I think OA may well have done a single run and people are substantially inflating the cost because they aren’t paying any attention to the background research or how the GPT-3 paper pointedly omits any discussion of hyperparameter tuning and implies only one run (eg the dataset contamination issue).
To simplify Daniel’s point: the pretraining paradigm claims that language draws heavily on important domains like logic, causal reasoning, world knowledge, etc; to reach human absolute performance (as measured in prediction: perplexity/cross-entropy/bpc), a language model must learn all of those domains roughly as well as humans do; GPT-3 obviously has not learned those important domains to a human level; therefore, if GPT-3 had the same absolute performance as humans but not the same important domains, the pretraining paradigm must be false because we’ve created a language model which succeeds at one but not the other. There may be a way to do pretraining right, but one turns out to not necessarily follow from the other and so you can’t just optimize for absolute performance and expect the rest of it to fall into place.
(It would have turned out that language models can model easier or inessential parts of human corpuses enough to make up for skipping the important domains; maybe if you memorize enough quotes or tropes or sayings, for example, you can predict really well while still failing completely at commonsense reasoning, and this would hold true no matter how much more data was added to the pile.)
As it happens, GPT-3 has not reached the same absolute performance because we’re just comparing apples & oranges. I was only talking about WebText in my comment there, but Omohundro is talking about Penn Tree Bank & 1BW. As far as I can tell, GPT-3 is still substantially short of human performance.
I think Omohundro is wrong here. His GPT-3 perplexity of 20.5 must be for Penn Tree Bank. However, his ‘humans’ perplexity of 12 is for a completely different dataset! Tracing his citations from his video to Shen et al 2017, which uses 1 Billion Word Benchmark. 1BW was not reported in the GPT-3 paper because it was one of the datasets affected by contamination and dropped from evaluation.
I’ve never read the Penn Tree Bank or 1BW so I can’t compare. At best, I’d guess that if 1BW is collected from “English newspapers”, that’s less diverse than the Brown Corpus which goes beyond newspapers, and so perplexities will be lower on 1BW than PTB. However, some searching turned up no estimates for human performance on either PTB or WebText, so I can’t guess what the real human vs GPT-3 comparison might be. I’m also a little puzzled what the ‘de-tokenizers’ are that the Radford GPT paper mentions are necessary for doing the perplexity calculations at all...
(There are a lot of papers estimating English text entropy in terms of bits per character, but because of the BPEs and other differences, I don’t know how to turn that into a perplexity which could be compared to the reported GPT-3 performance on Penn Tree Bank/WebText/LAMBADA/etc, which is why I didn’t include a human baseline in my comment there—I just don’t know.)
So, am I right in thinking that if someone took random internet text and fed it to me word by word and asked me to predict the next word, I’d do about as well as GPT-2 and significantly worse than GPT-3?
No.
Why is beam search missing? One possibility is that GPT-3 already does internal lookahead. OpenAI tried beam search, found it didn’t improve text generation, and didn’t bother adding it as an option. In other words, GPT-3 is already mesa-optimizing ὣ2
Beam search has never worked for likelihood-trained NNs, since at least char-RNNs back in 2015. Beam search does trigger repetition and other pathologies in GPT, see “The Curious Case of Neural Text Degeneration”, Holtzman et al 2019. And while unlikelihood training seems to help, it’s not a silver bullet, and is a bit ad hoc (especially if you think of it in terms of reinforcement learning).
Thus, it is probably important to be careful about not accelerating non-WBE neuromorphic AI while attempting to accelerate whole brain emulation. For instance, it seems plausible to me that getting better models of neurons would be useful for creating neuromorphic AIs while better brain scanning would not, and both technologies are necessary for brain uploading, so if that is true, it may make sense to work on improving brain scanning but not on improving neural models.
But what research improves brain imaging but not DL… One thing to point out about whole brain emulation vs ‘de novo’ AI is that it may be, in practice, nearly impossible to get WBEs without having already, much earlier, kickstarted ‘de novo’ AI.
If you can scan and run successfully a single whole brain, you got there by extensive brain imaging and brain scanning of much smaller chunks of many brains, and it seems like there is a lot of very transferable knowledge from the structure and activities of a human brain to artificial neural networks, which I dub “brain imitation learning”. Not only do ANNs turn out to have fairly similar activation patterns as human brains in some respects (primarily visual cortex stuff), the human brain’s activation patterns encode a lot of knowledge about how visual representations work which can be used to learn & generalize. (A particularly interesting example from this month is “Self-Supervised Natural Image Reconstruction and Rich Semantic Classification from Brain Activity”, Gaziv et al 2020.) You might consider this a version of the pretraining paradigm or lexical hypothesis—the algorithms of general intelligence, and world knowledge, are encoded in the connectivity and activation patterns of a human brain and so training on large corpuses of such data to imitate the connectivity & activation patterns will provide an extremely powerful prior/initialization à la GPT-3 pretraining on large text datasets.
So, it is entirely possible that by the time you get to BCIs or whole-brain scanning apparatuses, these are providing high-volume data embeddings or structural/architectural constraints which help push deep learning approaches over the finish line to AGI by providing informative priors & meta-learning capabilities by conditioning on <100% data from many brains. (In fact, if you believe this won’t happen, you have to explain what on earth is being done with all this extremely expensive data for decades on end, as it slowly ramps up from scanning insect-sized chunks to full monkey brains before finally an entire human brain is scanned 100% & they flip the giant red switch to make Mr John Smith, test subject #1918, wake up inside a computer. What is everyone doing before that?)
Whatever these DL systems may be, they won’t be a single specific person, and they won’t come with whatever safety guarantees people think an upload of Mr John Smith would come with, but they will come years or decades before.
Yes, there’s something to that, but you have to be careful if you want to use that as an objection. Maybe you wouldn’t easily think of it, but that doesn’t exclude the possibility of you doing it: you can come up with algorithms you can execute which would spit out Egan-like ideas, like ‘emulate Egan’s brain neuron by neuron’. (If nothing else, there’s always the ol’ dovetail-every-possible-Turing-machine hammer.) Most of these run into computational complexity problems, but that’s the escape hatch Egan (and Scott Aaronson has made a similar argument) leaves himself by caveats like ‘given enough patience, and a very large notebook’. Said patience might require billions of years, and the notebook might be the size of the Milky Way galaxy, but those are all finite numbers, so technically Egan is correct as far as that goes.
Equivocation. “Who’s ‘we’, flesh man?” Even granting the necessary millions or billions of years for a human to sit down and emulate a superintelligence step by step, it is still not the human who understands, but the Chinese room.
The latter. I didn’t notice it was a link to a different paper, but I think my point stands: the better results in this paper compared to the previous finetuning paper can’t be due to adding the KL constraint because they already had one. It has to be something else they changed, like more/better labels or bigger models.
The original paper & codebase definitely had KL penalties on the PPO policy. I spent a fair bit of time fiddling with it and letting it go high to see what adversarial ABC music examples it found in the hopes that it would train the reward model better when I labeled them. Didn’t seem to work, it would just find similar and only slightly different examples.
No, not yet. (IMO, the power of differentiability is greatly underused. Everyone is locked into a ‘optimize parameters based on data & loss’ mindset, and few ever use the alternatives like ‘optimize data/trajectory based on parameters & loss’ or ’optimize loss based on data/parameters.)
FWIW, Gwern reports trying OpenAI’s approach and finding the RL side specifically frustrating and unstable; this is pretty normal with RL, and compatible with the reward-model part being very successful in its own domain. It’s not clear whether OpenAI got the RL part to work well because they did something right, or because they have lots of resources and can keep trying over and over until it works.
At the time, I figured that it was probably a sample-efficiency problem: the reward model just wasn’t picking up on the subtle esthetics I wanted it to. I see this as supported by their new results: large models are more sample-efficient, so unsurprisingly, it works a lot better—the reward model can finally manage to understand what the preferences are, so it can provide a real signal to the RL training.
They seem to think it has more to do with label quality / better raters, which I didn’t think was my problem (who better than me to rate my preferred ABC samples?), but better label quality is sort of like better sample-efficiency; I haven’t read the paper in enough detail to see if they ablated model size vs label n vs label quality to get an idea of where the improvement is coming from.
Again, wouldn’t it be nice if we could avoid the need for this thing and just train on the preferences directly
Accept no substitutes! Gradient ascent directly on the differentiable reward/environment model!
Some new links on that topic: https://fraser-greenlee.github.io/2020/08/13/Transformers-as-Variational-Autoencoders.html https://fraser-greenlee.github.io/2020/08/25/Transformer-VAE-for-Program-Synthesis.html
Is there any connection to quadratic funding?