gwern
“Natural” can perhaps be defined with respect to the physical laws of this universe. So if you sum up the distributions from humans, Turning machines, tape recorders and everything else (properly weighted), you get a distribution for X if you knew X was a number being pulled out this physical universe. A god-distribution, if you wanna call it that—I’m sure you can come up with a better name for it.
Such as ‘universal prior’?
A question for the ‘Vassarites’, if they will: were you doing anything like the “unihemispheric sleep” exercise (self-inducing hallucinations/dissociative personalities by sleep deprivation) the Zizians are described as doing?
An incentive to make the existing SOTA seem as bad as possible, to maximize the gap between it and your own new, sparkly, putatively superior method.
Here’s an eyerolling example from yesterday or so: Delphi boasts about their new ethics dataset of n=millions & model which gets 91% vs GPT-3 at chance-level of 52%. Wow, how awful! But wait, we know GPT-3 does better than chance on other datasets like Hendrycks’s ETHICS, how can it do so bad where a much smaller model can do so well?
Oh, it turns out that that’s zeroshot with their idiosyncratic format. The abstract just doesn’t mention that when they do some basic prompt engineering (no p-tuning or self-distillation or anything) and include a few examples (ie. a lot fewer than ‘millions’), it gets more like… 84%. Oh.
“Accelerating progress in brain recording tech”? One reason to be optimistic about brain imitation learning: we may just be in the knee of the curve, well before the curves cross.
I wouldn’t call it an ‘out-of-context problem’. Thrawn understood exactly what happened as soon as the knife started going through his chest, because he always knew what a dangerous double-game he was playing with the Noghri, and the consequences of them discovering how they had been tricked; he just couldn’t help himself, no matter how disastrous any slipup would be, because it was too clever a trick to not play. (Again, look up the meaning of the word ‘thrawn’. Zahn didn’t pick it by accident.)
I think that this is notable because it’s the first time we’ve really seen powerful AI research orgs sharing infra like this. Typically everyone wants to do everything bespoke and make their work all on their own. This is good for branding but obviously a lot more work.
It may just be the incentives. “Commoditize your complement”.
Nvidia wants to sell GPUs, and that’s pretty much it; any services they sell are tightly coupled to the GPUs, they don’t sell smartphones or banner ads. And Microsoft wants to sell MS Azure, and to a lesser extent, business SaaS, and while it has many fingers in many pies, those tails do not wag the dog. NV/MS releasing tooling like DeepSpeed, and being pragmatic about using The Pile since it exists (instead of spending scarce engineer time on making one’s own just to have their own), is consistent with that.
In contrast, Facebook, Google, Apple, AliBaba, Baidu—all of these sell different things, typically far more integrated into a service/website/platform, like smartphones vertically integrated from the web advertising down to the NN ASICs on their in-house smartphones. Google may be unusually open in terms of releasing research, but they still won’t release the actual models trained on JFT-300M/B or ALIGN, or models touching on the core business vitals like advertising, or their best models like LaMDA* or MUM or Pathways. Even academics ‘sell’ very different things than happy endusers on Nvidia GPUs / MS cloud VMs: prestige, citations, novelty, secret sauces, moral high grounds. Not necessarily open data and working code.
* The split incentives lead to some strange behavior, like the current situation where there’s already like 6 notable Google-authored papers on LaMDA revealing fascinating capabilities like general text style transfer… all of which won’t use its name and only refer to it as “a large language model” or something. (Sometimes they’ll generously specify the model in question is O(100b) parameters.)
I’ve pointed out the cases of Moravec (1997) and Shane Legg pre-DM (~2009) as saying pretty much exactly that and in the case of Legg, influencing his DM founding timeline. I am pretty sure that if you were able to go back and do a thorough survey of the connectionist literature and influenced people, you’d find more instances.
For example, yesterday I was collating my links on AI Dungeon and I ran into a 1989 text adventure talk by Doug Sharp mostly about his King of Chicago & simple world/narrative simulation approach to IF, where before discussing King, to my shock, he casually drops in Moravec’s 1988 Mind Children’s forecast for human-level compute in 2030 and compute as a prerequisite for “having this AI problem licked”, and notes
When true AI does arrive, I think it will pretty much sweep away the groundwork we’re laying today and set up its own rules for interaction, narration, and a few other details. In the meantime, there’s a lot of fun to be had making the best games we can with our primitive machines and methods.
Well, I can’t disagree with that! It’s only 2021, and AI Dungeon and its imitators owe essentially nothing to the last 46 years of IF, and have to invent methodologies for neural text games from scratch. But it’s not very fun to just give up in 1986 and say you’ll sit around twiddling your thumbs for the next 33 years or so, waiting for Moore’s law to give you the most rudimentary NNs you can start experimenting with...
(Image is broken. Expired link?)
Zahn probably got the idea from the many anecdotes/stories of https://en.wikipedia.org/wiki/Military_animal#As_living_bombs historically going back before the Mongols to even https://en.wikipedia.org/wiki/Olga_of_Kiev#Drevlian_Uprising (or earlier https://historyofyesterday.com/5-bizarre-uses-of-animals-as-weapons-in-war-by-armies-7a57108afcb ).
I’m sure Zahn knows at least some of them: they are a semi-common trivia point, and stealing from military history is a time-honored strategy—history is far more clever and imaginative than you are, it has built-in verisimilitude (even if they wound up turning out to have never happened, clearly a lot of people believed they could have happened), readers won’t notice, and the ones who do will appreciate your allusion/twist.
If you’ve read the Thrawn books, do you agree with my analysis of why they end up working? Are there factors I missed?
I haven’t read the new ones, as I swore off the EU long ago after finding the NJO cringe, but I find it interesting how different your description is from the original Thrawn trilogy. After all, the entire point of it was that Thrawn was eventually killed at the height of his success when a needlessly elaborate and over-convoluted scheme (recall the definition of the word ‘thrawn’) he executed because it was ‘so artistically done’ backfired on him & he was assassinated. (Planning your campaigns by staring at random artworks for hours is, however entertaining as a novel, also not the sort of thing real generals do to win battles.)
Why would training GPT-3 on its own output improve it at all?
Self-distillation is a thing, even outside a DRL setting. (“Best-of” sampling is similar to self-distillation in being a way to get better output out of GPT-3 using just GPT-3.) There’s also an issue of “sampling can prove the presence of knowledge but not the absence” in terms of unlocking abilities that you haven’t prompted—in a very timely paper yesterday, OA demonstrates that GPT-3 models have much better translation abilities than anyone realized, and you can train on its own output to improve its zero-shot translation to English & make that power accessible: “Unsupervised Neural Machine Translation with Generative Language Models Only”, Han et al 2021.
That was just the tech demo. It obviously wasn’t actually trained, just demoed for a few steps to show that the code works. If they had trained it back then, they’d, well, have announced it like OP! OP is what they’ve trained for-reals after further fiddling with that code.
Not very, now. Compute & implementation are more limiting.
Corpus quality definitely affects how much bang per FLOP you get, at a moderate (neither extreme nor negligible) sort of constant factor. The Pile is better than what OA used, so an otherwise-identical GPT-3 trained on it would be noticeably better. (But this only goes so far and you will have a hard time doing much better than The Pile in terms of cleaner higher-quality text.)
Corpus size is unimportant because existing corpuses are enough: the optimal training method is to do 1 epoch, never reusing any data. But you are usually limited by compute, and the compute you have uniquely specifies the model size and n. Even for Nvidia/MS using their supercomputer with 5k+ A100s, that n is a lot smaller than what The Pile etc already contains. (See the part about over/undersampling and how few tokens they trained on compared to the full dataset. GPT-3 did the same thing: oversampled WP and books, but then didn’t use more than a fraction of their CC subset etc.) So in other words, PMC vs PM is irrelevant because even if you bothered to get the full PM corpus, they already had more text than they could afford to train on. They don’t want to throw out the other data in order to train on mostly PM, so just PMC is fine.
(When you have too much data, what you can do to get some value out of it is you can filter it more aggressively to cut it down to just that amount you can afford—but filtering itself is an open research challenge: lots of nasty software engineering, and you may wind up sabotaging your model if you eliminate too much data diversity by mistaking diverse difficult data for bad data & filtering it out. And if you do it right, you’re still limited by the first point that data cleaning only buys you so much in constant factors.)
In the rest of this post, we informally describe two versions of this project in more detail: An idealized version using XLand that actors with access to large amounts of compute could carry out, as well as a scaled-down and less resource-intensive version that is more realistic to pursue.
I wouldn’t use or try to reproduce XLand for open research. Procgen or Obstacle Tower might make more sense. Griddy, Alchemy, Meta-World, come to mind; or Minihack is an interesting new choice as a kind of procedurally-generated Nethack (there’s also the new Crafter but unclear to me if it’d let you really mix up item types for curriculum/generation of rules rather than merely new levels).
As an analogy, imagine GPT-3 would have only been trained and evaluated on abstract language, or sentences beginning with a subject, using the vocabulary of basic English. How confident would you be in extrapolating from such a model’s scaling with compute to the less restricted language models we have at this point?
Pretty confident? I mean, you’ve read the scaling papers. You know the upshot is that scaling curves are remarkably universal cross-modality, cross-architecture, cross-dataset, and cross-noise levels: they all look like straight lines on log graphs. The differences are usually in the constants, not the exponents, so I would expect a subsetted GPT-3 to have a lower constant in prediction loss on its constrained dataset and be that much closer to its intrinsic entropy, but otherwise extrapolate its scaling laws of loss vs parameters/FLOPS/n the same as always.
I think you might be trying to say that we would have more doubts about its performance on downstream tasks, such as benchmarks, and whether abilities like meta-learning would be induced? There, yes, we don’t have good ideas as to how much diversity matters or even how to measure either diversity or downstream tasks. (One would expect there to be some sort of bias-variance-esque tradeoff where diversity is good but the data can’t be too sparse as to be unlearnable: if pure diversity were the only goal, then you’d expect datacleaning would harm both perplexity & downstream performance, but we know that for language models, cleaning Internet data pays off. Beyond this, hard to say. We know that you probably want to broaden environments/data as you accumulate more and more data in any specific environment or task as they saturate, but...)
As an example, it seems quite unlikely that a system trained on English language only is going to be very useful for working with German text.
En->De: “Scaling Laws for Language Transfer Learning”, Christina Kim (as expected from Hernandez et al 2021).
If DeepMind had used a large fraction of their available compute resources on XLand, rerunning the code with a large number of variations might not be feasible, even for them.
Maybe he was embarrassed by the mistakes he made, like making up 3 different wrong citations for a claim about Audrey Tang despising rationalists (which was also not true), and reflected a bit.
Is the meta-learned net able to learn any other function at all and is not just frozen, or is the meta-learned stability tailored to protecting against specific tasks like sin(x)?
The biologist answer there seems to be question-begging. What reason is there to think it isn’t? Animals can’t split and merge themselves or afford the costs or store datasets for exact replay etc, so they would be unable to do that whether or not it was possible, and so they provide zero evidence about whether their internal algorithms would be able to do it. You might argue that there might be multiple ‘families’ of algorithms all delivering animal-level intelligence, some of which are parallelizable and some not, and for lack of any incentive animals happened to evolve a non-parallelizable one, but this is pure speculation and can’t establish that the non-parallelizable one is superior to the others (much less is the only such family).
From the ML or statistics view, it seems hard for parallelization in learning to not be useful. It’s a pretty broad principle that more data is better than less data. Your neurons are always estimating local gradients with whatever local learning rule they have, and these gradients are (extremely) noisy, and can be improved by more datapoints or rollouts to better estimate the update that jointly optimizes all of the tasks; almost by definition, this seems superior to getting less data one point at a time and doing noisy updates neglecting most of the tasks.
If I am a DRL agent and I have n hypotheses about the current environment, why am I harmed by exploring all n in parallel with n copied agents, observing the updates, and updating my central actor with them all? Even if they don’t produce direct gradients (let’s handwave an architecture where somehow it’d be bad to feed them all in directly, maybe it’s very fragile to off-policyness), they are still producing observations I can use to update my environment model for planning, and I can go through them and do learning before I take any more actions. (If you were in front of a death maze and were watching fellow humans run through it and get hit by the swinging blades or acid mists or ironically-named boulders, you’d surely appreciate being able to watch as many runs as possible by your fellow humans rather than yourself running it.)
In particular, for non-superhuman AIs-in-training, we already have tons of pedagogical materials like human textbooks and lectures. So I don’t see teams-of-AIs-who talk-to-each-other being all that helpful in getting to superhuman faster.
If we look at some of these algorithms, it’s even less compelling to argue that there’s some deep intrinsic reason we want to lock learning to small serial steps—look at expert iteration in AlphaZero, where the improved estimates that the NN is repeatedly retrained on don’t even come from the NN itself, but an ‘expert’ (eg a NN + tree search); what would we gain by ignoring the expert’s provably superior board position evaluations (which would beat the NN if they played) and forcing serial learning? At least, given that MuZero/AlphaZero are so good, this serial biological learning process, whatsoever it may be, has failed to produce superior results to parallelized learning, raising questions about what circumstances exactly yield these serial-mandatory benefits...
The parallelization discussion seems offbase to me. While it is of course important that any individual instance runs not too absurdly slowly, how much faster than realtime it runs isn’t that important, because you would be running many of them in parallel, no? AlphaZero trained in a few wallclock hours not by blazing through games in mere nanoseconds, but by having hundreds or thousands of actors in parallel playing through games at a reasonable speed like 0.05s per turn or something. Or OA5 used minibatches of millions of experiences, and GPT-3 had minibatches of like millions of tokens, IIRC.
If we look at the gradient noise scale, the more complicated the ‘task’ (ie set of tasks), the larger the batch size you need/can use before you are just wasting compute by overly-precisely estimating the gradient for the next update. Presumably any AGI would be training on a lot of tasks as complicated as Go or English text or DoTA2 or more complicated: generative and discriminatory multimodal training on text, video, and photos, DRL training on a bazillion games and procedurally-generated tasks, and so on, and so the optimal minibatch size would be quite large… Unless the hardware overhang is vastly more extreme than anyone anticipates (in which case the debate would be moot for other reasons), it seems like the most plausible answer for “how much parallel hardware can my seed AGI use?” is going to be “how much ya got?”.
This doesn’t guarantee a fast wallclock, of course, but it’s worth noting that in the limit of (full-batch, not stochastic minibatching) gradient descent, you can generally take large steps and converge in relatively few serial iterations compared to SGD. (Bunch of papers on scaling up CNNs to training on thousands of GPUs simultaneously to converge in minutes to seconds rather than days or weeks on smaller but more efficient clusters; yesterday I saw Geiping et al 2021 whose CNN requires 3,000 serial fullbatch iterations vs SGD’s 117,000 serial minibatch iterations, so hypothetically, you could finish in 39x less wallclock if you had ~unlimited compute.)
So even for an incredibly complicated family of tasks, as long as the individual instances can be run at all, the wallclock is potentially quite low because you have model parallelism out the wazoo within and across all of the tasks & modalities & problems, and only need to take relatively few serial updates.
One paper I forgot that bears especially on the question of why you use planning: “On the role of planning in model-based deep reinforcement learning”, Hamrick et al 2020:
Model-based planning is often thought to be necessary for deep, careful reasoning and generalization in artificial agents. While recent successes of model-based reinforcement learning (MBRL) with deep function approximation have strengthened this hypothesis, the resulting diversity of model-based methods has also made it difficult to track which components drive success and why. In this paper, we seek to disentangle the contributions of recent methods by focusing on three questions: (1) How does planning benefit MBRL agents? (2) Within planning, what choices drive performance? (3) To what extent does planning improve generalization? To answer these questions, we study the performance of MuZero (Schrittwieser et al., 2019), a state-of-the-art MBRL algorithm with strong connections and overlapping components with many other MBRL algorithms. We perform a number of interventions and ablations of MuZero across a wide range of environments, including control tasks, Atari, and 9x9 Go. Our results suggest the following: (1) Planning is most useful in the learning process, both for policy updates and for providing a more useful data distribution. (2) Using shallow trees with simple Monte-Carlo rollouts is as performant as more complex methods, except in the most difficult reasoning tasks. We show that deep, precise planning is often unnecessary to achieve high reward in many domains, with 2-step planning exhibiting surprisingly strong performance even in Go. (3) Planning alone is insufficient to drive strong generalization. These results indicate where and how to utilize planning in reinforcement learning settings, and highlight a number of open questions for future MBRL research.