gwern
More biology examples: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681688/
Michael Dennis tells me that population-based training typically sees strong diminishing returns to population size, such that he doubts that there were more than one or two dozen agents in each population/generation.
Makes sense given the spinning-top topology of games. These tasks are probably not complex enough to need a lot of distinct agents/populations to traverse the wide part to reach the top where you then need little diversity to converge on value-equivalent models.
Has the price dropped since AlphaStar?
One observation: you can’t run SC2 environments on a TPU, and when you can pack the environment and agents together onto a TPU and batch everything with no copying, you use the hardware closer to its full potential, see the Podracer numbers.
I was reading about endgame databases and happened to note Thompson discussing a bit scaling of brute force search, and it ‘asymptotic’ or flattening. Ken Thompson, oral history, February 2005
...
Q: So at that stage how many plies was it [Belle] going and what kind of rating did it have?Ken Thompson: It was going 8 full plies easily, the other one was probably going 6 and was easily master, probably 2300. I never worked for rating. You can squeeze rating points out by studying your opponents and booking up and taking away—put a punch of trap kind of stuff in. But I wanted to just play nice solid chess, so it was probably 2300.Q: And I think at some point I recall you talking about the ratings compared to compute power and you had I think extrapolated for and about what kinds of compute power it would take to get certain improvements and ratingsKen Thompson: Yeah...Q: I can’t remember when that was, it must have been early ’80s or something...Ken Thompson: Yeah it was early ’80s—It was asymptotic but it was in the high rise part of the asymptote so you can’t really tell whether—where it rolled over. But it would, certainly in that era you were gaining—for every doubling of horsepower you’d gain a hundred points. I came with some really fake-y back of the envelope calculations that said at around 12 or 13 ply you’d be world class. Maybe not—you’d compete for the world title, you may not win it but you’d be comparable.Q: Presumably the ratings in some sense get a little less accurate at the very top I would think...Ken Thompson: Yes, yes.The WP article on Belle notes some of this in more detail and gives two refers, a 1983 and 1997 book. Quoting a bit (both are on Libgen and easy to acccess):
1983 says
Newborn [155] has suggested that if you double the speed of a brute force program, you will gain 100 rating points. If you assume a 5.5 branching factor, then a ply will yield 250 points. In the linear range of P4-P6 (the range observed by Newborn in existing programs) this relation holds. In the extended range of this experiment, ratings do not grow as fast as predicted. Any major conclusions about the experiment we leave up to the reader. Here are some of our observations that may be of help.
From [the book](https://cloudflare-ipfs.com/ipfs/bafykbzacebnknut4q77j4foxnybstbry5w3g5poigcqase47g4cryanhweags?filename=Monty Newborn (auth.) - Kasparov versus Deep Blue_ computer chess comes of age-Springer-Verlag New York (1997).pdf#page=132)
Thompson conducted two experiments in the late 1970s to study this issue. In his first experiment, he had one version of BELLE play a series of twenty games against an identical second version, except that one version, call it BELLE(P), searched to a depth of, say, P levels while the other, BELLE(P + 1), searched to a depth of P + 1 levels. The results of this experiment are presented in Figure 6.24a for 3 ⇐ P ⇐ 8. It shows, for example, that BELLE(3) won four of the twenty points against BELLE(4), and that BELLE(4) won five and a half of twenty points against BELLE(5). His second experiment, carried out shortly thereafter, was more extensive: for 4 ⇐ P, Q ⇐ 9 and for P ≠ Q, BELLE(P) played a twenty-game series against BELLE(Q). The results are presented in Figure 6.24b. Rather than relating rating improvement to the number of positions searched, Thompson related rating improvement to the depth of search, pointing out that an improvement of one hundred points for each doubling of speed was approximately equivalent to an improvement of 250 points for each additional level of search.
The main difference between the curve of ratings shown in Figure 6.23 and Thompson’s results is that the relation between ratings and search depth appears to be almost linear to about 2500 while Thompson found that rating improvement drops off from linear at ratings above 2000 points. Certainly, eventually the curve approaches the limit imposed by perfect play but the question is how fast. Thompson evidently did not study deeper search depths because of the large amount of time required to obtain results.
[The next part discusses a different way to try to extrapolate based on differences in tree search choice which reminds me a little of Paul Christiano’s counting argument/coincidence in Andy Jone’s RL scaling paper.]
The descriptions here of flattening are suspicious, given the economizing on compute. It doesn’t look like Thompson ever clearly revisited the topic later on than the ’80s, or considered functional forms beyond increasing 1 ply at a time and expecting n ELO points. I would not be surprised if Belle continued scaling the same as Jones, and here again practitioners mistake a log or powerlaw for ‘flattening out’ (and thus futility compared to Bitter-Lesson-vulnerable-but-more-engineerable approaches).
It should be named, and someone should describe its essential pattern:
If you don’t like “slippery slope”, how about “unraveling”?
Looking qualitatively at our agents, we often see general, heuristic behaviours emerge — rather than highly optimised, specific behaviours for individual tasks. Instead of agents knowing exactly the “best thing” to do in a new situation, we see evidence of agents experimenting and changing the state of the world until they’ve achieved a rewarding state.
The blessings of scale strike again. People have been remarkably quick to dismiss the lack of “GPT-4” models as indicating that the scaling hypothesis is dead already. (The only scaling hypothesis refuted by the past year is the ‘budget scaling hypothesis’, if you will. All the other research continues to confirm it.)
Incidentally, it’s well worth reading the previous papers from DM on using populations to learn ever more complex and general tasks: AlphaStar, Quake, rats, VR/language robotics, team soccer.
We’ve all read about AlpaFold 2, but I’d also highlight VQ-VAE being used increasingly pervasively as a drop-in generative model; “Multimodal Few-Shot Learning with Frozen Language Models”, Tsimpoukelli et al 2021, further demonstrating the power of large self-supervised models for fast human-like learning; and the always-underappreciated line of work on MuZero for doing sample-efficient & continuous-action model-based RL: “MuZero Unplugged: Online and Offline Reinforcement Learning by Planning with a Learned Model”, Schrittwieser et al 2021; “Sampled MuZero: Learning and Planning in Complex Action Spaces”, Hubert et al 2021 (benefiting from better use of existing compute, like Podracer).
If we were to hypothesise that some environmental factor is causing a significant fraction of the obesity problem then how would we test it?
The statistical silver bullet you’re looking for, assuming you can’t run multi-decade RCTs or convince entire countries to dramatically change their water policies (sounds a bit tricky?), is variance components analysis.
You want a variance component analysis equivalent to heritability, but for water/obesity. This is a perfect match: you have a very high-dimensional measurement (GC-MS of drinking water & foodstuffs), whose components you can’t observe because you don’t even know how many there are much less which ones (like the unknown chemicals that the original contaminants may react into along the way**), where direct prediction/regression would fail because it requires absurd amounts of data (especially given the statistical issues they outline), and where your original research question is not “where is the needle in the haystack” but “does this haystack have any needles at all”.
In behavioral genetics, if you plot pairs of people by their genetic similarity and their phenotypic similarity, you will find that people similar on both tend to cluster; and the more heritable the trait is, the more they cluster. You can do this for kinds of similarity which aren’t ‘genetic’. The idea of variance components is to switch from directly correlating/regressing the specific chemical levels in water, trying to measure the exact direct effect of each chemical on obesity, to instead asking, “how similar, in terms of overall water chemical composition, are similarly far or thin people? Do the chemical spectrums of fat people tend to look similar, and the spectrums of thin people look similar? Do they co-vary, and cluster?”
Think of the water GC-MS as a sort of fingerprint or high-dimensional summary. It doesn’t need to contain the actual critical datapoint, just a lot of correlates/proxies thereof (eg you can just shine some light on leaves and it’ll work!). The point is not to directly measure the culprit, but to indirectly measure how distant the datapoints are; perhaps chemical A shows up and chemical B does not show up clearly, and B is the real cause, but as long as chemical A correlates with chemical B (maybe it’s a different reaction product, or just gets manufactured or used in close correlation with B), then you will see that fatties tend to have close-together water spectra (because of A) and you will have strong evidence that “there is something in the water” because where the bad water clusters the fatties cluster.
So, you’d do something like record the BMI of 10,000 people, take a sample of their drinking water, GC-MS it (ideally, but many other analytical techniques could be used and maybe some would be better here, I’ll defer to any actual analytical chemists on that topic), and then plug into a mixed linear-model. You can plug in covariates as necessary (families/pedigrees to jointly model genetic heritability, ZIP codes, age, sex, local mining/factories, already known harmful contaminants the GC-MS detected, that sort of thing). This can be done at multiple levels: individuals nested in households, within regions, within countries, etc. The amount of covariance between the water spectrum and BMI measures the amount of variance the contents of the water en masse can explain, and depending on measurement error and whatnot, given the genetic heritability bounding how much water-heritability can explain, you should get somewhere <30%. Under most theories of obesity, it’s difficult to see why one would predict a water-heritability different from 0%*, especially if one has already included altitude/SES/household/region as covariates (which should cover all the obvious confounders like “maybe contaminated water is just a proxy for living in a poor region / having mining nearby / being poor”), so anything >0% is highly suspicious.
This doesn’t require any RCTs (or interventions of any kind), longitudinal datasets, n in the millions, or any of the barriers to most of the proposed tests. You only need a cross-section of obesity/water-samples. Individual level is ideal, but doing it with geographic units can be useful too if that is what is available across enough units (individual n are much nosier than k, but also much easier to get high statistical power with). This may even already exist! Drinking water is often tested for many reasons, and obesity health data is standard medical data collected in almost any relevant research; so there ought to be a lot of possible cross-references.
* you could do this with food as well but I focus on water because water is a lot more consistent day to day, and food results would be ambiguous. A single day’s food is still not very representative of individual consumption (maybe your wife cooks healthy at home but then you eat processed garbage at the office, and the sample comes from the weekend), so it provides a very poor measurement of the overall diet’s chemical signature. And if you demonstrated a large food-heritability of obesity, this would just be taken as proving that “see? sugar / fat / calories / highly-palatable processing causes everything, just like we said all along!”, and be weak evidence for contamination at best.
** They also raise the problem of interactions & nonlinear responses. These are issues in behavioral genetics too, of course, with dominance/epistasis particularly, and can be dealt with similarly in the water context: “narrowsense” vs “broadsense” heritability covers simple additive vs interactions, and you can do things like DeFries-Fulker, I think, for nonlinear responses. My guess is that contamination would be ‘additive’ mostly, for similar underlying reasons as genetics tends to be ‘additive’: many interactions are just constants because they are done on a fixed background of chemistry, and promiscuous interactions average out to additive effects. But this is the sort of thing to worry about only after having done the straightforward work.
Rohin’s opinion: I really liked the experiment demonstrating misalignment, as it seems like it accurately captures the aspects that we expect to see with existentially risky misaligned AI systems: they will “know” how to do the thing we want, they simply won’t be “motivated” to actually do it.
In the end humanity was saved by adding “Super safe.” to all their requests of the AGI
My counter joke (in EAI) was:
“AGI but its supr safe.”
(GPT-3 is an agent-predicting agent.)
Might be good to revisit this idea in light of NFTs.
Has anyone pointed out to Chollet yet that scaling up bottle rockets is a great way to go to space?
That’s impressive, considering D-K isn’t real.
It’s based on those estimates and the systematic biases in such methods & literatures. Just as you know that psychology and medical effects are always overestimated and can be rounded down by 50% to get a more plausible real world estimate, such information-theoretic methods will always overestimate model performance and underestimate human performance, and are based on various idealizations: they use limited genres and writing styles (formal, omitting informal like slang), don’t involve extensive human calibration or training like the models get, don’t involve any adversarial examples, don’t try to test human reasoning by writing up texts made up of logical riddles and puzzles or complicated cause-and-effect scenarios or even things like Winograd Schemas, are time-biased, etc. We’ve seen a lot of these issues come up in benchmarking, like ImageNet models outside ImageNet despite hitting human parity or superiority. (If we are interested in truly testing ‘compression = intelligence’, we need texts which stress all capabilities and remove all of those issues.)
So given Shannon’s interval’s lower end is 0.6, and Grassberger’s asymptotic is 0.8 (the footnote 11) and a widespread of upper bounds going down to 1.3 along with extremely dumb fast algorithms hitting 2, I am comfortable with rounding them downish to get estimates of 0.7 bpc being the human performance; and I expect that to, if anything, be still underestimating true human peak performance, so I wouldn’t be shocked if it was actually more like 0.6 bpc.
And some of the past QC claims for ML have not panned out. Like, I think there was a Quantum Monte Carlo claimed to be potentially useful for ML which could be done on cheaper QC archs, but then it turned out to be doable classically...? In any case, I have been reading about QCs all my life, and they have yet to become relevant to anything I care about; and I assume Scott Aaronson will alert us should they suddenly become relevant to AI/ML/DL, so the rest of us should go about our lives until that day.
The obvious difference is that the second does not include Elgazzar, while the first includes Elgazzar, which is bad for the first one because Elgazzar faked its data so incompetently it has been retracted: https://grftr.news/why-was-a-major-study-on-ivermectin-for-covid-19-just-retracted/ https://gidmk.medium.com/is-ivermectin-for-covid-19-based-on-fraudulent-research-5cc079278602 https://www.theguardian.com/science/2021/jul/16/huge-study-supporting-ivermectin-as-covid-treatment-withdrawn-over-ethical-concerns
The previous AF2 discussions were largely a waste of space because the little abstract they had to provide for CASP14 provided hardly anything to go on. But now we have not just a full writeup but source code and models too! Now I consider it worth discussing.
Moore: yes.
QC: AFAIK it’s irrelevant?
GPT-3: it used up a particular kind of overhang you might call the “small-scale industrial CS R&D budget hardware overhang”. (It would certainly be possible to make much greater than GPT-3-level progress, but you’d need vastly larger budgets: say, 10% of a failed erectile-dysfunction drug candidate, or 0.1% of the money it takes to run a failed European fusion reactor or particle collider.) So, I continue to stand by my scaling hypothesis essay’s paradigm that as expected, we saw some imitation and catchup, but no one created a model much bigger than GPT-3, never mind one that was >100x bigger the way GPT-3 was to GPT-2-1.5b, because no one at relevant corporations truly believes in scaling or wishes to commit the necessary resources, or feels that it’s near a crunchtime where there might be a rush to train a model at the edge of the possible, and OA itself has been resting on its laurels as it turns into a SaaS startup. (We’ll see what the Anthropic refugees choose to do with their $124m seed capital, but so far they appear to be making a relaxed start of it as well.)
The overhang GPT-3 used up should not be confused with other overhangs. There are many other hardware overhangs of interest: the hardware overhang of the experience curve where the cost halves every year or two; the hardware overhang of a distilled/compressed/sparsified model; the hardware overhang of the global compute infrastructure available to a rogue agent. The small-scale industrial R&D overhang is the relevant and binding one… for now. But the others become relevant later on, under different circumstances, and many of them keep getting bigger.
Given the timing of Jensen’s remarks about expecting trillion+ models and the subsequent MoEs of Switch & Wudao (1.2t) and embedding-heavy models like DLRM (12t), with dense models still stuck at GPT-3 scale, I’m now sure that he was referring to MoEs/embeddings, so a 100t MoE/embedding is both plausible and also not terribly interesting. (I’m sure Facebook would love to scale up DLRM another 10x and have embeddings for every SKU and Internet user and URL and video and book and song in the world, that sort of thing, but it will mean relatively little for AI capabilities or risk.) After all, he never said they were dense models, and the source in question is marketing, which can be assumed to accentuate the positive.
More generally, it is well past time to drop discussion of parameters, and switch to compute-only as we can create models with more parameters than we can train (you can fit a 100t-param with ZeRo into your cluster? great! how you gonna train it? Just leave it running for the next decade or two?) and we have no shortage of Internet data either: compute, compute, compute! It’ll only get worse if some new architecture with fast weights comes into play, and we have to start counting runtime-generated parameters as ‘parameters’ too. (eg Schmidhuber back in like the ’00s showed off archs which used… Fourier transforms? to have thousands of weights generate hundreds of thousands of weights or something. Think stuff like hypernetworks. ‘Parameter’ will mean even less than it does now.)
Using CLIP is a pretty weird way to go. It’s like using a CNN classifier to generate images: it can be done, but like a dog walking, we’re more surprised to see it work at all. If you think about how a contrastive loss works, it’s less surprising why CLIP-guided images look the way they do, and do things like try to repeat an object many time: if you have a prompt like “Mickey Mouse”, what could be even more Mickey-Mouse-y than MM tiled a dozen times? That surely maximizes its embedding encoding ‘Mickey Mouse’, and its distance from non-Disney-related image embeddings like “Empire State Building”! Whether you can really induce sharp coherent images from any scaled-up CLIP like ALIGN is unclear: contrastive losses just might not learn these things, and the generative model needs to do all the work. (CLIP would still remain useful as an extremely handy way to take any image generative model and quickly turn it into an text-editable model, or possible text->image model. One could probably do better by an explicit approach like DALL-E, but CLIP is available now and DALL-E is not.)
If you are interested in SOTA image generation quality rather than weird CLIP hacks, you should look at:
“VQGAN: Taming Transformers for High-Resolution Image Synthesis”, Esser et al 2020
“SR3: Image Super-Resolution via Iterative Refinement”, Saharia et al 2021 / “Diffusion Models Beat GANs on Image Synthesis”, Dhariwal & Nichol 2021
“Alias-Free GAN (Generative Adversarial Networks)”, Karras et al 2021
If you want to zero out mutational load and create a modal genome, you’re approximately 2 orders of magnitude off in the number of edits you need to do. (The number of evolutionarily-conserved protein-coding regions has little to do with the number of total variants across the billions of basepairs in a specific human’s genome.) Considering that it is unlikely that we will get base editors, ever, which have total error rates approaching 1 in millions, one will have to be a little more clever about how one goes about it. (Maybe some sort of multi-generational mass mutagenesis-editing / screening loop?)
Anyway, I would point out that you can do genome synthesis on a much cheaper scale than whole-genome: whole-chromosome is an obvious intermediate point which would be convenient to swap out. And for polygenic traits, optimizing a single chromosome might push the phenotype out as far as you want to go in a single generation anyway.
Yes, I see that they used Unity, so the TPUs themselves couldn’t run the env, but the TPU CPU VM* could run potentially a lot of copies (with that like 300GB of RAM it’s got access to), and that’d be a lot nicer than running remote VMs. At least in Tensorfork, when we try to use TPU pods, a lot of time goes into figuring out correct use of the interconnect & traffic because the on-TPU ops are so optimized by default.
(And regardless of which of those tricks this open-ended paper uses, this is a point well worth knowing about how research could potentially gets way more performance out of a TPU pod than one would expect from knowing TPU usage of old stuff like AlphaStar.)
* advertisement: access to the VM was recently unlocked for non-Google TPU users. It really changes how you treat TPU use!