gwern
I would point out that it is in fact (a lot) easier to heal a rabbit kidney from the toxins that are put into it to vitrify it than to completely recreate a rabbit kidney.
December 2020 gwern.net links
That’s surprising low for LibGen. Are you perhaps looking for fiction works and forgetting to search the fiction database? It’s separate from the nonfiction (one of a number of really bad usability issues LG has is fragmented databases).
Markets can remain irrational, and inflated by monetary/fiscal policy or consumer repression, longer than your options can remain unexpired. Among other things—the real world is far more imaginative than any trader. You might as well ask, ‘how did markets go up with the first coronavirus?’ Better to bet on large changes through the VIX or something, rather than their exact direction & timing...
Perhaps I’m missing something, but it seems like the obvious strategy is to just buy volatility-related instruments: you don’t know if the market is going to go up or down, but if the new strains work anything like one would predict based on the infectiousness and the vaccine rollouts continue their shambolic slow-walk out, there will be extreme movements (even if the Fed or USG then step in) in financial instruments of all sorts.
If you can’t find anyone willing to sell you bets on stability at reasonable prices, then it would seem that the general import of the strains - ‘the future just got very uncertain and potentially even worse than everyone feared’ - is already priced in.
OpenAI was initially funded with money from Elon Musk as a not-for-profit.
This is commonly said on the basis of his $1b pledge, but AFAICT Musk wound up contributing little or nothing before he resigned ~2018. If you look at the OA Form 990s, Musk is never listed as a donor, only a board member; the only entities that are listed as contributing money or loans are Sam Altman, Y Combinator Research, and OpenAI LP.
Finally, the scramble task is about shuffling around letters in the right way, and arithmetic is about adding, subtracting, dividing, and multiplying numbers. The main interesting thing about these tasks is that performance doesn’t improve at all in the beginning, and then starts improving very fast. This is some evidence that we might expect non-linear improvements on particular tasks, though I mostly interpret it as these tasks being quite narrow, such that when a model starts getting the trick, it’s quite easy to systematically get right.
To beat my usual drum: I think the Arithmetic/Scramble task curves are just due to BPEs. The ‘trick’ here is not that scrambling or arithmetic are actually all that difficult, but that it needs to memorize enough of the encrypted number/word representations to finally crack the BPE code and then once it’s done that, the task itself is straightforward. The ‘breakthrough’, so to speak, is seeing through the scrambled BPE representations. I predict that using tricks like rewriting numbers to individual digits or BPE-dropout to expose all possible tokenizations, or better yet, character-level representations, would show much smoother learning curves and that much smaller models would achieve the GPT-3-175b performance.
I have some discussion in https://www.gwern.net/Subscripts https://www.gwern.net/About#confidence-tags
BBC article: “The science of addiction: Do you always like the things you want?”, Edmonds.
I think OP might be engaged in a sarcastic Borgesian exercise of their own playing with plagiarism and repetition and is not actually saying there is a global conspiracy between OpenAI and a dozen publishers to steal their draft and commission famous authors to rewrite it; unfortunately, it winds up falling flat for me because it leans too heavily in to the green ink, as it were, and itself repeats too much. Needs editing down and a clearer observation.
People tend to be text-blind (particularly when skimming), so just writing
This is a linkpost for https://www.gwern.net/newsletter/2020/11
is not enough. A redesign using icons or color to help distinguish link from text posts might be useful. Perhaps a green circle on the header as well to be consistent with links, or a hooked arrow prepended to the text to point ‘out’/indicate movement?
November 2020 gwern.net newsletter
I don’t think it is: AlphaGo/Master/Zero hardly overlap with AlphaStar’s architecture, training, or category of RL algorithms, and AlphaFold 1 used a modest sprinkling of RL—but none of its RL techniques were used in either AlphaGo or AlphaStar!
It’s branding. Or to be more specific, it’s DM’s label not for ‘cool things’ but for “things we think will set world-records in their field and which will change them forever”.
You missed the third baby announced as part of the trial:
Dr. He, 35, sent the scientific world into an uproar last year when he announced at a conference in Hong Kong that he had created the world’s first genetically edited babies — twin girls. On Monday, China’s state media said his work had resulted in a third genetically edited baby, who had been previously undisclosed...“He violated medical ethics, disrespected life and let three poor children bear the consequences, all for his fame and fortune,” one user wrote. “I think this punishment is too light.”
The doubling strategy also has counterparts in array allocation strategies and algorithmic analysis (it’s common to double an array’s size each time it gets too large, to amortize the copying). Successive Halving, a racing algorithm, is also of interest here if you think of a portfolio of tasks instead of a single task.
So this is the 2-of-2 exploding Nash equilibrium technique applied to multiple parties/transactions? What’s this generalized kind called?
(On a side note, it now strikes me that there’s a parallel to RL blackbox optimization: by setting up a large penalty for any divergence from the golden path, it creates an unbiased, but high variance estimator of credit assignment. When pirates participate in enough rollouts with enough different assortments of pirates, they receive their approximate honesty-weighted return. You can try to pry open the blackbox and reduce variance by taking into account pirate baselines etc, but at the risk of losing unbiasedness if you do it wrong.)
I think I see ‘domain-specific datasets’ as broader than you do. You highlight Github, and yet, when I think of Github, I think of thousands of natural and artificial languages, tackling everything related to software in the world (which is increasingly ‘everything’), by millions of people, doing things like uploading banned books for evading the Great Firewall, filing bugs and discussing things back and forth, often adversarially, all reliant on common sense and world knowledge. A GPT trained on Github at hundreds of gigabytes I would expect to induce meta-learning, reasoning, and everything else, for exactly the same reasons CC/books1/books2/WP do; yes, it would know ‘source code’ well (not a trivial thing in its own right), but that is a mirror of the real world. I see plenty of broad domain coverage from ‘just’ Github, or ‘just’ Arxiv. (Literotica, I’m less sure about.) I don’t see Github as having much of a disadvantage over CC in terms of broadness or what a model could learn from it. Indeed, given what we know about CC’s general quality and how default preprocessing can screw it up (I see a lot of artifacts in GPT-3′s output I think are due to bad preprocessing), I expect Github to be more useful than an equivalent amount of CC!
The big advantage of Common Crawl over a Github scrape is that, well, CC already exists. Someone has to invest the effort at some point for all datasets, after all. You can go download pre-cleaned versions of it—aside from EleutherAI’s version (which they expect to be substantially better than CC on a byte for byte basis), Facebook and Google recently released big multilingual CC. But of course, now that they’ve done it and added it to the Pile, that’s no longer a problem.
Yes, my hypothesis is that active learning should have a different asymptotic because in a lot of admittedly-simple scenarios like logistic regression, active learning has a much nicer asymptotic. Right now, it’s not too hard to run in <=1 epoch, and GPT-3 did, and that’s using just CC and books1/2/WP. There’s loads of other text datasets. (I think someone in EleutherAI was saying that Literotica alone was 500GB...?) Even if active learning ‘runs out’ of useful data before available compute, that will (a) save a whole lot of compute/time. and (b) tell us explicitly that we’ve ‘used up’ the default data and need to revise our approaches.
The Pile is an interesting experiment, but they’re mostly adding large quantities of single-domain text like Github, which is great for those domains but won’t help outside them.
I disagree. Transfer learning is practically the entire point. ‘Blessings of scale’ etc.
Filtering for difficulty like that is tricky. In particular the most difficult samples are random noise or Chinese or something that the model can’t begin to comprehend.
I would point out that GPT-2 nontrivially, and GPT-3 surprisingly well, understand Chinese. And see my link: GPT-2 is able to filter out garbage really well. It doesn’t have to be perfect. Even a ratio of, say, only 99:1 garbage:good data deleted is a big win. You’re trying to filter out really egregious horrible nonsense data of the sort that you can’t even imagine exists until you’ve actually waded through the sewer of Common Crawl and understood what garbage data really is out there. (Another fun example is: when you go looking for rare languages in Common Crawl, given the base rate, what do even really good natural-language identifier models pull up for rare models? Mostly garbage/natural adversarial examples...)
Amusing to read this on the same day as https://www.wired.com/story/group-house-covid-risk-points/