“How are we counting Chinese versus non-Chinese papers? Because often, it seems to be just doing it via, “Is their last name Chinese?” Which seems like it really is going to miscount.” seems unreasonably skeptical. It’s not too much harder to just look up the country of the university/organization that published the paper.
? “Skeptical” implies that this is speculation on Helen’s part, whereas I took her to be asserting as fact that this is the methodology that some studies in this category use, and that this isn’t a secret or anything. This may be clearer in the full transcript:
Julia Galef: So, I’m curious—one thing that people often cite is that China publishes more papers on deep learning than the US does. Deep learning, maybe we explained that already, it’s the dominant paradigm in AI that’s generating a lot of powerful results.
Helen Toner: Mm-hmm.
Julia Galef: So, would you consider that, “number of papers published on deep learning,” would you consider that a meaningful metric?
Helen Toner: I mean, I think it’s meaningful. I don’t think it is the be-all and end-all metric. I think it contains some information. I think the thing I find frustrating about how central that metric has been is that usually it’s mentioned with no sort of accompanying … I don’t know. This is a very Rationally Speaking thing to say, so I’m glad I’m on this podcast and not another one…
But it’s always mentioned without sort of any kind of caveats or any kind of context. For example, how are we counting Chinese versus non-Chinese papers? Because often, it seems to be just doing it via, “Is their last name Chinese,” which seems like it really is going to miscount.
Julia Galef: Oh, wow! There are a bunch of people with Chinese last names working at American AI companies.
Helen Toner: Correct, many of whom are American citizens. So, I think I’ve definitely seen at least some measures that do that wrong, which seems just completely absurd. But then there’s also, if you have a Chinese citizen working in an American university, how should that be counted? Is that a win for the university or is it win for China? It’s very unclear.
And they also, these counts of papers have a hard time sort of saying anything about the quality of the papers involved. You can look at citations, but that’s not a perfect metric. But it’s better, for sure.
And then, lastly, they rarely say anything about the different incentives that Chinese and non-Chinese academics face in publishing. [...]
? “Skeptical” implies that this is speculation on Helen’s part, whereas I took her to be asserting as fact that this is the methodology that some studies in this category use, and that this isn’t a secret or anything. This may be clearer in the full transcript: