Questions about doing literature searches
I’d like to get better at doing literature searches. Luke has written a bit about this, but I still have lots of questions.
I’ve mostly been using Google Scholar; it seems OK, but not stellar. What other tools are useful and for what? Added: is there a good way to find papers that cite a many of a set of papers (the idea being that if I find a number of relevant papers, I want to see if there’s a review which covers them)?
How do you look specifically for review papers and/or books? For example, I’m interested in cognitive skill acquisition and I’ve found a review paper from 1996 but I’d like to find recent review papers on the same or closely related topics and haven’t had any luck. Are there specific keywords or phrase permutations that often help? Is searching just the papers that cite an older paper useful? Is searching the same journal useful?
How do you search for criticism? Lets say I’ve found a paper that uses a particular method or theory but I want to know whether there are significant criticisms. Are there phrases that are often associated criticism papers?
Are there any activities that are especially good for practicing literature searches?
Ten years ago, Citeseer had a feature for finding criticism that google scholar still lacks (AFAIK): if you search for papers citing a particular paper, the excerpt it would show would be where the actual citation occurred, so that you could tell if it was positive or negative. Unfortunately, citeseer seems to have been abandoned years before the launch of google scholar and was inferior in every other way upon that launch.
Scopus and Web of Science are the big citation and paper services I’m aware of. Access is probably only reasonable through a nearby university, since this stuff is expensive.
I don’t have access to Scopus or Web of Science, so I mostly just use Google Scholar.
I usually recognize review articles simply by their titles, which are usually shorter and broader than the titles of experimental papers. Or, I find a handbook / companion / edited volume on the field and look for relevant review articles within.
Let’s use cognitive skill acquisition as our example. John located a decent review article from 1996. Since review articles often cite earlier reviews, I checked Google scholar for articles citing that paper from the past 5 years. When scanning through the titles in the results page, I try to guess at which papers are likely to be or to cite recent review articles on the topic.
The first one that looks promising is the third result (when I did the search): Neural and cognitive plasticity: from maps to minds (2008). The first sentence of the abstract mentions learning cognitive skills, so that’s promising. The paper begins:
The first and third citations are recent enough that they might be cited by a recent review article on cognitive skills learning, since such an article might presumably point out that there are individual diferences in the capacity to learn cognitive skills.
Two paragraphs later I read:
So this paper isn’t particularly likely to contain what I want by itself, but I skim the list of headings anyway. About 10 pages down I hit the heading ‘Explanations of Variability in Intellectual Capacity: Past and Present’. Some relevant studies might be cited here, so I skim through that section a bit more slowly. Most of the citations are bout neural plasticity. A few studies on cognitive skills learning in, for example, older adults are cited, but they’re from the mid-90s. I decide I may look at this paper more closely later, but for right now I have more promising leads.
I look up Ackerman & Cianciolo (2000) and Li et al. (2004) on Google scholar.
In the first paper, the authors use not just the term “skill learning” but also “skill acquisition” and “asymptotic performance”, which may be additional useful search terms. The results of experiment one showed mild cognitive skill acquisition with practice, using the Kanfer-Ackerman Air Traffic Controller Task. If that’s a standard test for cognitive skill acquisition, it may be a useful search term. The other two experiments reported in the paper are less directly concerned with skill acquisition.
In the second paper, under the section ‘Experimental Cognitive Tasks’, the authors name several cognitive skills: ‘visual search’, ‘response competition’, long-term and short-term ‘memory search’, and ‘choice reactions’. These may be useful keyword searches later. Other than that, a quick skim of the paper doesn’t suggest much immediate usefulness.
I use Google scholar to check for papers from the past four years that cite Ackerman & Cianciolo (2000). The first result is, again, ‘Neural and cognitive plasticity: From maps to minds’. The second result looks promising: Exercising your brain: A review of human brain plasticity and training-induced learning (2008).
That process took me about 15 minutes. I won’t continue the story at this point, but hopefully this gives some clarity to the processes I use, anyway. We can now check:
The promising review article of ‘training-induced learning’ from 2008.
Other recent papers that cite Ackermann & Cianciolo (2000).
Recent papers that cite Li et al. (2004).
Recent papers that cite ‘Neural and cognitive plasticity: from maps to minds’ (2008).
Recent papers that mention “cognitive skill learning”, “cognitive skills learning”, “cognitive skill acquisition”, “cognitive skills acquisition”, “asymptotic performance”, “Kanfer-Ackerman Air Traffic Controller Task”, “visual search”, “response competition”, “memory search”, or “choice reactions”.
Recent papers that mention “cognitive skill” or “cognitive skills” or “cognitive ability” or “cognitive abilities”, especially when coupled with the search term “learning” or “acquistion”.
Often, you may also be aware of useful leads because of past research. For example my recent posts on rationality skills cite some studies on teachable rationality skills.
Thank you, this is quite useful :)
I’ve found that a searching for ariticles through a major university library is a bit more effective than google scholar. I’m not sure how this varies from school to school, but I go to Iowa State and have had success with their website as well as Washington University in St. Louis (WUSTL) in the past. If you aren’t a student at a major university you typically can still use their search engine though you won’t get access to the articles. However, often if you can at least physically go to a major university library you will have full access while you’re there. This was the case with WUSTL—I was an undergrad at a small school nearby with terrible library resources, so I just went into the city to use WUSTL’s library for any major research projects.
If you’re at a small school you probably have more access than you think through a library loan program—I’ve had a library at a random school somewhere copy an article out of a (relatively obscure) paper journal, send it to Iowa State, who then sent it to me at Chapman University in California where I was for the summer. I was an ISU student at the time, so this isn’t an example of a small school pulling this off, but I’m pretty sure I could have done something similar when in undergrad. The specifics depend on the school you’re at, but it’s worth looking into.
Also, use web of science to do “cited reference searches”—i.e. searching for every article that cited a specific article or author. This can be useful to, e.g., find a newer review article or to see if anyone has addressed the error a paper made. You’ll probably have to go through a large institution to use this without paying a ridiculous sum of money, but as long as you can travel to a big university library (or are a student there), this shouldn’t be a problem.
I don’t follow your first paragraph. Could you point to the search engines at ISU and WUStL?
You can find any university’s library by googling , and somewhere on that page will be their search engine—which searches every service they have a subscription to. For example, Iowa State’s library page has the search box front and center on the main page.
The main point is that you can typically use the search engine as a non-student, but the results will just be the citations instead of the full article. If you want the full articles, you can often just go to the library and use a computer there to get access (true of WUSTL for sure).
As a side note, I think most libraries (at least Iowa State) will tell you if they have the full article in their search results for nonstudents so you can avoid making unnecessary trips.
No, that is not what the principal search box at most university libraries does.
Thanks for the condescending first paragraph, but the reason that I didn’t look myself was that I doubted that I would recognize the search feature when I saw it, because your original description sounded much, much worse than google scholar. Yes, there would be some value in only searching items that you can access, but only searching items that you subscribe to sounds like a terrible trade-off. Once you’ve found the article, checking the subscription status before going to campus is useful.
Assuming you select “articles” instead of “all” or “books” in the dropdown menu, this is how it’s always worked for me—admittedly small sample size.
I wan’t trying to be condescending. My time is scarce, and I didn’t want to waste it doing something you could easily do for yourself (though I did meet you halfway and give one link). Please don’t assume that I’m intentionally being mean, especially when I’m offering help.
In practice, searching through a university service has found me more relevant results quicker than using google scholar, which is the reason I prefer it. Searching through less items is often a feature rather than a bug when less items means less irrelevant items—which is often the issue I have with google scholar’s results (full disclosure, I don’t use scholar often so it may have improved). The key is that the library subscribes to enough services so that you aren’t really limited in any way—which is at least the case at WUSTL and ISU in my experience.
I have access through the university of washington. However I’ve had bad luck with their search functionality. Sometimes it doesn’t even find a paper when I give it a direct title. Perhaps I could use a different school’s search, though.
I’ll have to try web of science, is it better than google scholar’s citation search?
I haven’t used google scholar’s citation search, to be honest. In some random presentation to stat grad students at Iowa State, they told us about web of science and made that feature sound novel so I assumed nothing else had it.
One thing I have found useful is to search on the author of a paper you found especially interesting or useful. I have found a lot of related, but slightly different papers that way. That can help you get a slightly wider view of a subject, but it is even more useful when you are exploring a field and are not yet sure of the most helpful keywords.
I have a copy of Johnson-Laird and Miller’s Language and Perception; searching on Johnson-Laird’s name I found a really interesting short paper from the 1970s about modal connectives in natural language, for an example.