Randomness in Science

rogersbacon7 Dec 2021 18:17 UTC
19 points
15 comments9 min readLW link

I.

Humans have a randomness problem. We are bad at generating randomness as individuals (try to out-random a computer in Man vs. Machine) and in the aggregate—ask a group of people to choose a “random” number between 1-20 and the most common number will be 17. We are also bad at detecting randomness, that is we find patterns where there is none, known as patternicity—also see apophenia, pareidolia (no, the burn pattern in your toast that looks like Jesus is not a sign from the Holy Spirit), the clustering illusion, and the hot hand fallacy.

In addition to being “bad” at randomness, we also have an aversion to it. It’s not hard to see why from an evolutionary perspective – randomness is the antithesis of life’s imperative to minimize risk by controlling and predicting the environment. However, there are situations in which the best strategy is to utilize an element of randomness. In the modern world, we can precisely calculate when a particular decision might benefit from randomness and use computers to generate it (or at least pseudo-generate it), but of course our ancestors did not have this luxury. Nevertheless, cultures around the world have evolved certain practices that enabled them to harness the power of randomness. From The Secret of Our Success (2015):

“When hunting caribou, Naskapi foragers in Labrador, Canada, had to decide where to go. Common sense might lead one to go where one had success before or to where friends or neighbors recently spotted caribou.

However, this situation is like the Matching Pennies game. The caribou are mismatchers and the hunters are matchers. That is, hunters want to match the locations of caribou while caribou want to mismatch the hunters, to avoid being shot and eaten. If a hunter shows any bias to return to previous spots, where he or others have seen caribou, then the caribou can benefit (survive better) by avoiding those locations (where they have previously seen humans). Thus, the best hunting strategy requires randomizing.

Can cultural evolution compensate for our cognitive inadequacies? Traditionally, Naskapi hunters decided where to go to hunt using divination and believed that the shoulder bones of caribou could point the way to success. To start the ritual, the shoulder blade was heated over hot coals in a way that caused patterns of cracks and burnt spots to form. This patterning was then read as a kind of map, which was held in a pre-specified orientation. The cracking patterns were (probably) essentially random from the point of view of hunting locations, since the outcomes depended on myriad details about the bone, fire, ambient temperature, and heating process. Thus, these divination rituals may have provided a crude randomizing device that helped hunters avoid their own decision-making biases.”

As this passage demonstrates, there are some situations in which randomness can serve to compensate for biases in our decision making. Divination practices can provide a kind of metaphysical cover story for injecting chance into a strategic decision (e.g. the gods speak to us through the shoulder bones), yet it would seem that we have no such disguise in the modern secular world, and that any suggestion to use a randomness-based strategy is at an inherent disadvantage (i.e. randomness represents a “blind spot” in contemporary culture). Given that science consists of various activities in which the aim is to minimize randomness and unpredictability, we wonder if it is especially difficult to overcome randomness aversion when it comes to the organization and practice of science. This begs the question—could we improve science by exploring new ways to inject randomness into the research process?

II.

One application of randomness is the use of lotteries for grant funding. Numerous agencies are already experimenting with random allocation of funds (see ”Science funders gamble on lotteries”); we won’t recapitulate the arguments for funding lotteries (in brief, peer review of grants is biased and unreliable, costly in terms of time and effort for both reviewers and applicants) as these have already been discussed extensively in the literature (see “Mavericks and Lotteries” (Avin, 2019) for a recent comprehensive review), but suffice it to say that there is good reason to think that the scientific community would benefit from further experimentation with models of random funding allocation.

Scientific publishing may also benefit from journals that use randomness as a part of their peer review procedures as current standard practices suffer from many of the same issues described above for grant applications. A 2009 study that modeled scientific practice and different publication selection strategies for journals found that, “Surprisingly, it appears that the best selection method for journals is to publish relatively few papers and to select those papers it publishes at random from the available “above threshold” papers it receives″ (Zollman, 2009). Such a threshold + random selection model could help overcome many of the bad incentives and norms inherent in academic “publish or perish” culture (see “The Natural Selection of Bad Science” (2016) for discussion on how the current publishing landscape harms the quality of research). Randomness can also be used by authors as a tool for protesting unjust practices and conventions in science, particularly those surrounding the assignment of credit for research activities. Penders and Shaw (2020) discuss the nature of civil disobedience in science and highlight various examples of deviant author assignment strategies that involve randomness (e.g. flipping a coin, brownie bake-off, free throw shooting contest, authorial order by height, utilizing random fluctuation in the euro/​dollar exchange rate).

The scientific job market may suffer from many of the same issues that we see in funding and publishing—excessive competition that incentivizes individual scientists towards activities and practices that may ultimately be harmful for science as a whole (see “Competition Science: Is competition ruining science?”). Again, one solution may be to incorporate randomness into hiring decisions for scientific positions in academia and industry (e.g. a lottery amongst candidates that meet a certain threshold). To our knowledge, there are no examples of universities or private industry using such a method for hiring or promotion, however there are modeling studies which suggest that random promotion can outperform merit-based promotion in some situations (Phelan et al., 2001; Pluchino et al., 2011; Pluchino et al., 2011; Pluchino et al. 2018).

The history of science is filled with serendipitous events that lead to significant breakthroughs (penicillin, radioactivity, and the cosmic microwave background being some of the more famous examples). This suggests serendipity itself as a topic for further research; perhaps we can move beyond the anecdotal level and gain a more systematic understanding of how chance events drive discovery, thereby enabling a form of “serendipity engineering” in the sciences. Social scientist Ohid Yaqub has launched such a research project (see “Serendipity: Towards a taxonomy and a theory”, 2018).

“Starting in the archive of US sociologist Robert K. Merton, Yaqub gathered hundreds of historical examples. After studying these, he says, he has pinned down some of the mechanisms by which serendipity comes about. These include astute observation, errors and “controlled sloppiness” (which lets unexpected events occur while still allowing their source to be traced). He also identifies how the collaborative action of networks of people can generate serendipitous findings.”
(“The Serendipity Test”)

III.

Why does chance seem to play such a role in many significant scientific discoveries? In the previous sections, we suggested that randomness can be used to overcome biases and flaws in judgement, however the exact nature of these biases were not fully spelled out. While randomness can be used to compensate for the personal biases (e.g. a bias towards one’s own topic of study or a racial bias) of any individual decision maker (a grant reviewer, journal editor, head of department), the reason that “happy accidents” are involved in so many scientific discoveries is that randomness serves as a corrective for a more global conservative bias inherent in the structures of organized science. Numerous empirical and simulation studies evidence this conservative bias; below we provide quotes from two such studies in order to better elucidate the nature of this bias.

“By analyzing millions of biomedical articles published over 30 years, we find that biomedical scientists pursue conservative research strategies exploring the local neighborhood of central, important molecules. Although such strategies probably serve scientific careers, we show that they slow scientific advance, especially in mature fields, where more risk and less redundant experimentation would accelerate discovery of the network.”
— Rzhetsky et al. (2015)

Those who comment on modern scientific institutions are often quick to praise institutional structures that leave scientists to their own devices. These comments reveal an underlying presumption that scientists do best when left alone—when they operate in what we call the ‘scientific state of nature’. Through computer simulation, we challenge this presumption by illustrating an inefficiency that arises in the scientific state of nature. This inefficiency suggests that one cannot simply presume that science is most efficient when institutional control is absent. In some situations, actively encouraging unpopular, risky science would improve scientific outcomes.
— Kummerfeld and Zollman (2016)

In addition, metascientific research demonstrates that the most novel and impactful research often results from “unusual individual scientist backgrounds, atypical collaborations, or unexpected expeditions where scientists and inventors reach across disciplines and address problems framed by a distant audience” (Shi and Evans, 2020; also see Uzzi et al. 2013, and Lin et al., 2021). Overall, this paints the picture of a scientific community in need of more novelty and greater risk taking. One way to achieve this goal is to modify the incentives and norms of modern science, however systemic change of this kind is often difficult and only attainable in the long term. Alternatively, we may seek to increase randomness in all its forms as this will increase the generation of the unusual and atypical, thereby reducing redundancy and shifting the scientific community towards riskier research strategies.

At a collective level, enhancing randomness means (amongst other things) a greater number of chance encounters between scientists. In a post-COVID-19 world where remote work and virtual conferences become more common, we should be concerned that serendipitous meetings between researchers will become fewer and farther between. To some degree, we may be able to compensate for this reduction in collective randomness, by increasing the role of chance in the lives of individual scientists. One method for doing so would be increased interest in dreams. A recent hypothesis (the overfitted brain hypothesis), suggests that dreams are essentially random combinations of our daily experience:

“Research on artificial neural networks has shown that during learning, such networks face a ubiquitous problem: that of overfitting to a particular dataset, which leads to failures in generalization and therefore performance on novel datasets. Notably, the techniques that researchers employ to rescue overfitted artificial neural networks generally involve sampling from an out-of-distribution or randomized dataset. The overfitted brain hypothesis is that the brains of organisms similarly face the challenge of fitting too well to their daily distribution of stimuli, causing overfitting and poor generalization. By hallucinating out-of-distribution sensory stimulation every night, the brain is able to rescue the generalizability of its perceptual and cognitive abilities and increase task performance.” (Hoel, 2021)

Anecdotal evidence for the effectiveness of dreams as a tool for scientific creativity comes from the well-known examples of discoveries made in dreams, such as the benzene ring (Kekulé), The structure of the atom (Bohr), and the periodic table of elements (Mendeleev). In order to improve memory of dreams and thereby reap the benefits of increased randomness in their lives, we recommend that scientists begin a dream journaling practice. Lastly, we recommend (in all sincerity) that scientists adopt divination practices such as the burning of caribou shoulder blades, the reading of entrails, bird augury, or the I Ching. For a full list of possibilities, see the “Methods of Divination” Wikipedia page.

Postscript:

Between 1997 and 2012, Jussieu’s campus in Paris’s Left Bank (Paris Jussieu — the largest medical research complex in France) reshuffled its labs’ locations five times due to ongoing asbestos removal, giving the faculty no control and little warning of where they would end up. An MIT professor named Christian Catalini later catalogued the 55,000 scientific papers they published during this time and mapped the authors’ locations across more than a hundred labs. Instead of having their life’s work disrupted, Jussieu’s researchers were three to five times more likely to collaborate with their new odd-couple neighbors than their old colleagues, did so nearly four to six times more often, and produced better work because of it (as measured by citations)…Even an institution like Paris Jussieu, which presumably places a premium on collaboration across disciplines, couldn’t do better than scattering its labs at random. (“Engineering Serendipity”)

This essay was originally posted at Secretum Secretorum and is also one of the example articles for Seeds of Science (PDF found here). Seeds of Science is a new OA and fee-free scientific journal that publishes speculative or non-traditional articles with peer review by our community of “gardeners”. If you like weird science writing like this then please consider joining us as a gardener or becoming an author (See “How to Publish”). As a gardener, we will email you submitted manuscripts and you can vote/​comment at your leisure. It is free to join and participation is entirely at will — for more information and sign up instructions visit the gardeners page on our website.

Works Cited

Avin, S. (2019). Mavericks and lotteries. Studies in History and Philosophy of Science Part A, 76, 13-23.

Fang, F. C., & Casadevall, A. (2015). Competitive science: is competition ruining science?.

Henrich, Joseph. The Secret of our Success. Princeton University Press, 2015.

Hoel, E. (2021). The overfitted brain: Dreams evolved to assist generalization. Patterns, 2(5), 100244.

Kummerfeld, E., & Zollman, K. J. (2015). Conservatism and the scientific state of nature. The British Journal for the Philosophy of Science, 67(4), 1057-1076.

Lin, Y., Evans, J. A., & Wu, L. (2021). Novelty, Disruption, and the Evolution of Scientific Impact. arXiv preprint arXiv:2103.03398.

Penders, B., & Shaw, D. M. (2020). Civil disobedience in scientific authorship: Resistance and insubordination in science. Accountability in research, 27(6), 347-371.

Phelan, S. E., & Lin, Z. (2001). Promotion systems and organizational performance: A contingency model. Computational & Mathematical Organization Theory, 7(3), 207-232.

Pluchino, A., Biondo, A. E., & Rapisarda, A. (2018). Talent versus luck: The role of randomness in success and failure. Advances in Complex systems, 21(03n04), 1850014.

Pluchino, A., Garofalo, C., Rapisarda, A., Spagano, S., & Caserta, M. (2011). Accidental politicians: How randomly selected legislators can improve parliament efficiency. Physica A: Statistical Mechanics and Its Applications, 390(21-22), 3944-3954.

Pluchino, A., Rapisarda, A., & Garofalo, C. (2011). Efficient promotion strategies in hierarchical organizations. Physica A: Statistical Mechanics and its Applications, 390(20), 3496-3511.

Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015). Choosing experiments to accelerate collective discovery. Proceedings of the National Academy of Sciences, 112(47), 14569-14574.

Shi, F., & Evans, J. (2020). Science and technology advance through surprise. arXiv preprint arXiv:1910.09370.

Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468-472.

​​Yaqub, O. (2018). Serendipity: Towards a taxonomy and a theory. Research Policy, 47(1), 169-179.

Zollman, K. J. (2009). Optimal publishing strategies. Episteme, 6(2), 185-199.

rogersbacon7 Dec 2021 18:17 UTC
19 points
15 comments9 min readLW link