I think one of the biggest reasons I am worried about AI doom is that there doesn’t seem to be very good counter-arguments. Most of them are just bad, and the ones that aren’t a bit vague (usually something along the lines of “humanity will figure it out” or “maybe LLMs will scale in nice ways”).
However, I’m curious as to how accurate this heuristic is. My question: What examples in the past are there of “argument is widespreadly seen as ridiculous and intuitively false, but the argument is pretty solid and the counter-arguments aren’t”. (Sorry if that’s a bit vague, use your best judgement. I’m looking specifically for examples that are similar to the AI x-risk debate.) And did they turn out true or false? Try to include reasons why the argument was so strong, a typical counter-argument, and the strongest counter-argument.
Please use spoiler text for the final answer so that I can try to predict it before seeing it!
The number of people who actually had the deep technical skills and knowledge to evaluate the risk of ignition of the atmosphere from nuclear weapons was very small, and near completely overlapped with the people developing the very same weapons of mass destruction that were the source of that risk.
The number of people who have the deep technical knowledge, skills, and talent necessary to correctly evaluate the actual risk of AGI doom is probably small, and probably overlaps necessarily with the people most capable of creating it.
I’m curious how this fits into the context. Regardless of whether or not one believes it’s true, doesn’t it seem reasonable and intuitively right—so the opposite of what is asked for?
I think the argument that would have been seen as ridiculous by most in the nuclear weapons example is, “The right arrangement of (essentially, what look like) rocks and metals will not only make a big explosion, but could destroy all life on earth in seconds.” The argument in favor (and eventual, correct argument against) were both highly technical and inaccessible. Also the people most involved in the deep technical weeds were both the ones capable of seeing the danger, and the ones needed to figure out if the danger was real or not.
So it would be: Claim: A nuclear bomb could set the atmosphere on fire and destroy everything on earth Argument: Someone did a calculation. Counterargument: Clearly, that’s absurd. Good Counterargument: Someone else did another calculation.
And I guess the analogy to AI applies foom/room a the bottom, where one can actually do calculations to at least in principle estimate some OOMs.
Historical examples of things that once sounded ridiculous but turned out to be true:
Spherical Earth (vs Flat Earth)
Heliocentrism (vs Geocentrism)
Germ theory (vs e.g. Miasmatic theory)
Evolution via natural selection
Quantum mechanics
Relativity
Plate tectonics
Black holes
It’s harder to know what qualifies as false examples since they do (now) have good counterarguments, but maybe something like these:
Phlogiston theory
Vitalism
Luminiferous aether
Lamarckian evolution
Cold fusion
Steady State cosmology (vs Big Bang)
Caloric theory
Spontaneous generation
Examples of ideas with less certain status:
String theory / quantum gravity / unified physics
Multiverse hypothesis / simulation hypothesis
Existence and nature of extraterrestrial life
Nature of dark matter & dark energy
Epigenetic roles in disease and inheritance
Origins of life / abiogenesis / panspermia
Nature of consciousness and reality
Does spherical earth count? I couldn’t find any sources saying the idea was seen as ridiculous, especially around the time that they actually discovered it was round via physical measurements.
There is indeed the Myth of the flat Earth that is a misconception about the beliefs of scholars in the Middle Ages and some scholars certainly understood the concept of a spherical Earth since at least Eratosthenes. I’m referring to earlier history like ancient Egyptian, Chinese, and pre-Socratic Greek cosmologies. Admittedly, it’s not a great example since most of the debates about it are lost to history, and such debates wouldn’t involve the same kind of reasoning and evidential standards we use today.
What’s wrong with caloric theory?
It helps to look up what the term means: “The caloric theory is an obsolete scientific theory that heat consists of a self-repellent fluid called caloric that flows from hotter bodies to colder bodies. Caloric was also thought of as a weightless gas that could pass in and out of pores in solids and liquids.”
Thanks, appreciate the polite explanation!
Maybe the risk of nuclear war during the Cold War? This 1961 argument by Bertrand Russell is cogent enough to sound correct, it would have probably sounded like a pretty wild prediction to most people, and all considered, it was indeed kinda false, though modest steps along the lines of option 3 did happen (we didn’t get a literal political union but we did get the fall of the Iron Curtain and globalization entangling the world into mutual economic interest, as well as the UN). Anyone betting on the status quo fallacy on a gut instinct against Russell there would have won the bet.
Yeah that definitely seems very analogous to the current AI x-risk discourse! I especially like the part where he says the UN won’t work:
Do you know what the counter-arguments were like? I couldn’t even find any.
I would agree that the UN is a sham. I don’t see why you count Russell as being wrong on this point.
Not really, though I expect there must have been some objections. I think digging in the 1950s-60s era and the way they talked about nuclear risk would probably be very instructive. The talk “The AI Dilemma” (look it up on YouTube if you haven’t seen it already) even brings up specifically the airing of the TV movie “The Day After”, and how it concluded with a panel discussion between experts on nuclear risk, including among others Henry Kissinger and Carl Sagan. The huge amount of rightful holy terror of nuclear war back in that day most likely worked, and led to enough people being scared enough of it that in the end we avoided it. Worryingly, it’s now, far from the peak of that scare, that we start seeing “but maybe nuclear war wouldn’t be that destructive” takes (many appeared around the beginning of the war in Ukraine).
Oh, but one weakness is that this example has anthropic shadow. It would be stronger if there was an example where “has a similar argument structure to AI x-risk, but does not involve x-risk”.
So like a strong negative example would be something where we survive if the argument is correct, but the argument turns out false anyways.
That being said, this example is still pretty good. In a world where strong arguments are never wrong, we don’t observe Russell’s argument at all.
Disagree about anthropic shadow, because the argument includes two possible roads to life, barbarism or a world government. If the argument was correct, conditioned upon being still alive and assuming that barbarism would lead to the argument to be lost, an observer still reading Russell’s original text should see a world government with 100% probability. And we don’t.
Oh right, good point. I still think anthropic shadow introduces some bias, but not quite as bad since there was the world government out.
The idea of a bias only holds if e.g. what Russell considered 100% of all possibilities only actually constituted 80% of the possibilities: then you might say that if we could sample all branches in which an observer looks back at argument, they’d see the argument right with less-than-80% probability, because in a part of the branches in which either of those three options come to pass there are no observers.
But while that is correct, the argument is that those are the only three options. Defined as such, a single counterexample is enough to declare the argument false. No one here is denying that extinction or civilisation collapse from nuclear war have been very real possibilities. But the road we care about here—the possible paths to survival—turned out to be wider than Russell imagined.
Yeah, Russell’s argument is ruled out by the evidence, yes.
I’m considering the view of a reader of Russell’s argument. If a reader thought “there is a 80% chance that Russell’s argument is correct”, how good of a belief would that be?
Because IRL, Yudkowsky assigns a nearly 100% chance to his doom theory, and I need to come up with the x such that I should believe “Yudkowsky’s doom argument has a x% chance of being correct”.
That Malthusianism is wrong (for predicting the future). Prior to the demographic transition, arguments in favor of this view could be basically summarized as “somehow it will be fine”.
Doom skeptics have the daunting task of trying to prove a negative. It is very hard to conclusively prove anything is safe in a generalized, unconditional way.
As to arguments that were seen as intuitively false yet were solid, there is a huge class of human rights topics: racial and gender equality, gay rights, even going back to giving peasants the vote and democratic government. All of these were intuitively ridiculous, yet it’s hard to make credible counter-arguments.
There were some counter-arguments against democracy that seemed pretty good. Even the founding fathers were deeply worried about them. They aren’t seen as credible today, but back then they were viewed as quite strong.
Agreed, but you can the same about interracial marriage or allowing women to vote: within the frameworks and assumptions people had then, there were strong arguments that made the ideas ridiculous and obviously wrong on the face of it.
Biological evolution is the origin of species.
Simple mathematical laws govern the behavior of everything, everywhere, all the time.
Negative, imaginary, irrational, and transcendental numbers, as well as infinite sets and infinitesimals, are logically coherent and useful.
(Many of the above are things many people still don’t believe).
The Roman Empire is large, powerful, and well organized, and will have no problem dealing with the new Christian religion.
Did you notice:? You can take these statements and make the case it’s stupid, for one reasonable set of priors, or profound, for a different but somewhat possible set of priors. And that’s true for each negation. As well as for several forms of negation. Bohr called that deep truth, or something like that.