RHollerith

Karma: 2,944

Richard Hollerith. 15 miles north of San Francisco. hruvulum@gmail.com

My probability that AI research will end all human life is .92. It went up drastically when Eliezer started going public with his pessimistic assessment in April 2022. Till then my confidence in MIRI (and knowing that MIRI has enough funding to employ many researchers) was keeping my probability down to about .4. (I am glad I found out about Eliezer’s assessment.)

Currently I am willing to meet with almost anyone on the subject of AI extinction risk.

Last updated 26 Sep 2023.

RHollerith 1 Nov 2025 22:01 UTC
7 points
4
on: Shouldn’t taking over the world be easier than recursively self-improving, as an AI?
Yes, even an AI that has not undergone any recursive self-improvement might be a threat to human survival. I remember Eliezer saying this (a few years ago) but please don’t ask me to find where he says it.

RHollerith 23 Oct 2025 18:15 UTC
8 points
0
in reply to: Haiku’s comment on: Which side of the AI safety community are you in?
Parenthetically, I do not yet know of anyone in the “never build ASI” camp and would be interested in reading or listening to such a person.

RHollerith 9 Oct 2025 13:43 UTC
7 points
−4
on: Are We Leaving Literature To The Psychotic?
I’m basically not worried about this.

Google Search has proven pretty OK at preventing spam and content farms from showing up in search results at rates that would make Google Search useless despite the fact the spammers and SEO actors spend billions of dollars per year trying to influence the search results (in ways that are good for the SEO actor or his client, but bad for users of Google Search).

Moreover, even though neither OpenAI, Anthropic nor DeepSeek had access to the expertise, software or data Google was using to filter this bad content from search result, this bad content (spam, content farms and other efforts by SEO actors) has very little influence (as far as I can tell) on the answers given by the current crop of LLM-based services from these companies.

A creator of an LLM is motivated to make the LLM as good as possible at truthseeking (because truthseeking correlates to usefulness to users). If it hasn’t happened already, then in at most a couple of years LLM’s will have become good enough at truthseeking to filter out the kind of spam you are worried about even though the creator of the LLM never directed large quantities of human attention and human skill specifically at the problem like Google has had to do over the last 25 years against the efforts of SEO actors. The labs are also motivated to make the answers provided by LLM services as relevant as possible to the user, which also has the effect of filtering out content produced by the psychotic people.

RHollerith 9 Oct 2025 1:46 UTC
4 points
2
on: You Should Get a Reusable Mask
Also useful for filtering out the smoke from forest fires although it gets tedious to wear one all day for days in a row, so in addition to a mask it is nice to own an air purifier with a HEPA rating or a MERV rating for when you are indoors.

RHollerith 5 Oct 2025 19:31 UTC
9 points
2
in reply to: Cole Wyeth’s comment on: Cole Wyeth’s Shortform
It tends to be bad (or at least costly) to have a rule that has the property that violations of the rule cannot reliably be detected, which leads to the question of how you propose to detect LLM-written content.

RHollerith 4 Oct 2025 17:09 UTC
3 points
0
in reply to: StanislavKrym’s comment on: faul_sname’s Shortform
We’re discussing whether the US could have stopped the Soviet nuclear program in the late 1940s or early 1950s (to see whether that sheds any light on how practical it is to use military power to stop AI “progress”) so what is the relevance of your comment?

But since we’ve started on this tangent, allow me to point out that most of the public discussion about nuclear war (including by The Bulletin of the Atomic Scientists) is wildly wrong because no one had any strong motivation to step into the discussion and correct the misinformation (because no one had a strong motive to advance arguments that there should be a nuclear war) until the last few years, when advocates for AI “progress” starting arguing that AI “progress” should be allowed to continue because an aligned superintelligence is our best chance to avert nuclear war, which in their argument is the real extinction risk—at which time people like me who know that continued AI “progress” is a much more potent extinction risk than nuclear war acquired a strong motive to try to correct misinformation in the public discourse about nuclear war.

RHollerith 4 Oct 2025 16:05 UTC
8 points
1
in reply to: frontier64’s comment on: faul_sname’s Shortform

Clearly destroying a large portion of a country’s government, research scientists, and manufacturing base would halt or destroy all progress on nukes.

It might have completely halted all progress for a year or 2, but what does the US do then?

People think that if a nation is hit by nukes, it becomes impotent. I think it becomes very determined and unified and is likely to become very determined to acquire nukes so it I use them on the country that attacked them. Again, someone who has spent his career thinking about such things (John Mearsheimer) agrees with me: he spoke specifically of what he thinks would have happened if the US had attacked the USSR at the start of the Cold War when the US arsenal consisted of many bombs, but the USSR had no bombs yet (and then he went on to say that no country or coalition of countries can prevent Iran from acquiring nukes if it is determined to get them).

A nuclear attack would have definitely slowed down the Soviet nuclear program, and one can argue that since the US’s program has not been slowed down, then next attack by the US on the USSR would be even more devastating than the first attack, which in turn increases the advantage enjoyed by the US relative to the USSR so that the third attack is even more devastating, and so on, but that leaves out what I consider the controlling consideration: namely, Moscow would have learned from the first attack with the result that the Soviet nuclear program (which again I admit has been set back at least a few years and possibly 15 or 20 years) can no longer be significantly slowed down by nuclear attacks (because it is now more distributed, with many facilities under ground, with more effort spent to keep the locations secret, and a careful analysis done of what industrial resources the program is likely to need so that similar hardening measures can be applied to the supply chain for those resources) which is why I believe the US would have needed to follow up the first attack with an invasion or occupation (at least of Moscow and the ports) which famously has never been successfully done after the Russian empire acquired territory all the way to the Bering Strait, but then Hitler, Napoleon and Charles XII of Sweden didn’t have nukes to help them with their attempted invasions and occupations of Russia.

And yeah, I think once the Soviet program has been hardened in the way I describe above (i.e., after Moscow has learned from the first American attack) then unless the US can obtain location information from spying, then the American nuclear arsenal cannot be used effectively to stop or even significantly slow down the Soviet program (more than it is already slowed down by the need to keep the program distributed and secret from prying eyes) unless nuclear attacks can destroy most large buildings in the country, which my “mathematics” shows would have been quite impossible.

Apparently Tel Aviv was able this year to get a lot of location information about the Iranian nuclear program (and Iranian missile launchers and air defense facilities) through spying, so it is possible that Washington would have been able to do the same in the USSR. I doubt it, but I hereby mark this part of my arguments as less certain than the other parts.

RHollerith 4 Oct 2025 16:04 UTC
3 points
0
in reply to: frontier64’s comment on: faul_sname’s Shortform
I mistakenly pasted in 2 copies (then I modified copy 2). Corrected now.

RHollerith 4 Oct 2025 15:37 UTC
3 points
−1
in reply to: faul_sname’s comment on: faul_sname’s Shortform
Even if Washington had zero compunctions against using nukes (including against cities), it would not have been able to keep Moscow or Beijing from obtaining nukes for long. John Mearsheimer has asserted this explicitly (during a discussion on Iran’s nuclear program, but please don’t ask me to find the web page where I heard it).

Even when the strategic arsenals of the US and the USSR were at their height (in the early 1980s IIRC), there was not enough nukes to completely destroy even all above-ground buildings in a country as large in area as the US or the USSR, let alone buried structures: specifically, even a large 1-megaton nuke can destroy heavily-reinforced above-ground concrete buildings only within a 2-mile radius, and if a person tries to cover the entire area of the USSR with circles that size, he will find that there have never existed enough nukes in the world to cover the entire area. IIRC you cannot even cover it with circles of a radius of 5 miles, inside which it is not possible to destroy even 90% of unreinforced non-wooden structures even with the largest nuke in the US inventory. (A 10-megaton nuke can destroy an area only slightly larger than a 1-megaton nuke, which is why after an initial period of enthusiasm, both the US and the USSR stopped making nukes larger than about 1 megaton, focusing instead on putting multiple nukes on one ICBM.) Note that we haven’t even started to analyze how many nukes it would take to destroy buried structures in the USSR when you don’t know where in the country those buried structures are, and I’ve seen credible reports from about 15 years ago stating that Moscow has a facility built into a mountain of quartz in the southern Urals that Moscow believes can withstand a determined nuclear attack even if the US knows exactly where it is.

The people of most countries will become very determined to fight back after the country is invaded and occupied, which is why much weaker powers like Afghanistan and Vietnam tend to prevail after being invaded and occupied even by great powers. We can expect the same determination after a nuclear attack—and yes, there would have been enough survivors in the USSR to continue the fight. Analysis by the US government in the early 1980s (again when IIRC nuclear stockpiles were at their greatest number) estimated that a full nuclear attack on the USSR would kill only about 55% of the population even if the USSR had no warning of the attack. The number for a full attack on the US was a little lower (50%) because the population of the US is more spread out as opposed to concentrated in cities.

Yes, the people most useful to advancing a Soviet nuclear program would preferentially have been in the 55% that die (especially if Washington attacks mid-week, when fewer of the Soviet upper class would be at their dachas) but Moscow could have used the surviving nuclear scientists to teach new nuclear scientists (and this time required them to live somewhere other the the probable targets of the next US nuclear attack).

The page you link to is silent on whether Von Neumann believed the US would have been able to keep the USSR from obtaining nukes indefinitely or whether the attack he proposed was intended merely to slow their nuclear program down. If the former, I bet no one in the Pentagon took his proposal seriously for more than a few days: the Pentagon would know that to have a realistic chance of keeping the USSR from obtaining nukes indefinitely, the US and its allies would have needed to permanently occupy Moscow and all of the USSR’s ports (after nuking the USSR of course) the success of which would have been in severe doubt, and even if successful would probably have caused the deaths of many millions of men on the US side.

People are arguing that AI “progress” should be allowed to continue because nuclear war is a bigger threat to continued human survival than AI “progress” is, which is wrong, which is why I am reacting against this very widespread notion that nuclear weapons are more destructive than they actually are.

RHollerith 29 Sep 2025 15:41 UTC
2 points
0
on: Could China Unilaterally Cause an AI Pause?
Could China Unilaterally Cause an AI Pause?

Definitely not (barring some very drastic change in the global situation). The most powerful country in the world couldn’t prevent the Taliban from returning to power in Afghanistan even though they spent many trillions of dollars trying—and Afghanistan’s economy is 6% of 1% of the size of the US economy. Well-funded actors like Moscow and Beijing have some small measure of influence on the US through propaganda (such as Beijing’s efforts to make China appear more powerful than it actually is) and through trying to intensify internal divisions (e.g., by funding extremist groups in the US), but neither can make the US make any important change with anything approaching reliability.

RHollerith 20 Sep 2025 23:03 UTC
2 points
0
on: Looking for a ray of hope in IABIED
As far as I can tell, Eliezer and Nate are relying on no results of experiments done on AI models (REAIMs) to conclude that superintelligence is dangerous, and if some clever young person (or more realistically a series of clever young people each building on the work of their predecessors) comes up with a good plan for creating an aligned superintelligence (which probably won’t happen any year soon) that plan probably also will rely on no REAIMs nor will Eliezer and Nate require any REAIMs to conclude that the plan is safe.

Experiments and tests are very useful; human engineers and designers facing sufficiently difficult challenges will usually choose to use them; but sufficiently capable people are far from helpless even in domains in which they cannot do experiments (because the experiments would be too risky or because they would rely an GPUs and data centers that have been banned by international agreement).

For more information, a person could do worse than read Eliezer’s Einstein’s arrogance.

RHollerith 19 Sep 2025 13:32 UTC
6 points
0
in reply to: Ben Pace’s comment on: Obligated to Respond
Before Christianity was discredited, it acted as a sort of shared lens through which the value of any proposed course of action could be evaluated. (I’m limiting my universe of discourse to Western society here.) I’m tempted to call such a lens an “ideological commitment” (where the “commitment” is a commitment to view everything that happens to you through the lens of the ideology—or at least a habit of doing so).

Committing to an ideology is one of the most powerful things a person can do to free himself from anxiety (because the commitment shifts his focus from his impotent vulnerable self to something much less vulnerable and much longer lived). Also, people who share a commitment to the same ideology tend to work together effectively: a small fraction of the employees of an organization for example who share a commitment to the same ideology have many times taken the organization over by using loyalty to the ideology to decide who to hire and who to promote. They’ve also taken over whole countries in a few cases.

The trouble with reducing the prestige and the influence of Christianity even now in 2025 is that the ideologies that have rushed in to fill the void (in the availability of ways to reduce personal anxiety and of ways to coordinate large groups of people) have had IMHO much worse effects than Christianity.

You, Ben, tend to think that society should “eat the costs and spend the decades/centuries required to build better things” than Christianity. The huge problem with that is that the extreme deadliness of one of ideologies that has rushed in to fill the void caused by the discrediting of Christianity: namely, the one (usually referred to vaguely by “progress” or “innovation”) that views every personal, organizational and political decision through the lens of which decision best advances or accelerates science and technology.

In trying to get frontier AI research stopped or paused for a few decades, we are facing off against not only trillions of dollars in economic / profit incentives, but also an ideology, and ideologies (including older ideologies like Christianity) have proven to be fierce opponents in the past.

Reducing the prestige and influence of Christianity will tend to increase the prestige and influence of all the other ideologies, including the ideology, which is already much more popular than I would prefer, that we can expect to offer up determined sustained opposition to anyone trying to stop or long-pause AI.

RHollerith 19 Sep 2025 1:30 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform

the importance of drones, I think, is not going to go down thanks to AI.

I agree. What I tried to say though was that my guess is that for drones to stay as effective as they currently are for 5 years would require AI capable enough that it would transform so many aspects of society that for us in 2025 to try to project out that far becomes futile.

RHollerith 17 Sep 2025 17:45 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
When you imagine a force overwhelming a position using very many drones, are you imagining one human per drone or are you imagining most of the drones (or more precisely, most of the drone flying time in “drone hours”) being flown by AI?

I would imagine that by the time a drone doesn’t need a human operator for most of the time it is in combat, lots of other things about war will have changed, e.g., whether an infantry soldier is obsolete.

RHollerith 17 Sep 2025 12:42 UTC
4 points
2
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I agree that multirotor helicopter drones have fundamentally transformed the war in Ukraine.

I am willing to believe 70-90% Russian casualties as being caused by these weapons, but the fraction of Ukrainian casualties will be significantly lower even though Russia has been innovating furiously with drones because Russia is less constrained in supply of artillery shells. “While estimates vary, a common figure cited is a 5-to-1 or even higher ratio of shells fired by Russian forces compared to Ukrainian forces in some areas” (Google Gemini which has a knowledge cutoff date of Jan 2025).

Relevant to your original question (many levels of indentation ago) particularly the part about Israel’s vulnerability, is still think the crux is that countermeasures will probably reduce very significantly the effectiveness of this class of drones over the next 3 years at least in areas protected by well-funded militaries that are not in a state of civil war. The tech does not seem to have as much potential to stay very important as for example the fuzed artillery shell, which has remained very important for over 100 years because it is relatively difficult to develop countermeasures for it.

Aside from the sound issue already discussed, weapons makers will probably be unable to make the class of weapon we are discussing much faster than they already are (namely 50 to 120 km per hour) and if they do manage to increase the speed significantly, that will probably make the sound problem worse. In contrast, according to Gemini, during the terminal portion of its trajectory, a large artillery shell travels at 1080 to 2160 kilometers per hour.

RHollerith 17 Sep 2025 0:17 UTC
3 points
1
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform

You might think that, because LLMs are grown without much understanding and trained only to predict human text, they cannot do anything except regurgitate human utterances. But that would be incorrect. [...]

Furthermore, AIs nowadays are not trained only to predict human-generated text. An AI-grower might give their AI sixteen tries at solving a math problem, thinking aloud in words about how to solve it; then, the “chain-of-thought” for whichever of the sixteen tries went best would get further reinforced by gradient descent, yielding what’s called a reasoning model. That’s a sort of training that can push AIs to think thoughts no human could think.

How does that conclusion follow? If a base model can only regurgitate human utterances, how is generating sixteen utterances and then reinforcing some of them leads to it… not regurgitating human utterances?

In the first sentence, Eliezer and Nate are (explicitly) stating that LLMs can say things that are not just regurgitations of human utterances.

RHollerith 16 Sep 2025 14:28 UTC
4 points
2
in reply to: Thomas Kwa’s comment on: Daniel Kokotajlo’s Shortform
Consider though that if a defense contractor were able to reduce the noise of a helicopter enough to matter militarily, the Pentagon would have poured many billions into that contractor, especially during the Vietnam War during which helicopters were relied on very extensively. Also, the main constraint on the use of civilian helicopters is probably complaints about the noise. And the fans at the front of engines of airliners is responsible for producing most of the thrust on the airliner, and there have been large economic incentives to make those quiet (to eliminate the copious restrictions on airliners designed to limit noise) and although airliners have gotten quieter, they remain quite loud, loud enough to detect and triangulate with arrays of microphones many many miles away. (The reason the are called “fans” and not “propellers” is merely the number of blades.)

Remotely-piloted gliders or glide bombs of course don’t have much of a noise signature, which is why I have tried to be careful in my comments to restrict the scope of my statements to multirotor helicopter-style drones.

RHollerith 16 Sep 2025 14:16 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
The host definitely says that the guest (the soldier) was a drone operator or worked on a team the purpose of which is to operate drones during the first 3 minutes: I re-listened to that much before I wrote my description. The word “drone” was definitely used.

RHollerith 16 Sep 2025 0:53 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
This soldier spent 2 years fighting for Ukraine, including 6 months recently as an operator of FPV drones, and he is also skeptical that drones will revolutionize military affairs during the next few years. I don’t recall anything about his arguments, but my recollection is he does provide some argumentation in this interview.

RHollerith 15 Sep 2025 23:49 UTC
7 points
0
in reply to: Arjun Pitchanathan’s comment on: Daniel Kokotajlo’s Shortform
I want to revise my statement that “I listen to defense experts talk as a weird form of relaxation.” Actually what I listen to are geopolitics experts, who often have hours-long conversations specifically about military matters. Here are some suggestions:

https://www.youtube.com/@DecodingGeopoliticsPodcast

https://www.youtube.com/@GeopoliticalFuturesGPF

https://www.youtube.com/@DAlperovitch

the interviews with John Mearsheimer on https://www.youtube.com/@DanielDavisDeepDive