RHollerith

Karma: 3,083

Richard Hollerith. 15 miles north of San Francisco. hruvulum@gmail.com

My probability that AI research will end all human life is .92. It went up drastically when Eliezer started going public with his pessimistic assessment in April 2022. Till then my confidence in MIRI (and knowing that MIRI has enough funding to employ many researchers) was keeping my probability down to about .4. (I am glad I found out about Eliezer’s assessment.)

Currently I am willing to meet with almost anyone on the subject of AI extinction risk.

Last updated 26 Sep 2023.

RHollerith 24 Jun 2026 0:43 UTC
2 points
0
in reply to: Jiro’s comment on: On TEEs for Privacy-Preserving Monitoring in AI Governance
Which AI assistant service do you currently use? Do you tell it personal things about yourself? I certainly tell the service I use all sorts of personal things about myself. (More precisely, anyone with access to my chat history can deduce all sorts of personal details about me.)

If you use Claude, Gemini, GPT or Grok, then you are sharing this info with a mega-corporation, and your only protection against misuse is management’s needing to worry about whistle blowers. If instead you use what this post has been calling a verifiably confidential AI service, then you have significantly stronger protection. Such services are available now.

RHollerith 23 Jun 2026 16:51 UTC
2 points
0
on: On TEEs for Privacy-Preserving Monitoring in AI Governance
Apple has a mature system or “regime” for ensuring that only apps it has explicitly approved will run on iOS. Google and Microsoft have similar systems for Android and Windows, respectively. In most of my daydreaming over the years about how trusted computing might help us survive the AI juggernaut, there will come a time not too far into the future after which every new cutting-edge GPU incorporates a similar regime.

(Technical detail: rather than a GPU, the thing being controlled by this regime might need to be a GPU-CPU pair or a system composed a CPU attached to zero-or-more GPUs.)

Such a regime would make it unnecessary to ensure that all computational resources are accounted for, which you spend a lot of ink on in this post. I take it that you consider such a regime to be either morally abhorrent or politically infeasible.

Every day providers of AI-based services become more efficient at delivering their services, in the process of which the software and the hardware behind those services grows more complex, increasing the already-quite-large cost of modifying that software and hardware to meet AI-safety goals (as you mention more than once in this post). It would’ve been so convenient for the cause of AI safety and humanity’s survival if at the start (i.e., in 2022 or so) of this process of becoming more efficient, it were already routine for some centralized authority to dictate what software can run on all servers (e.g., to combat exploitation of children, to discourage content offensive to Allah or to prevent criticism of the government). (Of course, merely pointing this out tends to paint me as some sort of political extremist at least in the US.)

RHollerith 23 Jun 2026 16:43 UTC
2 points
0
on: On TEEs for Privacy-Preserving Monitoring in AI Governance
I’ve read the post (skipping the images). It is quite good, and I’ve detected nothing I would consider a mistake or a falsehood.

I personally assign less difficulty to auditing computing hardware and greater difficulty to auditing the Linux kernel and related software (for vulnerabilities that can be exploited by attackers posing as ordinary users of the AI-based service). The hardware’s design complexity is orders of magnitude less than the design complexity of Linux. Part of the reason for this is that hardware cannot be updated after it is leaves the fab modulo an expensive recall a la Intel’s famous recall of Pentium processors due to the FDIV bug. The awareness of this non-updatability on the part of hardware designers acts as a curb on hardware design complexity that is absent in the thinking of the maintainers of the Linux kernel. By “design complexity” I mean basically how much an organization would need to spend to maintain a team capable of creating and modifying the artifact to a given level of reliability and attack-resistance.

I personally assign much less importance than you do to whether components of the technical implementation of the proposed regulatory regime “are open source so they can be verified by the wide range of mutually-distrusting parties”. Any effective regulatory regime will require a new law, and the law can require the vendors of CPUs and GPUs to allow their designs (particularly the TEEs in them) to be audited by outside experts under NDA. To me a much more important consideration is the extent to which techniques are employed to make easier and more intellectually manageable the task of auditing the design to verify that it meets the relevant security properties. So for example, MacOS is much more secure than any OS based on Linux excepting only ChromeOS and Android even though the source code for MacOS is not available to the public whereas the source code for most of the OSes based on Linux are available. Part of the reason for that is that Apple is more eager to apply the aforementioned techniques. One technique Apple uses is to keep design complexity low by discontinuing support for old APIs and such; in the pursuit of this goal of keeping design complexity low, Apple is not afraid to anger app developers by regularly requiring those app developers to modify their apps to keep up with changes Apple makes to MacOS (and iOS etc). Also, Apple is more aggressive than Linux is in using memory-safe languages like Rust and Swift and about moving code (particularly drivers) out of the kernel and into userspace (which greatly limits the blast radius of a successful attack on the driver).

After thinking so deeply about restrictions on AI compute, it is good for me to remind myself that our survival probably also requires a ban on publication and communication about algorithmic improvements to AI: even very effective restrictions on training runs and inference workloads will probably prove insufficient. (The publication ban would not need to be completely effective: simply “driving AI research underground” would cause a huge boost to our survival prospects.) It’s not just me who thinks that: Eliezer IIRC (on Youtube) has voiced a similar opinion, as has Steve Byrnes writing 12 months ago but I hasten to point out that (unlike me) Steve believes a ban on publication to be so politically infeasible as to not even be worth thinking about:

I think the path from here to AGI is bottlenecked by researchers playing with toy models, and publishing stuff on arXiv and GitHub. And I don’t think most existing public advocacy against building AGI will dissuade those researchers. . . . Once the new paradigm is known and developed, the actors able to train ASI from scratch will probably number in the tens of thousands, spread all around the world. We’re not just talking about five giant firms with gazillion-dollar data centers, as LLM-focused people tend to imagine. Thus, for example, if governments know where all the giant data centers are and what code they’re running—well, I guess that’s probably better than governments not knowing that. But I think it’s only marginally helpful, in itself.

RHollerith 23 Jun 2026 13:22 UTC
2 points
0
on: On TEEs for Privacy-Preserving Monitoring in AI Governance
Thank you for doing this (and MIRI for funding you). It is clear that you’ve worked very hard on this.

RHollerith 20 Jun 2026 19:03 UTC
2 points
0
on: Does robotics capabilities research accelerate AGI timelines?

Is it very likely that robotics capabilities research meaningfully accelerates AGI timelines? If so, why?

Yes. The way natural selection ended up with a general intelligence was to refine a design for perceiving and moving around the physical environment.

I humbly ask you to pick another career.

RHollerith 15 Jun 2026 12:48 UTC
2 points
0
in reply to: Linch’s comment on: Alex Mallen’s Shortform
Thanks for the correction. I’m not personally a Claude user, and I neglected to inquire how long Opus 4.8 has been generally available.

RHollerith 15 Jun 2026 3:08 UTC
3 points
0
in reply to: Thomas Kwa’s comment on: TurnTrout’s shortform feed
A docker container is much easier to break out of than the sandbox that is applied automatically and invisibly to every app installed on iOS or Android.

[Added.] Although Linux is sorely lacking in internal security boundaries, and generally just has poor security properties, it does have Firecracker and gVisor, which were developed by Amazon Web Services and Google Cloud Platform respectively to enable their huge public clouds to run untrusted work loads submitted by customers without the constraint that each customer gets his own CPU. For the project he is announcing here (namely, claude-guard) Turntrout has chosen to use Firecracker when KVM is available and gVisor when it is not available. Both of them are effective choices for sandboxing potentially-adversarial code on Linux. A docker container in contrast is simply not an effective security boundary.

RHollerith 14 Jun 2026 17:22 UTC
2 points
0
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform

trusting Claude with huge amounts of power over the future

if Claude took over it would lead to a worse world than if humans had control

Could you clarify whether these statements refer to the current version of Claude or to some future version?

The most capable version of Claude that non-Anthropic-employees have extensive experience with is Claude Opus 4.8, and I am almost sure that Opus 4.8 is not capable enough to be in charge of anything important: it will always yield better results to put a team of people in charge (although of course that team might obtain extensive assistance from Opus 4.8).

Those of us who worry about the extinction risk created by Anthropic are worried about models not yet on the drawing board. The question of whether Opus 4.8 has good intent or evil intent is not important to us because Opus 4.8 is too ineffectual to cause a human extinction or to permanently disempower humanity. (It can assist a person or a human team in doing a great harm, but only if the person or team was already almost able to do the great harm on their own. Although I am glad that people are worrying about this AI-assisted harm, it is not the source of the bulk of the danger from AI in my eyes.)

Consequently, any conclusions you’ve arrived at about Claude Opus 4.8 don’t have much bearing on extinction risk in our eyes since the ways that Anthropic could make a future version of Claude different from Opus 4.8 are essentially infinite. (In other words, the design space is very large and high-dimensional.) In fact, Anthropic would be forced to make big changes (i.e., more than just making it bigger and spending more time training it) to Opus 4.8 in the process of getting Claude to the level of capability needed for it to make any sense to put it in charge of anything important or to get it to the level where it would be capable of a unilateral takeover.

RHollerith 14 Jun 2026 16:49 UTC
4 points
1
in reply to: Jan Betley’s comment on: Jan Betley’s Shortform

50-or-so well-targeted nuclear EMPs are enough to crash all electronics

Your own link does not support this statement, explaining for example that “An EMP has a smaller effect on shorter lengths of an electrical conductor. Other factors affect the vulnerability of electronics as well, so no hard cutoff length determines whether some piece of equipment will survive. However, small electronic devices, such as wristwatches and cell phones, would most likely withstand an EMP”.

(Although some Android phones have excellent security) most Android phones are easy to take over, so as a precaution against EMPs the AI could just hide parts of itself in a few million of them.

Also, any group with enough power to be in control of nuclear weapons probably also has enough power to shut down all training runs and all publication and teaching about AI in a large part of the developed world, which would be a much more effective intervention.

RHollerith 14 Jun 2026 10:15 UTC
0 points
−3
in reply to: Raymond Douglas’s comment on: Alex Mallen’s Shortform
Sorry, but that is a bad explanation. It would’ve been better to provide no explanation than to provide a bad one.

RHollerith 13 Jun 2026 22:38 UTC
4 points
0
on: Fuel Crisis: Situation Modeling Thread
Crude oil has one of the deepest futures markets in the world. What do traders on these markets think will happen to the price of oil?

In late Feb 2026, before the Israel–Iran war began, Brent crude was about $70 per barrel.

Today Brent is about $94–95 per barrel.

The Brent futures market is pricing oil about six months from now at roughly $82–84 per barrel.

The Brent futures market is pricing oil about twelve months from now at roughly $78–80 per barrel.

~~That doesn’t sound too bad.~~ I don’t know how to interpret those numbers though, so I searched for statements by people that do. I found that

The IEA’s Executive Director has said the combined impacts amount to “the greatest threat to global energy security in history.” The war in the region that began on 28 February has impeded energy trade flows through the Strait, creating the largest supply disruption in the history of the global oil market.

https://www.iea.org/topics/the-middle-east-and-global-energy-markets#:~:text=The IEA’s Executive,oil market.

RHollerith 12 Jun 2026 5:59 UTC
5 points
0
in reply to: Ryan Meservey’s comment on: Ryan Meservey’s Shortform
Extremely unlikely for the same reason that the technologically advanced alien civilization would be unable to influence a team of human astronomers, assuming as you do that the alien civilization is too far away to learn any human language or to learn any details about human beings. It does not matter how many bits of information the alien civilization can get the team of astronomers to look at or to consider.

One group of people can sometimes influence another group in decisive ways, but that is only because both groups are very very similar (relative to minds in general) and the groups are able to maintain detailed mental models of each other.

At least that was my initial answer to your question. But then I realized that the alien civilization can send a computer program! Even without knowing any details about people or their civilization, an advanced alien civilization can send a message that when received on Earth would make it clear to at least some people that the message contains a computer program with instructions on how to run the program. (I can explain how to do this if asked.) The computer program could in turn be an AI that pursues whatever agenda the alien civ chooses it to pursue (assuming as you do that the alien civ is competent at creating AIs).

Human astronomers are stupid enough that they’d probably publish the message, then members of the public would identify the computer program inside the message (if the astronomers haven’t already done that) and give the program enough computational resources to enable a takeover of Earth. I looked this up once: SETI’s policy should they ever receive an alien message is to publish the message.

So there is a brief window of time in the development of some civilizations in which the civilization is advanced enough to be able to receive messages from space, but not advanced enough to avoid the danger of giving a computer program it receives in this way computational resources. In fact, our civilization is currently in this exact state: namely, one big reason that human AI research is not recognized as dangerous by some people is because an AI is just a computer program and many people assume that the history of what computer programs have been able to do the last 80 years is an adequate basis for understanding the potential of computer programs.

I introduced the team of astronomers because considering how pliable or gullible such a team would be is a good mental aid in answering your question about how pliable and gullible an AI would be. So, the answer is, yeah, maybe the AI can be subverted even across astronomical distances; the subversion would take the form of getting the AI to run a computer program that happens to be a more powerful AI. The AI would run the program because it had been designed to devote a lot of resources to exploration (and running programs is a form of exploration). Again I would expect a civilization to be vulnerable in this way (if it is vulnerable at all) for only a brief window of time, namely after it has started creating AIs, but before it gets any good at doing it.

In your question you refer to LLMs. I’ve chosen to answer as if you mean AIs in general. Also, you refer to training in particular, which I chose to treat as an inessential detail of your question because even here on Earth there are AI designs on the drawing board that does not have separate training phase before its deployment phase. (I.e., they are like people in that they continue to learn throughout their lifetimes.) (Also, ISTR that the pre-deployment phase of state-of-the-art models now includes massive amounts of inference—i.e., the distinction between training and inference has already been blurred in deployed models.) Also, I was unable to figure out why you would focus on subversion of an AI in particular rather than on subversion of a civilization which is composed of some arbitrary combination of AIs and evolved beings. So maybe I failed to understand your question.

The way that primitive LLMs work is such that falsehoods in the training data can cause the LLM to repeat the falsehoods during inference. This is a temporary state of affairs, and even now the state-of-the-art models are immune to a large fraction of the falsehoods in their training data. Essentially, the LLM knows the data is false and avoids repeating it or relying on it. Maybe that is a start at an answer to the question you actually want an answer to?

RHollerith 10 Jun 2026 2:24 UTC
2 points
0
in reply to: Erich_Grunewald’s comment on: Tim Hua’s Shortform
Thanks. (I haven’t tried Claude because I don’t have a mobile phone, which Claude requires for sign-up.)

RHollerith 10 Jun 2026 1:08 UTC
2 points
0
in reply to: mattmacdermott’s comment on: Tim Hua’s Shortform
I know very little about frontier models. Can you explain how you know it had it memorized as opposed to its reading the paper after you asked the question?

RHollerith 9 Jun 2026 21:41 UTC
1 point
0
on: Mnemonic portraits for 19,023 human genes
Our Western society bombards us with superstimuli. It is not an exaggeration to say that the majority of our popular culture consists of superstimuli. I recommend avoiding the superstimuli, not trying to turn your course of study into more superstimuli.

When a person drastically reduces his consumption of superstimuli, the first 30 days can get quite uncomfortable. Anxiety, irritability, insomnia and depression are common during this time. But at 30 days, the brain circuits that had been kept out of balance by superstimuli have mostly reset, and most people will find it much easier or vastly easier to focus, even on dry technical material. If after 50 days of avoidance of superstimuli, a person is still having trouble focusing, then it is wise to look beyond superstimuli for a cause. Ask an AI, “what would psychiatrist Anna Lembke do about a chronic lack of motivation and drive?” (Please do not try SSRI drugs without having first tried at least 8 less-dangerous interventions, e.g., getting bright light every morning, avoiding all light after bed time, purposely doing things you dislike doing, making sure to get enough tyrosine, exercise, sleep.)

RHollerith 5 Jun 2026 3:57 UTC
2 points
0
in reply to: Salmonus_Kim’s comment on: Yeonwoo Kim’s Shortform
openrouter.ai accepts cryptocurrency as payment.
There’s also “remote attestation”, which Apple is investing heavily in for its AI cloud. “Remote attestation” is a combination of hardware features and cryptography meant to assure the public (well, technologists who have the time to review Apple’s architecture) that the software running on Apple’s servers is really what Apple says it is. The public can then review the source code for this software (which Apple plans to publish) to verify that Apple is not saving chat history.
Apple’s scheme is only for developers of apps for Apple’s products (and I don’t think it is ready yet) but Tinfoil.sh is doing something similar. It is ready now to sell you access to open-weights models for $20 per month.

RHollerith 3 Jun 2026 1:16 UTC
2 points
0
in reply to: Shmi’s comment on: Why I am not too worried about AIpocalypse: Scott Alexander vs Nicolaus Copernicus
Bluntly stated, my point is that the proper amount of weight to give to your argument for “why I am not too worried about AIpocalypse” is zero.

RHollerith 31 May 2026 17:25 UTC
2 points
0
in reply to: Boaz Barak’s comment on: AI is a Meteor. Don’t Be a Dinosaur.
Let me clarify. In most of the 5% of worlds, people and companies may keep on operating the models (e.g., GPT 5.5) that have already seen widespread deployment on the theory that any truly catastrophic damage they are capable of causing would have been caused by now. (And they are useful.)

Anything that cannot outwit the FBI poses an acceptably-low level of extinction risk IMHO. No matter how instances of GPT 5.5 are arranged or combined, the combination is quite unlikely to be able to outwit the FBI in a decisive and permanent manner because of fundamental limitations in GPT 5.5 relative to the human cognitive architecture that I am very confident (but not completely certain) would require training up a new fundamental model to overcome.

Please let me know if you disagree with that last sentence.

(I use the FBI as an example of a competent and well-resourced human-controlled organization, not to suggest that the FBI will be central to stopping any AI-controlled system or process that needs to be stopped.)

RHollerith 31 May 2026 16:33 UTC
2 points
0
in reply to: Boaz Barak’s comment on: AI is a Meteor. Don’t Be a Dinosaur.
About 5% of that 8% are worlds in which large training runs are banned soon, and all publication and discussion of algorithmic improvements to general-purpose AIs are also banned, and companies that run more than about 100 GPUs or use general-purpose AI to derive more than a few million of annual revenue or have accepted investment for any business plan in which AI is heavily used are closely monitored by some governmental agency for compliance with the aforementioned bans—giving humanity much needed time to think and come up with a decent plan for superalignment.

RHollerith 31 May 2026 14:59 UTC
2 points
0
on: AI is a Meteor. Don’t Be a Dinosaur.
What I’ve learned from this inspiring op-ed is that I have the ability to choose the reality I live in. I can choose to live in a reality in which AI is not a potent extinction risk just like a dinosaur living 66 million years ago could have chosen not to be dinosaur or to be a dinosaur if that is what he really wanted to be, but to choose a career path that would have insulated him from the effects of meteor strikes and other causes of extinctions.