Richard Hollerith. 15 miles north of San Francisco. hruvulum@gmail.com
My probability that AI research will end all human life is .93. It went up drastically when Eliezer started going public with his pessimistic assessment in late 2021. Till then my confidence in MIRI was keeping my probability down to about .4. (I am glad I found out about Eliezer’s assessment.)
Currently I am willing to meet with almost anyone on the subject of AI extinction risk.
Last updated 10 Aug 2026.
There a difference between an apology and giving the reader any sign at all that you have any awareness of how your message is likely to land.
a house suitable to raise a small family in London, an MP on an MP’s salary could not get a mortgage on that, they would need some other source of funds
There are powerful non-monetary incentives to be in Parliament. Since UK law requires all government ministers to be MPs, most of the most powerful 500 people in the UK are MPs. Many people find power to be very intoxicating even if one of the conditions for holding that power is that it is illegal to convert that power into personal wealth. But the same analysis does not apply to the other 99.999% of jobs in the UK, so it would be nice if you could come up with an example population of workers other than MPs. Almost any other population would do. (Rock musician would not do because there are powerful non-monetary incentives to being a male rock musician because many young women find rock musicians extremely attractive.)
Are the rates of having high personal wealth lower in the US than in the UK? I doubt it is much lower. So if (as you claim) it is the presence of high-net-worth individuals in the labor market that keeps salaries low in the UK, then why is the mechanism you claim is operating in the UK so much weaker in the US? That is, why are US salaries, particularly for jobs in computing and AI, much higher than UK salaries?
Do you claim that the same mechanism operates in other markets or is it restricted to the labor market? For example, do high-net-worth individuals on average pay more for the same car as a no-net-worth individual because they can afford to pay more? Do high-net-worth individuals sometimes pretend to be no-net-worth individuals when talking to car salesmen to get a better price? It seems to me that they almost never bother doing that because they don’t have to because the salesman (and the car dealership he works for) regards one pound as just as desirable as any other pound regardless of the wealth status of the person giving them the pound. There a people who hold an animus towards wealthy people, but even such a person if employed as car salesman will almost always care much more about making a sale (and consequently earning a commission on that sale) than on disadvantaging the wealthy person or making the wealthy person suffer a little. Increasing personal income is just a much more compelling consideration than most considerations for the average person.
Would the cost of a particular make and model of car in country X or city X be lower if no one in country X or city X had any savings? My guess is no. It costs Y pounds to make the car, so that Y pounds acts as a strict lower bound on the average selling price of the car. A seller would like the average selling price to be as high as possible, but buyers are price sensitive, so if the seller’s price is higher than the price of another seller of the same make and model of car, buyers will just buy from the other seller till the other seller gets low on inventory and the prices equalize. These are the main 2 mechanisms determining the prices of cars, and I don’t see how the net worth of the buyers affects these mechanisms in any significant way.
I always thought the labor market works basically the same way: if employer A is willing to pay salary X to job Y and additionally if employer B stubbornly refuses to pay more than 0.9X, then pretty soon employer B will be unable to hire any employees willing and able to do Y, so the salaries offered by A and B will tend to equalize (unless for example B’s workplace is much easier to commute to than A’s or some other factor, but all of these factors tend to be minor factors relative to salary).
I expect wealthy people to care about salary, too, such that as long as employer B stubbornly refuses to pay more than 0.9X, it will find it difficult to hire even the very wealthiest people. People don’t stop wanting more money just because they already have a lot of money. If you ask people in a survey how much income would they need to be happy, they answer about 30% more than their current income regardless of whether their current income is 10,000 pounds a year, 20,000, 40,000, 80,000 or 160,000.
If the salaries A and B are willing to pay get high enough, then people who previously never considered being a Y will start considering it and start acquiring the necessary qualifications. Also, if the economy changes such that the products of A and B no longer command as high a price as they once did, then the amount A and B are able to pay for labor used in the production of those products must go down.
Again I always thought that this is the basic dynamics by which salaries are set, and again I fail to see how the average net worth of the employees and job applicants affects the dynamics significantly. In fact, the largest effect (although not particularly large) I imagine is its raising salaries because there are unavoidable aversive aspect to holding down most high-paying jobs, and people of wealth will tend to choose to avoid those aversive aspects by taking themselves out of the market for those jobs, requiring employers to raise salaries (at least a little) to attract sufficient numbers of applicants.
Neglecting to include any sign that you are aware of the basic context in which you are writing is a sign that you are not worth paying attention to. But maybe persuading people is not your motivation here?
Did it ever occur to you to prepend your comment with, “I know most people on this site are going to disagree with me on this,” or, “I know this is the opposite of what most readers on this site would consider a sane response to the current situation,” or words to that effect?
Because there is too much text on LW, I’ve taken this conversation private. If you (Mayowa or anyone else) want to be part of the conversation or would prefer the conversation to remain public, let me know.
It clearly has the formal power to do it.
The move can be made more palatable to voters and lobbyists by letting a new entity or a handful of new entities, none of whose managers has ever worked for a lab, continue to offer the services that have already seen widespread deployment, e.g., Claude 5 and GPT 5.6. The new entity would be heavily regulated, i.e., would be more like a utility company than like a typical tech company, and monitored to make sure it is not doing research or trying to create better models.
During WWII, the automobile industry in Detroit was the star of the economy, like Silicon Valley is today, but that did not stop the War Production Board, established by President Franklin D. Roosevelt in 1942, from making it illegal for the private sector (in Detroit and elsewhere) to continue to produce civilian cars (ditto refrigerators, radios, vacuum cleaners, and metal toys) or to use strategic raw materials (steel, copper, aluminum, rubber) to serve the civilian market. They did this to ensure plenty of raw materials and industrial capacity to fulfill defense contracts for tanks, aircraft engines, trucks, and artillery.
This post offers only a handful of vague category labels — “bad things,” “bad policies,” “AI capabilities,” “morally wrong” — but no definition or operationalization of any of those terms and no example of a task that straddles the border between morally acceptable and morally wrong.
It is important for us to choose (as a community) and then to promote a standard of conduct, then to induce researchers, engineers, entrepreneurs, investors, etc, to adhere to that standard of conduct. Since the number of AI engineers and researchers is low compared to the number of adults in general, any standard not comprehensible or legible to an educated adult who is paying attention, but is not an AI researcher or engineer would be an unsatisfactory, ineffective standard.
I personally doubt that it should be acceptable according to our movement’s standard of conduct to be employed by or take money from a big AI lab — even for a “safety” role or a charitable mission — but of course I don’t get to decide the standard — the community as a whole does — and I acknowledge that many think that such a step would be too hostile and divisive. (Lesswrong has never tried to prevent or discourage employees of the labs from posting here, and I have no objection to the continuation of that policy.) I’d make an exception for employees increasing the security of the labs, i.e., to make it harder to steal model weights.
It is time for the anti-extinctionist community to start to use effective social measures to oppose the AI labs. The benefits of doing so will outweigh the negative effects of the defensive reaction the measures will provoke. In particular, it relatively unimportant for more information about the labs to be published: anyone who cannot be persuaded that AI research is a potent extinction risk using the information already in the public domain won’t be persuaded by any new information out of the labs.
One social measure we need to start to use is pressuring and forcing people to pick a side. In order to start to do that, we first need to define as a community what standard of conduct will be required for someone to be a member in good standing of our movement.
Holly has put more pressure on people to pick a side than anyone else in AI anti-extinctionism.
Our side’s putting strong pressure on people to pick a side seems essential to me.
It must be done skillfully though. I don’t know enough about it to judge Holly’s level of skill.
I agree with everything in your latest comment, specifically, that implementation of a good food backup plan would have a reliably positive effect on p(extinction).
In my previous comment, I wrote “X-risk reduction” where I should’ve written “AI X-risk reduction” and “extinction” where I should’ve written “extinction caused by AI research”. Oops.
AI extinction risk is significantly harder to analyze than nuclear-war extinction risk. MIRI received a donation of roughly $50 million (from a crytpocurrency entrepreneur) and as of Feb 2023 it had been sitting in a bank account for many years because MIRI knew of no effective way to use it to reduce p(extinction from AI):
If I just have another $50 million, I’m not quite sure what to do with that, but if you donate that to MIRI, then you at least have the assurance that we will not randomly spray money on looking like we’re doing stuff and we’ll reserve it, as we are doing with the last giant crypto donation somebody gave us until we can figure out something to do with it that is actually helpful.
https://www.lesswrong.com/posts/Aq82XqYhgqdPdPrBA
In summary, I doubt a person can offset the harm caused by working for an AI lab by donating money.
My point was to respond to my interlocutor’s challenge, “unless you can concretely claim that, had you stumbled onto any similar cause instead of AI Safety, you wouldn’t have taken up their crusade instead”.
I wish one of the labs would sue some of us for defamation: at this stage in the game, that would be a serious blunder on their part.
(At least it would be if they did it in an American court. I don’t know enough about the legal systems of other countries to say).
As for AI, donating to existential risk reduction might have a cost effectiveness of ~~$300 M per basis point reduction in X risk.
I’m very skeptical that you or anyone contemplating any plan of X-risk reduction can know the sign of any effect on p(extinction) if the magnitude is that small.
Although reasonable people can disagree on whether the effect of a plan on X-risk reduction would be positive or negative (on net), clearly most spending on X-risk reduction past and future has had or will have significant negative effects as well as positive effects, so it is extremely unlikely that a Bayesian reasoner would assign a probability that close to zero for the effect of any particular intervention, and if by chance one ever did, it would just as likely be on the negative side of zero as the positive side.
(I’m assuming that your “basis point” means a 0.01 change in p(extinction).)
I will go further: I will conjecture that there is no intervention or plan available to us (that we can identify with our puny rationality skills) whose effect on p(extinction) is reliably positive. If you have a plan in mind whose effect you believe to be reliably positive, please share it with us. Until I see such a plan, though, I’m going to regret your comment because the only significant effect it will have is to give capability researchers another justification for continuing to do what they want to do.
This comment might interest you:
If these people were so concerned, they’d by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. . . . Maybe they don’t actually feel very threatened. Look at their actions, not their words.
In the 1970s (when I was a teenager) many people were concerned that overpopulation plus pollution would wreck the environment to such an extent as to cause widespread social chaos, which might not be recoverable from. I never worried much about that. For decades, I knew that nuclear war among the great powers, although it would be tragic, had very little chance of causing human extinction. (It required 100s of hours of reading for me to conclude that.) I think we should all make sacrifices to reduce the magnitude of global warming, but I never took seriously the possibility that even drastic warming might cause human extinction (as opposed to a severe reduction in the human population).
I never imagined marriage counseling to be a useful analogy to anti-extinction advocacy.
Rather, I was refuting the assertion that “it’s absurd to make this distinction” by pointing out an endeavor in which practitioners routinely make the distinction to apparently good effect, which is strong evidence against the hypothesis that it’s internally contradictory to make the distinction, which is what I took great grandparent to be asserting.
I wouldn’t call it absurd. Marriage counselors frequently point out that when one spouse insists the other has a character problem, the marriage is rarely salvageable. There is a vital difference between criticizing a spouse’s behavior and declaring that they have a bad character (even though an onlooker who hears enough of the former will tend to infer the latter).
Having just defended the principle as non-absurd, let me add that I’m undecided on whether the principle should be adhered to by people doing what Holly Elmore does. On the one hand, if the movement is too well-behaved, some people will ignore us because they will see us as ineffectual or as not actually believing what we say we believe. On the other hand, a potential ally just getting to know us can form a long-lasting negative impression of us from even minor instances of combative behavior.
My guess is that we’re not doing enough combative things, but that relative to what Holly’s been doing, it would alienate fewer onlookers for us to start running boycotts and to take every opportunity to sue accelerationists in court. Again, the main benefit from doing such things is to show people we’re effective and actually believe what we say we believe. Many won’t take any trouble to learn about AI extinction danger until they see such signs.
Who cares? Why should labs listen to you?
The leaders of the labs want to survive just like everyone else does. If someone were to figure out and announce an actually satisfactory alignment plan, those leaders would probably embrace it, maybe not at first, but after the number of people publicly promoting the plan had grown large, because at some level many of them probably strongly suspect that the plan they have now is unsatisfactory.
They persist with their current plan because the pursuit of power is very motivating and compelling to them. Give them a way to hold on to their power while decreasing extinction risk, and there is a good chance they would take it. Of course, that only helps us if someone comes up with a satisfactory alignment plan, which is a huge if.
Of course not everyone at the labs is motivated by power or personal prestige. Another big motivator is an ideological commitment to technological progress and the bright science-fiction future. But maybe the basic analysis above partly applies to people with that motivation, too.
I never gave any money to OpenAI or xAI (the most reckless large labs in my humble estimation) and do not intend to.
I’ve started a trial of using an open-weights model for most of my AI needs. If the trial is successful, I would still tend to be enriching and enlarging an inference-serving company and (indirectly) makers of hardware for inference, which is not good, but at least I would have stopped enriching and enlarging any of the labs except to the very minor extent of providing organic advertising to the lab that created the model I would be using.
I don’t consider any of the issues you mention to shed any light because none of them were ever potent threats the survival of the power elite. A nuclear exchange between the great powers does shed light (because while the power elite in both the US and the USSR would’ve probably survived a nuclear exchange initiated by their country, it would have severely impacted the quality of life of themselves and their families for many years) and of course no decisionmaker has ever chosen to initiate such a nuclear exchange.
non-AI-researchers don’t have any means of distinguishing fantastical-but-true from fantastical-and-false,
If that is what you believe then I can see how you would end up believing the rest of what you wrote.
I’ve never been an AI researcher and I can correctly distinguish AI extinction danger as an actual potent extinction danger among the many candidates I’ve heard over my lifetime.
Holly Elmore (head of Pause AI US) has never been a researcher or engineer in AI or even in computing, and yet she’s been loudly proclaiming since 2023 that AI is a potent extinction danger whereas I’ve never heard her talk publicly about any extiction danger other than AI.
Jaan Tallinn has never been a researcher or engineer in AI (his technical career was in game development and peer-to-peer networking; he has a degree in physics) and yet he considers AI to be such a potent danger that he has spent $150 million of his money on AI safety. His spending on AI safety started early 2009 and continues.
Philosopher David Chalmers has never been anything like an AI researcher or engineer and yet he signed a public statement that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” So did many other non-AI-researchers.
This is why I dislike the phrase “existential risk” and have started using “extinction danger” or “potent extinction danger” instead along with making it clear that it is human extinction in particular that I’m warning about.
It is too easy for a reader to conclude that an “existential risk” must lie far in the future or must have a probability so tiny that the people who consider it a risk or a danger must be relying on a Pascal’s-wager type of argument.