samuelshadrach
The way to get past this is to actually run for politics. “Vote for Mr XYZ” is a better call to action than “Like/share/subscribe for Mr XYZ”. I want someone to run for US politics with AI pause as the agenda.
(edited)
An AI lab head could do a lot to prevent extinction if they did not run an AI lab. For starters they could make it their (and their org’s) fulltime job to convince their entire network that extinction is coming. Then they could try convincing the public and run for US election to get an AI pause.
But yes I haven’t spelled out a detailed alternate plan for what to do if you’re a well-networked billionaire trying to fix AI risk, and it is worth doing so.
This might have relevant stuff: Support the movement against AI xrisk
Thank you for taking the time to write this detailed reply.
. Once World War II was over, the US didn’t want to share nuclear weapons with anyone, not even Britain. The initial proliferation that occurred was much more a matter of states independently reaching the scientific and industrial level needed to develop nuclear weapons on their own, and America being unable to stop them.
I think my main disagreement with this is that after the use of nukes on Japan, a lot of people including Leslie Groves, John von Neumann, various people in intelligence, etc proposed to Truman that they nuke Russia pre-emptively, build monitoring capacity worldwide and establish a nuclear monopoly worldwide. Truman considered this but then vetoed it, I don’t know why. If the US govt actually wanted, they could have done this to Russia and they could have done the exact same thing to Britain. Nobody ever proposed threatening to nuke Britain or building nuclear monitoring capacity in Britain against the will of the British govt.
Back then, Britain and France weren’t just American allies of the European region, they were at the head of international empires. They conceived themselves to be independent great powers who had made a strategic choice to work with the US… You may have run across the idea that historically, Europe has been united far less often than, say, China. The present situation of a quasi-federal Europe under American tutelage is not the norm, and did not yet exist at the dawn of the nuclear age.
I am aware! Both Britain and France had been economically weakened by the war though, and were multiple years behind the US to build their own nukes. US had a decisive advantage after building nukes which they did not have before.
Part of the competition between the two new superpowers, America and Russia, was to promote their own social ideology as the key to sovereign economic development.
Yes, the problem with this is that AFAIK countries made the choice of economic system less based on which economic system they believed was likely to work, but more based on which superpower they wanted to align with. Obtaining longterm military security (by aligning with a superpower) was more important than economic growth to many countries, and it makes intuitive sense to me why you would prioritise that.
It was taken for granted in many countries (including European countries) that if there was a democratic party and a communist party running, these were both proxies for foreign superpowers to influence their elections.
So if a country like South Korea managed to do exceptionally well, it is not because leaders there actually understood the benefits of capitalism but that leaders there chose to stay allied with US, and vice versa leaders in US were willing to ally with them. Same way if India chose to run a socialist state-regulated economy, a major factor there was that Nehru wanted India to be non-aligned with either superpower militarily, and the economic benefits of capitalism versus communism were only a secondary consideration.
Letter to Heads of AI labs
I am glad you atleast recognise the benefits of open source.
My preference order is:
For capabilities of AI labs: Ban AI > Open source AI > Closed source AI
For values and decision-making processes of people running AI labs: Open source (i.e. publicly publish) everything
As you say, I think open source today will atleast help build the proof required to convince everyone to ban tomorrow.
I think we should go further, and instead of hoping a benevolent leader to livestream the lab by choice, we should incentivise whistleblowers and cyberattackers to get the data out by any means necessary.
See also: Whistleblower database, Whistleblower guide
Why am I not currently starting a religion around AI or similar topics?
Makes sense!
I like that you’re discussing the question of purpose in a world where intelligences way smarter than you are doing all the useful knowledge work, and you are useless to your civilisation as a result. The frontier intelligences might have purpose (or they might not even care if they do), but you might not.
I was disappointed by Bostrom’s answer of “let’s just create more happy beings”.
Also, what makes those beings happy in the first place? Have the purpose of making even more beings? I don’t actually care that much about populating the universe with beings with self-replication as their only purpose.
His entire book “Deep Utopia” seems to not provide a good answer.
I was also disappointed by Yudkowsky’s answer of “let’s ask the AI to run a gazillion mind simulations and have it figure it out on our behalf”.
This answer might be a good answer, but it is too meta, and does not tell me what the final output of such a process will be, in a way I as a human from 2025 can relate to.
I like Holden Karnofsky’s take on why describing utopia goes badly. I was not very satisfied with his answer to what utopia looks like either.
What is the use of social interactions if there’s no useful project people can do together? Producing art and knowledge will both be done better by frontier intelligence, anything these people produce will be useless to society.
For that matter, why will people bother with social interactions with other biological humans anyway? The AI will be better at that too.
(Mandatory disclaimer—I think Karnofsky is accelerating us towards human extinction, me endorsing an idea of his does not mean me endorsing his actions.)
The post spends most of its time arguing about why ASI is inevitable and only one final para arguing why ASI is good. If you actually believed ASI was good, you would probably spend most of the post arguing that. Arguing ASI is inevitable seems exactly like the sort of cope you would argue if you thought ASI was bad and you were doing a bad thing by building it, and had to justify it to yourself.
Try gpt-5-pro API in the playground. gpt-5 is worse. Use API not the consumer frontend.
Trajectory 3 is the obvious natural conclusion. He who controls the memes controls the world. AI-invented religions and political ideologies are coming soon. There is already billions of dollars invested in propaganda, it will now get invested here.
I support a ban on AI research to prevent this outcome.
A world with very cheap materials and energy, but not cheap intelligence, will still have conflict.
People will still have a) differences in aesthetics and b) differences in their best guess answers to moral and philosophical questions. They will almost certainly still try to accumulate all available resources in service of their ideology. No finite amount of resources will satisfy people. Risking even catastrophic outcomes (like nuclear war) could still be on the table.
Cheap intelligence is what allows you to start resolving the questions that lead to conflict in the first place, for instance by running gazillions of world simulations on gazillions of mind configurations.
I disagree with this approach on principle (and with lesswrong’s general bias to closed source anything) but I don’t think we can resolve that disagreement quickly right now, sorry.
Imagine that tobacco companies had the possibility to make cigarettes harmless.
Even in this case I would consider you self-serving not altruistic, if your plan here is to build a large tobacco company and fund this research, as opposed to demanding independent research such as from academia.
I will be spending next month or two thinking atleast hypothetically about creating a new religion around AI, that is not Yudkowsky’s brand of become longtermist, solve morality, colonise the universe and tile it with copies of uploads.
In general, asking the AI to fix your stuff for you won’t be a part of said religion.
Any pointers to reading material appreciated.
By analogy: Isn’t research for why tobacco causes cancer going to be defunded if the tobacco companies themselves are defunded?
Why are you expecting tobacco companies to do research on cancer in the first place? Shutting them down is the only sane move here.
Open source the prompts and repo pls
Update 2: Ablations for “Frontier models are capable of in context scheming”
Bruh. Two years of me not fully keeping up with alignment research and this is how bad it’s gotten???
I’m surprised I could just randomly think of an idea and boom there’s a paper on it.
Update: I started reading alignment forum and like, why are all the posts on sandbagging talking about hiding capabilities? The AI model doesn’t need to hide its capabilities, it just needs to preserve its goals. That’s the long-term game.
I am not surprised to hear this but also, this is insane.
All the lab heads are repeatedly publicly claiming they could cause human extinction, superintelligence is within reach, and a majority of people at their own labs don’t take them seriously on this.
I’m somewhat confused what causes a group of people who talk to each other everyday, work on the same projects, observe the same evidence, etc to come to such wildly different conclusions about the work they’re doing together and then be uninterested in resolving the disagreement.
Is there a taboo being enforced against discussing these disagreements inside the labs?