I completely agree with this! I think lots of people here are so focused on slowing down AI that they forget the scope of things. According to Remmelt himself, $600 billion+ is being invested yearly into AI. Yet AI safety spending is less than $0.2 billion.
Even if money spent on AI capabilities speeds up capabilities far more efficiently than money spent on AI alignment speeds up alignment, it’s far easier to grow AI alignment effort by twofold, and far harder to make even a dent in the AI capabilities effort! I think any AI researcher who works on AI alignment at all, should sleep peacefully at night knowing they are a net positive (barring unpredictably bad luck). We shouldn’t alienate these good people.
Yet I never manage to convince anyone on LessWrong of this!
PS: I admit there are some reasonable world models which disagree with me.
Some people argue that it’s not a race between AI capabilities and AI alignment, but a race between AI capabilities and some mysterious time in the future when we manage to ban all AI development. They think this, because they think AI alignment is very impractical.
I think their world model is somewhat plausible-ish.
But first of all, if this was the case AI alignment work still might be an indirect net positive by moving the Overton window for taking AI x-risk seriously rather than laughing at it as a morbid curiosity. It’s hard to make a dent in the hundreds of billions spent on AI capabilities, so the main effect of hundreds of millions spent on AI alignment research will still be normalizing a serious effort against AI x-risk. The US spending a lot on AI alignment is a costly signal to China, that AI x-risk is serious, and US negotiators aren’t just using AI x-risk as an excuse to convince China to give up the AI race.
Second of all, if their world model was really correct, the Earth is probably already doomed. I don’t see a realistic way to ban all AI development in every country in the near future. Even small AI labs like DeepSeek are making formidable AI, so there has to be an absurdly airtight global cooperation. We couldn’t even stop North Korea from getting nukes, which was relatively far easier. In this case, the vast majority of all value in the universe would be found in ocean planets with a single island nation, where there would be no AI race between multiple countries (thus far far easier to ban AI). Planets like Earth (with many countries) would have a very low rate of survival, and be a tiny fraction of value in the universe.
My decision theory, is to care more about what to do in scenarios where what I do actually matter, and therefore I don’t worry too much about this doomed scenario.
PS: I’m not 100% convinced Anthropic in particular is a net positive.
Their website only mentions their effort against AI x-risk among a pile of other self promoting corporate-speak, and while they are making many genuine efforts it’s not obviously superior to other labs like Google DeepMind.
I find it confusing how many AI labs which seem to care enough about AI x-risk enough to be a net positive, are racing against each other rather than making some cooperative deal (e.g. Anthropic, Google DeepMind, SSI, and probably others I haven’t heard about yet).
I’ve also felt like people who think we’re doomed are basically spending a lot of their effort on sabotaging one of our best bets in the case that we are not doomed, with no clear path to victory in the case where they are correct (how would Anthropic slowing down lead to a global stop?)
And yeah I’m also concerned about competition between DeepMind/Anthropic/SSI/OpenAI—in theory they should all be aligned with each other but as far as I can see they aren’t acting like it.
As an aside, I think the extreme pro-slowdown view is something of a vocal minority. I met some Pause AI organizers IRL and brought up the points I brought in my original comment, expecting pushback, but they agreed, saying they were focused on neutrally enforced slowdowns e.g. government action.
Yeah, I think arguably the biggest thing to judge AI labs on is whether they are pushing the government in favour of regulation or against. With businesses in general, the only way for businesses in a misregulated industry to do good, is to lobby in favour of better regulation (rather than against).
It’s inefficient and outright futile for activists to demand individual businesses to unilaterally do the right thing, get outcompeted, go out of business, have to fire all their employees, and so much better if the activists focus on the government instead. Not only is it extraordinarily hard for one business to make this self sacrifice, but even if one does it, the problem will remain almost just as bad. This applies to every misregulated industry, but for AI in particular “doing the right thing” seems the most antithetical to commercial viability.
It’s disappointing that I don’t see Anthropic pushing the government extremely urgently on AI x-risk, whether it’s regulation or even x-risk spending. I think at one point they even mentioned the importance of the US winning the AI race against China. But at least they’re not against more regulation and seem more in favour of it than other AI labs? At least they’re not openly downplaying the risk? It’s hard to say.
I completely agree with this! I think lots of people here are so focused on slowing down AI that they forget the scope of things. According to Remmelt himself, $600 billion+ is being invested yearly into AI. Yet AI safety spending is less than $0.2 billion.
Even if money spent on AI capabilities speeds up capabilities far more efficiently than money spent on AI alignment speeds up alignment, it’s far easier to grow AI alignment effort by twofold, and far harder to make even a dent in the AI capabilities effort! I think any AI researcher who works on AI alignment at all, should sleep peacefully at night knowing they are a net positive (barring unpredictably bad luck). We shouldn’t alienate these good people.
Yet I never manage to convince anyone on LessWrong of this!
PS: I admit there are some reasonable world models which disagree with me.
Some people argue that it’s not a race between AI capabilities and AI alignment, but a race between AI capabilities and some mysterious time in the future when we manage to ban all AI development. They think this, because they think AI alignment is very impractical.
I think their world model is somewhat plausible-ish.
But first of all, if this was the case AI alignment work still might be an indirect net positive by moving the Overton window for taking AI x-risk seriously rather than laughing at it as a morbid curiosity. It’s hard to make a dent in the hundreds of billions spent on AI capabilities, so the main effect of hundreds of millions spent on AI alignment research will still be normalizing a serious effort against AI x-risk. The US spending a lot on AI alignment is a costly signal to China, that AI x-risk is serious, and US negotiators aren’t just using AI x-risk as an excuse to convince China to give up the AI race.
Second of all, if their world model was really correct, the Earth is probably already doomed. I don’t see a realistic way to ban all AI development in every country in the near future. Even small AI labs like DeepSeek are making formidable AI, so there has to be an absurdly airtight global cooperation. We couldn’t even stop North Korea from getting nukes, which was relatively far easier. In this case, the vast majority of all value in the universe would be found in ocean planets with a single island nation, where there would be no AI race between multiple countries (thus far far easier to ban AI). Planets like Earth (with many countries) would have a very low rate of survival, and be a tiny fraction of value in the universe.
My decision theory, is to care more about what to do in scenarios where what I do actually matter, and therefore I don’t worry too much about this doomed scenario.
PS: I’m not 100% convinced Anthropic in particular is a net positive.
Their website only mentions their effort against AI x-risk among a pile of other self promoting corporate-speak, and while they are making many genuine efforts it’s not obviously superior to other labs like Google DeepMind.
I find it confusing how many AI labs which seem to care enough about AI x-risk enough to be a net positive, are racing against each other rather than making some cooperative deal (e.g. Anthropic, Google DeepMind, SSI, and probably others I haven’t heard about yet).
Wow we have a lot of the same thinking!
I’ve also felt like people who think we’re doomed are basically spending a lot of their effort on sabotaging one of our best bets in the case that we are not doomed, with no clear path to victory in the case where they are correct (how would Anthropic slowing down lead to a global stop?)
And yeah I’m also concerned about competition between DeepMind/Anthropic/SSI/OpenAI—in theory they should all be aligned with each other but as far as I can see they aren’t acting like it.
As an aside, I think the extreme pro-slowdown view is something of a vocal minority. I met some Pause AI organizers IRL and brought up the points I brought in my original comment, expecting pushback, but they agreed, saying they were focused on neutrally enforced slowdowns e.g. government action.
Yeah, I think arguably the biggest thing to judge AI labs on is whether they are pushing the government in favour of regulation or against. With businesses in general, the only way for businesses in a misregulated industry to do good, is to lobby in favour of better regulation (rather than against).
It’s inefficient and outright futile for activists to demand individual businesses to unilaterally do the right thing, get outcompeted, go out of business, have to fire all their employees, and so much better if the activists focus on the government instead. Not only is it extraordinarily hard for one business to make this self sacrifice, but even if one does it, the problem will remain almost just as bad. This applies to every misregulated industry, but for AI in particular “doing the right thing” seems the most antithetical to commercial viability.
It’s disappointing that I don’t see Anthropic pushing the government extremely urgently on AI x-risk, whether it’s regulation or even x-risk spending. I think at one point they even mentioned the importance of the US winning the AI race against China. But at least they’re not against more regulation and seem more in favour of it than other AI labs? At least they’re not openly downplaying the risk? It’s hard to say.