A bunch of people in the AI safety landscape seem to argue “we need to stop AI progress, so that we can make progress on AI safety first.”
One flip side to this is that I think it’s incredibly easy for people to waste a ton of resources on “AI safety” at this point.
I’m not sure how much I trust most technical AI safety researchers to make important progress on AI safety now. And I trust most institutions a lot less.
I’d naively expect if any major country would throw $100 Billion on it today, the results would be highly underwhelming. I rarely trust these governments to make progress on concrete technologies with clear progress measures, and “AI Safety” is highly ambiguous and speculative.
As I’ve written about before, I think it’s just hard to know what critical technical challenges will be bottlenecks around AI alignment, given that it’s unclear when this will become an issue or what sorts of architectures we will have then.
All that said, slowing things down seems much safer to me. I assume that at [year(TAI) − 3] we’ll have a decent idea of what’s needed, and extending that duration seems like a safe bet.
I really want to see better strategic discussion about AI safety. If somehow we could spend $10B just to get a better idea of what to actually do, I’d easily suggest that, though strategy is something that’s typically very difficult to spend money on.
Personally, this is one reason why I favor the meta approach of “make better epistemic tools, using AI.” This is an area that can be very concrete and achievable, though it does have its own problems.
This is an orthogonal question. I agree that if we’re there now, my claim is much less true.
I’d place fairly little probability mass on this (<10%) and believe much of the rest of the community does as well, though I realize there is a subset of the LessWrong-adjacent community that does.
I’m saying that just because we know algorithms that will successfully leverage data and compute to set off an intelligence explosion (...ok I just realized you wrote TAI but IDK what anyone means by anything other than actual AGI), doesn’t mean we know much about how they leverage it and how that influences the explody-guy’s long-term goals.
I assume that current efforts in AI evals and AI interpretability will be pretty useless if we have very different infrastructures in 10 years. For example, I’m not sure how much LLM interp helps with o1-style high-level reasoning.
I also think that later AI could help us do research. So if the idea is that we could do high-level strategic reasoning to find strategies that aren’t specific to specific models/architectures, I assume we could do that reasoning much better with better AI.
I agree that increasing duration has a greater impact than increasing funding. But increasing duration is harder than increasing funding.
AI safety spending is only $0.1 billion while AI capabilities spending is $200 billion. Increasing funding by 10x is relatively more attainable, while increasing duration by 10x would require more of a miracle.
Even if you believe that funding today isn’t very useful and funding in the future is more useful, increasing funding now moves the Overton window a lot. It’s hard for any government which has traditionally spent only $0.01 billion to suddenly spend $100 billion. They’ll use the previous budget as an anchor point to decide the new budget.
For inventive steps, having twice as many “inventors” reduces the time to invention by half, while for engineering steps, having twice as many “engineers” doesn’t help very much.
(Assuming the time it takes each inventor to think of an invention is an independent exponential distribution)
I’m not sure if it means much, but I’d be very happy if AI safety could get another $50B from smart donors today.
I’d flag that [stopping AI development] would cost far more than $50B. I’d expect that we could easily lose $3T of economic value in the next few years if AI progress seriously stopped.
I guess, it seems to me like duration is basically dramatically more expensive to get than funding, for amounts of funding people would likely want.
I do think that convincing the government to pause AI in a way which sacrifices $3000 billion economic value, is relatively easier than directly spending $3000 billion on AI safety.
Maybe spending $1 is similarly hard to sacrificing $10-$100 of future economic value via preemptive regulation.[1]
But $0.1 billion AI safety spending is so ridiculously little (1000 times less than capabilities spending), increasing it may still be the “easiest” thing to do. Of course we should still push for regulation at the same time (it doesn’t hurt).
PS: what do you think of my open letter idea for convincing the government to increase funding?
Maybe “future economic value” is too complicated. A simpler guesstimate would be “spending $1 is similarly hard to sacrificing $10 of company valuations via regulation.”
A bunch of people in the AI safety landscape seem to argue “we need to stop AI progress, so that we can make progress on AI safety first.”
One flip side to this is that I think it’s incredibly easy for people to waste a ton of resources on “AI safety” at this point.
I’m not sure how much I trust most technical AI safety researchers to make important progress on AI safety now. And I trust most institutions a lot less.
I’d naively expect if any major country would throw $100 Billion on it today, the results would be highly underwhelming. I rarely trust these governments to make progress on concrete technologies with clear progress measures, and “AI Safety” is highly ambiguous and speculative.
As I’ve written about before, I think it’s just hard to know what critical technical challenges will be bottlenecks around AI alignment, given that it’s unclear when this will become an issue or what sorts of architectures we will have then.
All that said, slowing things down seems much safer to me. I assume that at [year(TAI) − 3] we’ll have a decent idea of what’s needed, and extending that duration seems like a safe bet.
I really want to see better strategic discussion about AI safety. If somehow we could spend $10B just to get a better idea of what to actually do, I’d easily suggest that, though strategy is something that’s typically very difficult to spend money on.
Personally, this is one reason why I favor the meta approach of “make better epistemic tools, using AI.” This is an area that can be very concrete and achievable, though it does have its own problems.
What makes you think that we’re not at year(TAI)-3 right now? I’ll agree that we might not be there yet, but you seem to be assuming that we can’t be.
This is an orthogonal question. I agree that if we’re there now, my claim is much less true.
I’d place fairly little probability mass on this (<10%) and believe much of the rest of the community does as well, though I realize there is a subset of the LessWrong-adjacent community that does.
Why?? What happened to the bitter lesson?
Can you explain this position more? I know the bitter lesson, could imagine a few ways it could have implications here.
I’m saying that just because we know algorithms that will successfully leverage data and compute to set off an intelligence explosion (...ok I just realized you wrote TAI but IDK what anyone means by anything other than actual AGI), doesn’t mean we know much about how they leverage it and how that influences the explody-guy’s long-term goals.
I assume that current efforts in AI evals and AI interpretability will be pretty useless if we have very different infrastructures in 10 years. For example, I’m not sure how much LLM interp helps with o1-style high-level reasoning.
I also think that later AI could help us do research. So if the idea is that we could do high-level strategic reasoning to find strategies that aren’t specific to specific models/architectures, I assume we could do that reasoning much better with better AI.
I think both duration and funding are important.
I agree that increasing duration has a greater impact than increasing funding. But increasing duration is harder than increasing funding.
AI safety spending is only $0.1 billion while AI capabilities spending is $200 billion. Increasing funding by 10x is relatively more attainable, while increasing duration by 10x would require more of a miracle.
Even if you believe that funding today isn’t very useful and funding in the future is more useful, increasing funding now moves the Overton window a lot. It’s hard for any government which has traditionally spent only $0.01 billion to suddenly spend $100 billion. They’ll use the previous budget as an anchor point to decide the new budget.
My guess is that 4x funding ≈ 2x duration.[1]
For inventive steps, having twice as many “inventors” reduces the time to invention by half, while for engineering steps, having twice as many “engineers” doesn’t help very much.
(Assuming the time it takes each inventor to think of an invention is an independent exponential distribution)
I’m not sure if it means much, but I’d be very happy if AI safety could get another $50B from smart donors today.
I’d flag that [stopping AI development] would cost far more than $50B. I’d expect that we could easily lose $3T of economic value in the next few years if AI progress seriously stopped.
I guess, it seems to me like duration is basically dramatically more expensive to get than funding, for amounts of funding people would likely want.
I do think that convincing the government to pause AI in a way which sacrifices $3000 billion economic value, is relatively easier than directly spending $3000 billion on AI safety.
Maybe spending $1 is similarly hard to sacrificing $10-$100 of future economic value via preemptive regulation.[1]
But $0.1 billion AI safety spending is so ridiculously little (1000 times less than capabilities spending), increasing it may still be the “easiest” thing to do. Of course we should still push for regulation at the same time (it doesn’t hurt).
PS: what do you think of my open letter idea for convincing the government to increase funding?
Maybe “future economic value” is too complicated. A simpler guesstimate would be “spending $1 is similarly hard to sacrificing $10 of company valuations via regulation.”