Eighth, yes, working in AI capabilities is absolutely a reasonable alignment plan that raises odds of success immensely. I know, you’re so overconfident on this point that even reading this will trigger you to dismiss my comment. And yet it’s still true—and what’s more, obviously so. I don’t know how you and others egged each other into the position that it doesn’t matter whether the people working on AI care about AI risk, but it’s insane.
I agreed with most of your comment until this line. Is your argument that, there’s a lot of nuance to getting safety right, we’re plausibly in a world where alignment is hard but possible, and the makers of AGI deeply caring about alignment, being cautious and not racing, etc, could push us over the line of getting alignment to work? I think this argument seems pretty reasonable, but that you’re overstating the case here and that this strategy could easily be net bad if you advance capabilities a lot. And “alignment is basically impossible unless something dramatically changes” also seems like a defensible position to me
I don’t expect most people to agree with that point, but I do believe it. It ends up depending on a lot of premises, so expanding on my view there in full would be a whole post of its own. But to try to give a short version:
There are a lot of specific reasons I think having people working in AI capabilities is so strongly +EV. But I don’t expect people to agree with those specific views. The reason I think it’s obvious is that even when I make massive concessions to the anti-capabilities people, these organizations… still seem +EV? Let’s make a bunch of concessions:
1. Alignment will be solved by theoretical work unrelated to capabilities. It can be done just as well at an alignment-only organization with limited funding as it can at a major AGI org with far more funding.
2. If alignment is solved, that automatically means future ASI will be built using this alignment technique, regardless of whether leading AI orgs actually care about alignment at all. You just publish a paper saying “alignment solution, pls use this Meta” and Meta will definitely do it.
3. Alignment will take a significant amount of time—probably decades.
4. ASI is now imminent; these orgs have reduced timelines to ASI by 1-5 years.
5. Our best chance of survival is a total stop, which none of the CEOs of these orgs support.
Even given all five of these premises… Demis Hassabis, Dario Amodei, and Sam Altman have all increased the chance of a total stop, by a lot. By more than almost anyone else on the planet, in fact. Yes, even though they don’t think it’s a good idea right now and have said as much (I think? haven’t followed all of their statements on AI pause).
That is, the chance of a total stop is clearly higher in this world than in the counterfactual one where any of Demis/Dario/Sam didn’t go into AI capabilities, because a CEO of a leading AI organization saying “yeah I think AI could maybe kill us all” is something that by default would not happen. As I said before, most people in the field of AI don’t take AI risk seriously; this was even more true back when they first entered the field. The default scenario is one where people at NVIDIA and Google Brain and Meta are reassuring the public that AI risk isn’t real.
So in other words, they are still increasing our chances of survival, even under that incredibly uncharitable set of assumptions.
Of course, you could cook these assumptions even more in order to make them -EV—if you think that a total stop isn’t feasible, but still believe all of the other four premises, then they’re -EV. Or you could say “yeah, we need a total stop now, because they’ve advanced timelines, but if these orgs didn’t exist then we totally would have solved alignment before Meta made a big transformer model and trained it on a lot of text; so even though they’ve raised the chances of a total stop they’re still a net negative.” Or you could say “the real counterfactual about Sam Altman isn’t if he didn’t enter the field. The real counterfactual is the one where he totally agreed with all of my incredibly specific views and acted based on those.”
I.e. if you’re looking for excuses to be allowed to believe that these orgs are bad, you’ll find them. But that’s always the case. Under real worldviews—even under Connor’s worldview, where he thinks a total stop is both plausible and necessary—OAI/DM/Anthropic are all helping with AI risk. Which means that their beneficiality is incredibly robust, because again, I think many of the assumptions I outlined above are false & incredibly uncharitable to AGI orgs.
That is, the chance of a total stop is clearly higher in this world than in the counterfactual one where any of Demis/Dario/Sam didn’t go into AI capabilities, because a CEO of a leading AI organization saying “yeah I think AI could maybe kill us all” is something that by default would not happen. As I said before, most people in the field of AI don’t take AI risk seriously; this was even more true back when they first entered the field. The default scenario is one where people at NVIDIA and Google Brain and Meta are reassuring the public that AI risk isn’t real.
I have the impression that the big guys started taking AI risk seriously when they saw capabilities that impressed them. So I expect that if Musk, Altman & the rest of the Dreamgrove did not embark in pushing the frontier faster than it was moving otherwise, at the same capability level AI researchers would have taken it seriously the same. Famous AI scientists already knew about the AI risk arguments; where OpenAI made a difference was not in telling them about AI risk, but shoving GPT up their nose.
I think the public would then have been able to side with Distinguished Serious People raising warnings about the dangers of ultra-intellingent machines even if Big Corp claimed otherwise.
I agreed with most of your comment until this line. Is your argument that, there’s a lot of nuance to getting safety right, we’re plausibly in a world where alignment is hard but possible, and the makers of AGI deeply caring about alignment, being cautious and not racing, etc, could push us over the line of getting alignment to work? I think this argument seems pretty reasonable, but that you’re overstating the case here and that this strategy could easily be net bad if you advance capabilities a lot. And “alignment is basically impossible unless something dramatically changes” also seems like a defensible position to me
I don’t expect most people to agree with that point, but I do believe it. It ends up depending on a lot of premises, so expanding on my view there in full would be a whole post of its own. But to try to give a short version:
There are a lot of specific reasons I think having people working in AI capabilities is so strongly +EV. But I don’t expect people to agree with those specific views. The reason I think it’s obvious is that even when I make massive concessions to the anti-capabilities people, these organizations… still seem +EV? Let’s make a bunch of concessions:
1. Alignment will be solved by theoretical work unrelated to capabilities. It can be done just as well at an alignment-only organization with limited funding as it can at a major AGI org with far more funding.
2. If alignment is solved, that automatically means future ASI will be built using this alignment technique, regardless of whether leading AI orgs actually care about alignment at all. You just publish a paper saying “alignment solution, pls use this Meta” and Meta will definitely do it.
3. Alignment will take a significant amount of time—probably decades.
4. ASI is now imminent; these orgs have reduced timelines to ASI by 1-5 years.
5. Our best chance of survival is a total stop, which none of the CEOs of these orgs support.
Even given all five of these premises… Demis Hassabis, Dario Amodei, and Sam Altman have all increased the chance of a total stop, by a lot. By more than almost anyone else on the planet, in fact. Yes, even though they don’t think it’s a good idea right now and have said as much (I think? haven’t followed all of their statements on AI pause).
That is, the chance of a total stop is clearly higher in this world than in the counterfactual one where any of Demis/Dario/Sam didn’t go into AI capabilities, because a CEO of a leading AI organization saying “yeah I think AI could maybe kill us all” is something that by default would not happen. As I said before, most people in the field of AI don’t take AI risk seriously; this was even more true back when they first entered the field. The default scenario is one where people at NVIDIA and Google Brain and Meta are reassuring the public that AI risk isn’t real.
So in other words, they are still increasing our chances of survival, even under that incredibly uncharitable set of assumptions.
Of course, you could cook these assumptions even more in order to make them -EV—if you think that a total stop isn’t feasible, but still believe all of the other four premises, then they’re -EV. Or you could say “yeah, we need a total stop now, because they’ve advanced timelines, but if these orgs didn’t exist then we totally would have solved alignment before Meta made a big transformer model and trained it on a lot of text; so even though they’ve raised the chances of a total stop they’re still a net negative.” Or you could say “the real counterfactual about Sam Altman isn’t if he didn’t enter the field. The real counterfactual is the one where he totally agreed with all of my incredibly specific views and acted based on those.”
I.e. if you’re looking for excuses to be allowed to believe that these orgs are bad, you’ll find them. But that’s always the case. Under real worldviews—even under Connor’s worldview, where he thinks a total stop is both plausible and necessary—OAI/DM/Anthropic are all helping with AI risk. Which means that their beneficiality is incredibly robust, because again, I think many of the assumptions I outlined above are false & incredibly uncharitable to AGI orgs.
I have the impression that the big guys started taking AI risk seriously when they saw capabilities that impressed them. So I expect that if Musk, Altman & the rest of the Dreamgrove did not embark in pushing the frontier faster than it was moving otherwise, at the same capability level AI researchers would have taken it seriously the same. Famous AI scientists already knew about the AI risk arguments; where OpenAI made a difference was not in telling them about AI risk, but shoving GPT up their nose.
I think the public would then have been able to side with Distinguished Serious People raising warnings about the dangers of ultra-intellingent machines even if Big Corp claimed otherwise.