I notice that there is this idea among AI safety people that conditional on AIs not being misaligned, building superintelligence is a public good and is a pretty exciting prospect.
This is not how many average people in the US feel. I was describing to an older family member that Anthropic focuses on code because they are trying to build a claude that can build a smarter claude which can build a smarter claude which can …
The reaction to this prospect was disgust, not because he intuitively felt AIs would likely be misaligned. It was more like on a gut level this amount of “playing god” felt totally antisocial and demonic and in general not respectful of an intuitive taboo against divine transgression (see Jurassic Park, Frankenstein, the recent popularization of Oppenheimer, Tower of Babel, etc).
Seems important to consider that people can feel this way when communicating with the public or policy makers.
I think most peoples’ revealed preferences will not match their stated preferences here, and revealed preferences are likely to dictate policy.
Conditional on AIs not taking over / until they do, AI is likely to generate a huge economic boom and consumer surplus that increases prosperity, safety, and comfort for many. Yes, there will be some bumps / weirdness / adjustment / inequality, etc., but economic growth can paper over a lot of that. And even if it brings new societal ills and discomfort, AI in the short term is likely to counteract some existing ones—great stagnation, bureaucratic strangulation, vehicle deaths, etc.
The public cannot even bring itself to regulate much less economically useful vices with fewer tradeoffs (online sports gambling, shortform video brainrot, etc.). So I think it’s unlikely that AI will be regulated for any reason short of it becoming common knowledge / deeply-felt belief by nation-state leadership that ASI is in fact likely to cause swift and total human extinction.
I also think a hands-off approach to AI regulation is not going to be particularly off-putting or seem weird or antisocial to a large fraction of the country, namely the ~40% of the country that is right-leaning, and will tend to follow the beliefs of elite republicans who (mostly) still favor a light touch when it comes to any kind of government regulation. Regulating AI for any reason other than extinction is likely to become a relatively standard partisan issue, if it isn’t already.
On the other hand, I would say that we do have examples of the public preferring policies that minimise bumps/weirdness/adjustments over economic growth; most obviously limits on construction/NIMBYism, but also around novel technologies like nuclear energy, GMO crops, mRNA vaccines and self-driving cars. Depending on your views on the economic impacts of migration (I think in practice it’s been less clearly beneficial than many consider it to be in theory), you could add that to the list too.
This is not to say that there won’t be a laissez-faire approach but there is precedent for the public forgoing economic growth for other priorities.
NB: I live in the UK where I think this is more true than in the US, but the examples I gave seem to apply to both countries (and the mRNA example is US-specific).
I think it’s somewhat likely (and quite bad) that various patchwork regulations will be passed in some jurisdictions that limit the benefits and dispersion of AI in the name of protecting jobs or whatever, or various things will already be illegal under current law and AI won’t change that one way or the other, e.g. AI will make it easier / cheaper to build a house or a nuclear power plant, but it will still be illegal to do so in most places that you’d want to.
OTOH I think it’s unlikely (but would be good), if there were a national (and eventually global) moratorium on frontier training runs and research.
In practice I think what is actually on track to happen is that the former kind of regulation will drive more resources (GPUs, human capital, etc.) away from inference and applications and towards research.
(It would be a sad choice, but I would trade away self-driving cars to stave off unaligned ASI. In practice though I think a lot of AI safety advocacy, especially outside of LW, is not actually offering that trade, and will in practice bring about something close to the opposite.)
AI’s not entirely separable from the infrastructure it runs on and companies managing it, and infrastructure and companies do get heavily regulated, even when that has massive negative consequences for economic growth. New York just implemented a moratorium on data centers. We’ve banned AI chip exports. Anthropic was declared a supply chain risk. Every form of energy, from nuclear, to coal, to oil, to solar/wind, now has a potent opposition group.
Overall, it’s seemed to me that software has seen surprisingly little direct regulation, especially when it stays in the consumer category. But the hardware it runs on gets intensively regulated, and software that’s classified as having military applications has also historically been on the receiving end of a lot of restrictions. Since AI is extremely reliant on heavy-duty hardware and also has clear military applications, it will tend to inherit those regulations.
So it will be about the preferences of the many for an economic boom over the NIMBY anti-infrastructure impulse, and we’ve seen the latter win consistently for some time now. I’m not sure how to think about the military angle. As a layman, it seems like we deal with this by having separate markets for military and consumer products, and most military products just don’t have a civilian application so civilians don’t lose much by being denied access to military technology. But if sufficiently powerful AI is intrinsically difficult to guard against being used for destructive purposes by civilians (similar to, say, explosives that do have legitimate economic applications), then we may see the more powerful models being available to civilians only with some combination of safeguards and monitoring that make regulators feel comfortable. And in the limit that probably looks like a hard ceiling on the amount of intelligence being vended to consumers, or regulators surrendering the problem of doling it out to a governing AI that’s entrusted with the problem of figuring out how to give humans access to a lower level of superhuman intelligence that’s still within the power of the governing AI to reliably predict and control.
My guess is that past a certain point, AI becomes able to satisfy “revealed preferences” trivially and cheaply, without actually requiring an unlimited amount of intelligence or AI infrastructure. That might look something like wireheading, or it might look like utopia. Then you get into a bifurcated world where some agents are pursuing goals that genuinely require ever-higher-levels of intelligence, while many people are pursuing traditional consumer goals that are satisfiable with a far less than frontier model. I’m skeptical democracy remains in recognizeable format in this scenario, but if it did, I can’t see ordinary people much caring about how intelligence gets regulated, since they’d still be getting their immediate consumer desires met even if the ultra-intelligence was restricted by the power players.
Aren’t those being widely / effectively circumvented though?
So it will be about the preferences of the many for an economic boom over the NIMBY anti-infrastructure impulse, and we’ve seen the latter win consistently for some time now.
The state-level moratoriums / NIMBYism are exactly the kind of thing I expect not to have their intended effect because they will be patchwork; red states are likely to embrace both the demand and supply side of the AI boom, e.g. by building lots of data centers and rolling out Waymos quickly.
Aren’t those being widely / effectively circumvented though?
To some extent, but overall they’re still having a very large impact, despite that leakage. Anyway, regardless of whether they’re well enforced, it is still a fact that they happened, and I take OP’s point to be more along the lines of “here’s an example of the US government doing a big thing related to AI”.
Disagree voted because, come on, everyone who plays god says the exact same thing. The whole point is that we are not gods and won’t be able to understand the consequences of our actions before they play out; in fact, this very assertion underpins the entire project of AI safety!
actually, we played god many times in the past with positive results. for example:
eradicating smallpox
synthetic fertilizer and high yield crops
vaccines
water sanitation
antibiotics
surgery
genetic engineering
of course, sometimes it goes poorly too. but when it goes well it goes really well; I’m exceedingly grateful that i’m unlikely to ever die from smallpox or cholera or tetanus or appendicitis or starvation. let’s work on playing god well.
Fair enough, and I’m also grateful for those advances. I would just like to push towards good things without explicitly claiming the “playing god” mantle, which seems inherently arrogant to me; I would feel much better about someone who is trying to cure cancer if they didn’t say things like “playing god is good if you are good at it” and instead said things like “I want to advance human knowledge and try to make our lives better by defeating this disease.”
Look. I’m the last person who’s going to deny that the road we’re on is littered with the skulls of the people who tried to do this before us. But we’ve noticed the skulls. We’ve looked at the creepy skull pyramids and thought “huh, better try to do the opposite of what those guys did”. Just as the best doctors are humbled by the history of murderous blood-letting, the best leftists are humbled by the history of Soviet authoritarianism, and the best generals are humbled by the history of Vietnam and Iraq and Libya and all the others – in exactly this way, the rationalist movement hasn’t missed the concerns that everybody who thinks of the idea of a “rationalist movement” for five seconds has come up with. If you have this sort of concern, and you want to accuse us of it, please do a quick Google search to make sure that everybody hasn’t been condemning it and promising not to do it since the beginning.
We’re almost certainly still making horrendous mistakes that people thirty years from now will rightly criticize us for. But they’re new mistakes. They’re original and exciting mistakes which are not the same mistakes everybody who hears the word “rational” immediately knows to check for and try to avoid. Or at worst, they’re the sort of Hofstadter’s Law-esque mistakes that are impossible to avoid by knowing about and compensating for them.
And I hope that maybe having a community dedicated to carefully checking its own thought processes and trying to minimize error in every way possible will make us have slightly fewer horrendous mistakes than people who don’t do that. I hope that constant vigilance has given us at least a tiny bit of a leg up, in the determining-what-is-true field, compared to people who think this is unnecessary and truth-seeking is a waste of time.
And how do you know if you are good at it? The argument in this regard appears to be approximately “my brain has judged that my brain is better than your brain at playing god. Therefore, I am good at playing god. Therefore, I should play god.”
Strong agree. Working on unconventional topics naturally places you in an epistemic bubble, which is probably fine most of the time but pretty bad if you’re working on politics/advocacy; I personally shy away from talking about AI with my non-AI-pilled friends just because I expect them to react weirdly, but to be honest, that’s probably why I should go out of my way to do it more.
I’m not sure you’re drawing a clear distinction there. Stories like Frankenstein etc. revolve around the idea that you shouldn’t mess around with powerful forces you don’t truly understand, because it’s liable to backfire terribly. And also that people would do it anyway, in their pursuit of money, power and fame. Both are observations that are wise in general, and accurate in the context of AI in particular. “Misalignment” is just our technical STEM-nerd framing of what it looks like when things backfire terribly in this context.
I actually am more saying that this is a way people think and less saying everyone should adopt this. My perspective is that don’t-do-divine-transgression is a good starting-point/prior and should require a lot of activation energy to break it, but I don’t think its an absolute moral rule.
I will say I don’t think this is entirely correct:
“Misalignment” is just our technical STEM-nerd framing of what it looks like when things backfire terribly in this context.
There are other underrated failure modes like that we die in the window where models are good enough at bio to do bioterrorism but not good enough at bio to stop bio terrorism. There also may be a window where open source models are good enough at bio to build a bioweapon while closed source models are still not good enough to reliably stop this.
There are so many different possible ways things can backfire. I don’t think that some people reason about this kind of thing very much and in general I wish those people indexed stronger on the don’t-do-divine-transgression prior .
I notice that there is this idea among AI safety people that conditional on AIs not being misaligned, building superintelligence is a public good and is a pretty exciting prospect.
This is not how many average people in the US feel. I was describing to an older family member that Anthropic focuses on code because they are trying to build a claude that can build a smarter claude which can build a smarter claude which can …
The reaction to this prospect was disgust, not because he intuitively felt AIs would likely be misaligned. It was more like on a gut level this amount of “playing god” felt totally antisocial and demonic and in general not respectful of an intuitive taboo against divine transgression (see Jurassic Park, Frankenstein, the recent popularization of Oppenheimer, Tower of Babel, etc).
Seems important to consider that people can feel this way when communicating with the public or policy makers.
I think most peoples’ revealed preferences will not match their stated preferences here, and revealed preferences are likely to dictate policy.
Conditional on AIs not taking over / until they do, AI is likely to generate a huge economic boom and consumer surplus that increases prosperity, safety, and comfort for many. Yes, there will be some bumps / weirdness / adjustment / inequality, etc., but economic growth can paper over a lot of that. And even if it brings new societal ills and discomfort, AI in the short term is likely to counteract some existing ones—great stagnation, bureaucratic strangulation, vehicle deaths, etc.
The public cannot even bring itself to regulate much less economically useful vices with fewer tradeoffs (online sports gambling, shortform video brainrot, etc.). So I think it’s unlikely that AI will be regulated for any reason short of it becoming common knowledge / deeply-felt belief by nation-state leadership that ASI is in fact likely to cause swift and total human extinction.
I also think a hands-off approach to AI regulation is not going to be particularly off-putting or seem weird or antisocial to a large fraction of the country, namely the ~40% of the country that is right-leaning, and will tend to follow the beliefs of elite republicans who (mostly) still favor a light touch when it comes to any kind of government regulation. Regulating AI for any reason other than extinction is likely to become a relatively standard partisan issue, if it isn’t already.
On the other hand, I would say that we do have examples of the public preferring policies that minimise bumps/weirdness/adjustments over economic growth; most obviously limits on construction/NIMBYism, but also around novel technologies like nuclear energy, GMO crops, mRNA vaccines and self-driving cars. Depending on your views on the economic impacts of migration (I think in practice it’s been less clearly beneficial than many consider it to be in theory), you could add that to the list too.
This is not to say that there won’t be a laissez-faire approach but there is precedent for the public forgoing economic growth for other priorities.
NB: I live in the UK where I think this is more true than in the US, but the examples I gave seem to apply to both countries (and the mRNA example is US-specific).
I think it’s somewhat likely (and quite bad) that various patchwork regulations will be passed in some jurisdictions that limit the benefits and dispersion of AI in the name of protecting jobs or whatever, or various things will already be illegal under current law and AI won’t change that one way or the other, e.g. AI will make it easier / cheaper to build a house or a nuclear power plant, but it will still be illegal to do so in most places that you’d want to.
OTOH I think it’s unlikely (but would be good), if there were a national (and eventually global) moratorium on frontier training runs and research.
In practice I think what is actually on track to happen is that the former kind of regulation will drive more resources (GPUs, human capital, etc.) away from inference and applications and towards research.
(It would be a sad choice, but I would trade away self-driving cars to stave off unaligned ASI. In practice though I think a lot of AI safety advocacy, especially outside of LW, is not actually offering that trade, and will in practice bring about something close to the opposite.)
AI’s not entirely separable from the infrastructure it runs on and companies managing it, and infrastructure and companies do get heavily regulated, even when that has massive negative consequences for economic growth. New York just implemented a moratorium on data centers. We’ve banned AI chip exports. Anthropic was declared a supply chain risk. Every form of energy, from nuclear, to coal, to oil, to solar/wind, now has a potent opposition group.
Overall, it’s seemed to me that software has seen surprisingly little direct regulation, especially when it stays in the consumer category. But the hardware it runs on gets intensively regulated, and software that’s classified as having military applications has also historically been on the receiving end of a lot of restrictions. Since AI is extremely reliant on heavy-duty hardware and also has clear military applications, it will tend to inherit those regulations.
So it will be about the preferences of the many for an economic boom over the NIMBY anti-infrastructure impulse, and we’ve seen the latter win consistently for some time now. I’m not sure how to think about the military angle. As a layman, it seems like we deal with this by having separate markets for military and consumer products, and most military products just don’t have a civilian application so civilians don’t lose much by being denied access to military technology. But if sufficiently powerful AI is intrinsically difficult to guard against being used for destructive purposes by civilians (similar to, say, explosives that do have legitimate economic applications), then we may see the more powerful models being available to civilians only with some combination of safeguards and monitoring that make regulators feel comfortable. And in the limit that probably looks like a hard ceiling on the amount of intelligence being vended to consumers, or regulators surrendering the problem of doling it out to a governing AI that’s entrusted with the problem of figuring out how to give humans access to a lower level of superhuman intelligence that’s still within the power of the governing AI to reliably predict and control.
My guess is that past a certain point, AI becomes able to satisfy “revealed preferences” trivially and cheaply, without actually requiring an unlimited amount of intelligence or AI infrastructure. That might look something like wireheading, or it might look like utopia. Then you get into a bifurcated world where some agents are pursuing goals that genuinely require ever-higher-levels of intelligence, while many people are pursuing traditional consumer goals that are satisfiable with a far less than frontier model. I’m skeptical democracy remains in recognizeable format in this scenario, but if it did, I can’t see ordinary people much caring about how intelligence gets regulated, since they’d still be getting their immediate consumer desires met even if the ultra-intelligence was restricted by the power players.
Aren’t those being widely / effectively circumvented though?
The state-level moratoriums / NIMBYism are exactly the kind of thing I expect not to have their intended effect because they will be patchwork; red states are likely to embrace both the demand and supply side of the AI boom, e.g. by building lots of data centers and rolling out Waymos quickly.
To some extent, but overall they’re still having a very large impact, despite that leakage. Anyway, regardless of whether they’re well enforced, it is still a fact that they happened, and I take OP’s point to be more along the lines of “here’s an example of the US government doing a big thing related to AI”.
i bite the bullet and say, yes, playing god is good if you are good at it. i’m fully aware that this is an unpopular opinion broadly.
Upvoted for the honesty.
Disagree voted because, come on, everyone who plays god says the exact same thing. The whole point is that we are not gods and won’t be able to understand the consequences of our actions before they play out; in fact, this very assertion underpins the entire project of AI safety!
actually, we played god many times in the past with positive results. for example:
eradicating smallpox
synthetic fertilizer and high yield crops
vaccines
water sanitation
antibiotics
surgery
genetic engineering
of course, sometimes it goes poorly too. but when it goes well it goes really well; I’m exceedingly grateful that i’m unlikely to ever die from smallpox or cholera or tetanus or appendicitis or starvation. let’s work on playing god well.
Fair enough, and I’m also grateful for those advances. I would just like to push towards good things without explicitly claiming the “playing god” mantle, which seems inherently arrogant to me; I would feel much better about someone who is trying to cure cancer if they didn’t say things like “playing god is good if you are good at it” and instead said things like “I want to advance human knowledge and try to make our lives better by defeating this disease.”
Related:
And how do you know if you are good at it? The argument in this regard appears to be approximately “my brain has judged that my brain is better than your brain at playing god. Therefore, I am good at playing god. Therefore, I should play god.”
Strong agree. Working on unconventional topics naturally places you in an epistemic bubble, which is probably fine most of the time but pretty bad if you’re working on politics/advocacy; I personally shy away from talking about AI with my non-AI-pilled friends just because I expect them to react weirdly, but to be honest, that’s probably why I should go out of my way to do it more.
Haldane said it well:
What does “aligned” mean to the people answering these questions?
Nothing yet. But I will say there is an increasing amount of AI safety stuff in more mainstream news.
I’m not sure you’re drawing a clear distinction there. Stories like Frankenstein etc. revolve around the idea that you shouldn’t mess around with powerful forces you don’t truly understand, because it’s liable to backfire terribly. And also that people would do it anyway, in their pursuit of money, power and fame. Both are observations that are wise in general, and accurate in the context of AI in particular. “Misalignment” is just our technical STEM-nerd framing of what it looks like when things backfire terribly in this context.
I actually am more saying that this is a way people think and less saying everyone should adopt this. My perspective is that don’t-do-divine-transgression is a good starting-point/prior and should require a lot of activation energy to break it, but I don’t think its an absolute moral rule.
I will say I don’t think this is entirely correct:
There are other underrated failure modes like that we die in the window where models are good enough at bio to do bioterrorism but not good enough at bio to stop bio terrorism. There also may be a window where open source models are good enough at bio to build a bioweapon while closed source models are still not good enough to reliably stop this.
There are so many different possible ways things can backfire. I don’t think that some people reason about this kind of thing very much and in general I wish those people indexed stronger on the don’t-do-divine-transgression prior .