There’s a cottage industry that thrives off of sneering, gawking, and maligning the AI safety community. This isn’t new, but it’s probably going to become more intense and pointed now that there are two giant super PACs that (allegedly[1]) see safety as a barrier to [innovation/profit, depending on your level of cynicism]. Brace for some nasty, uncharitable articles.
I think the largest cost of this targeted bad press will be the community’s overreaction, not the reputational effects outside the AI safety community. I’ve already seen people shy away from doing things like donating to politicians that support AI safety for fear of provoking the super PACs.
Historically, the safety community often freaked out in the face of this kind of bad press. People got really stressed out, pointed fingers about whose fault it was, and started to let the strong frames in the hit pieces get into their heads.[2] People disavowed AI safety and turned to more popular causes. And the collective consciousness decided that the actions and people who ushered in the mockery were obviously terrible and dumb, so much so that you’d get a strange look if you asked them to justify that argument. In reality I think many actions that were publicly ridiculed were still worth it ex-ante despite the bad press.
It seems bad press is often much, much more salient to the subjects of that press than it is to society at large, and it’s best to shrug it off and let it blow over. Some of the most PR-conscious people I know are weirdly calm during actual PR blowups and are sometimes more willing than the “weird” folks around me to take dramatic (but calculated) PR risks.
In the activist world, I hear this is a well-known phenomenon. You can get 10 people to protest a multi-billion-dollar company and a couple journalists to write articles, and the company will bend to your demands.[3] The rest of the world will have no idea who you are, but to the executives at the company, it will feel the world is watching them. These executives are probably making a mistake![4] Don’t be like them.
With all these (allegedly anti-safety[1]) super PACs, there will probably be a lot more bad press than usual. All else being equal, avoiding the bad press is good, but in order to fight back, people in the safety community will probably take some actions, and the super PACs will probably twist any actions into headlines about cringe doomer tech bros.
I do think people should take into account when deciding what to do that provoking the super PACs is risky, and should think carefully before doing it. But often I expect it will be the right choice and the blowback will be well worth it.
If people in the safety community refuse to stand up to them, then they super PACs will get what they want anyway and the safety community won’t even put up a fight.
Ultimately I think the AI safety community is an earnest, scrupulous group of people fighting for an extremely important cause. I hope we continue to hold ourselves to high standards for integrity and honor, and as long as we do, I will be proud to be part of this community no matter what the super PACs say.
They haven’t taken any anti-safety actions yet as far as I know (they’re still new). The picture they paint of themselves isn’t opposed to safety, and while I feel confident they will take actions I consider opposed to safety, I don’t like maligning people before they’ve actually taken actions worthy of condemnation.
I think it’s really healthy to ask yourself if you’re upholding your principles and acting ethically. But I find it a little suspicious how responsive some of these attitudes can be to bad press, where people often start tripping over themselves to distance themselves from whatever the journalist happened to dislike. If you’ve ever done this, consider asking yourself before you take any action how you’d feel if the fact that you took that action was on the front page of the papers. If you’d feel like you could hold your head up high, do it. Otherwise don’t. And then if you do end up on the front page of the papers, hold your head up high!
To a point. They won’t do things that would make them go out of business, but they might spend many millions of dollars on the practices you want them to adopt.
Tactically, that is. In many cases I’m glad the executives can be held responsible in this way and I think their changed behavior is better for the world.
I hope we continue to hold ourselves to high standards for integrity and honor, and as long as we do, I will be proud to be part of this community no matter what the super PACs say.
I don’t think the AI safety community has particularly much integrity or honor. I would like to make there be something in the space that has those attributes, but please don’t claim valor we/you don’t have!
For context, how would you rank the AI safety community w.r.t. integrity and honor, compared to the following groups:
1. AGI companies 2. Mainstream political parties (the organizations, not the voters, so e.g. the politicians and their staff) 3. Mainstream political movements e.g. neoliberalism, wokism, china hawks, BLM, 4. A typical university department 5. Elite opinion formers (e.g. the kind of people whose Substacks and op-eds are widely read and highly influential in DC, silicon valley, etc.) 6. A typical startup 7. A typical large bloated bureaucracy or corporation 8. A typical religion e.g. christianity, islam, etc. 9. The US military
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
A lot of that is the result of agency, which is often good, but in this case a double-edged sword. Naive consequentialism and lots of intense group-beliefs make the appropriate level of paranoia when interfacing with the AI Safety community higher than with most of these places.
“Appropriate levels of paranoia when interfacing with you” is of course not the only measure of honor and integrity, though as I am hoping to write about sometime this week, it’s kind of close to the top.
On that dimension, I think the AI Safety community is below AGI companies and the US military, and above all the other ones on this list. For the AGI companies, it’s unclear to me how much of it is the same generator. Approximately 50% of the AI Safety community are employed by AI labs, and they have historically made up a non-trivial fraction of the leadership of those companies, so those datapoints are highly correlated.
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
This is a wild claim. Don’t religions sort of centrally try to get you to believe known-to-be-false claims? Don’t politicians famously lie all the time?
Are you saying that EAs are better at deceiving people than typical members of those groups?
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Can you be more specific about the way in which you think AI Safety folk are worse?
Don’t religions sort of centrally try to get you to believe known-to-be-false claims?
I agree that institutionally they are set up to do a lot of that, but the force they bring to bear on any individual is actually quite small in my experience, compared to what I’ve seen in AI safety spaces. Definitely lots of heterogeneity here, but most optimization that religions do to actually keep you believing in their claims are pretty milquetoast.
Are you saying that EAs are better at deceiving people than typical members of those groups?
Definitely in-expectation! I think SBF, Sam Altman, Dario, Geoff Anders plus a bunch of others are pretty big outliers on these dimensions. I think in-practice there is a lot variance between individuals, with a very high-level gloss being something like “the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”.
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Yeah, that’s a good chunk of it. Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here. Some of it is a competence side-effect. Paranoia becomes worse the more competent your adversary is. The AI Safety community is a particularly scary adversary in that respect (and one that due to relatively broad buy-in for something like naive-consequentialism can bring more of its competence to bear on the task of deceiving you).
Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here.
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?[1]
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
(Is this the kind of thing you mean?)
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
Pretty much every day, I check “If someone had made this argument against [social media], would that have made me think that it was imperative to shut it down?”, about proffered anti AI arguments.
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?
Also, come on, this seems false. I am pretty sure you’ve seen Leverage employees do this, and my guess is you’ve seen transcripts of chats of this happening with quite a lot of agency at FTX with regards to various auditors and creditors.
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
Yeah, that does sound roughly like what I mean, and then I think most people just drop the second part:
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
I do not think that SBF was doing this part. He was doing the former though!
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
My best guess you are doing a mixture of:
Indeed self-selecting yourself out of these environments
Having a too-narrow conception of the “AI Safety community” that forms a Motte where you conceptually exclude people who do this a lot (e.g. the labs themselves), but in a way that then makes posts like the OP we are commenting on misleading
Probably have somewhat different standards for this (indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception).
indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here
Is this not common in politics? I thought this was a lot of what politics was about. (Having never worked in politics.)
Is this not common in politics? I thought this was a lot of what politics was about.
I have been very surprised by how non-agentic politics is! Like, there certainly is a lot of signaling going on, but when reading stuff like Decidingtowin.org it becomes clear how little optimization actually goes into saying things that will get you voters and convince stakeholders.
I do think a lot of that is going on there, and in the ranking above I would probably put the current political right above AI safety and the current political left below AI safety. Just when I took the average it seemed to me like it would end up below, largely as a result of a severe lack of agency as documented in things like deciding-to-win.
Re corporate campaigns: I think those are really very milquetoast. Yes, you make cool ads, but the optimization pressure here seems relatively minor (barring some intense outliers, like Apple and Disney, which I do think are much more agentic here than others, and have caused pretty great harm in doing so, like Disney being responsible for copyright being far too long in the US because Disney was terribly afraid of anyone re-using their characters and so tainting Disney’s image).
“the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”
Are you combining Venkatesh Rao’s loser/clueless/sociopath taxonomy with David Chapman’s geek/mop/sociopath?
(ETA: I know this is not relevant to the discussion, but I confuse these sometimes.)
Very low, though trending a bit higher over time. The policy-focused playbook has to deal with a lot more trickiness here than AI-2027, and you have to deal more with policymakers and stuff, but currently y’all don’t do very much of the kind of thing I am talking about here.
I really appreciate your clear-headedness at recognizing these phenomena even in people “on the same team”, i.e. people very concerned about and interested in preventing AI X-Risk.
However, I suspect that you also underrate the amount of self-deception going on here. It’s much easier to convince others if you convince yourself first. I think people in the AI Safety community self-deceive in various ways, for example by choosing to not fully think through how their beliefs are justified (e.g. not acknowledging the extent to which they are based on deference—Tsvi writes about this in his recent post rather well).
There are of course people who explicitly, consciously, plan to deceive, thinking things like “it’s very important to convince people that AI Safety/policy X is important, and so we should use the most effective messaging techniques possible, even if they use false or misleading claims.” However, I think there’s a larger set of people who, as they realize claims A B C are useful for consequentialist reasons, internally start questioning A B C less, and become biased to believe A B C themselves.
Sure! I definitely agree that’s going on a lot as well. But I think that kind of deception is more common in the rest of the world, and the things that set this community apart from others is the ability to do something more intentional here (which then combined with plenty of self-deception can result in quite catastrophic outcomes, as FTX illustrates).
I do think it’s not good! But also, it’s an important issue and you have to interface with people who aren’t super principled all the time. I just don’t want people to think of the AI Safety community as some kind of community of saints. I think it’s pretty high variance, and you should have your guard up a good amount.
For those that rely on intelligence enhancement as a component of their AI safety strategy, it would be a good time to get your press lines straight. The association of AI safety with eugenics (whether you personally agree with that label or not) strikes me as a soft target and a simple way to keep AI safety as a marginal movement.
I think a good counter to this from the activism perspective is avoiding labels and producing objective, thoughtful, and well-reasoned content arguing your point. Anti-AI-safety content often focuses on attacking the people or the specific beliefs of the people in the AI safety/rationalist community. The epistemic effects of these attacks can be circumvented by avoiding association with that community as much as is reasonable, without being deceptive. A good example would be the YouTube channel AI in Context run by 80,000 Hours. They made an excellent AI 2027 video, coming at it from an objective perspective and effectively connecting the dots from the seemingly fantastical scenario to reality. That video is now approaching 10 million views on a completely fresh channel! See also SciShows recent episode on AI, which also garnered extremely positive reception.
The strong viewership on this type of content demonstrates that people are clearly receptive to the AI safety narrative if it’s done tastefully and logically. Most of the negative comments on these videos (anecdotally) come from people who believe that superintelligent AI is either impossible or extremely distant, not that reject the premise altogether. In my view, content like this would be affected very weakly by the type of attacks you are talking about in this post. To be blunt, to oversimplify, and to take the risk of being overconfident, I believe safety and caution narratives have the advantage over acceleration narratives by merit of being based in reality and logic! Imagine attempting to make a “counter” to the above videos trying to make the case that safety is no big deal. How would you even go about that? Would people believe you? Arguments are not won by truth alone, but it certainly helps.
The potential political impact seems more salient, but in my (extremely inexpert) opinion getting the public on your side will cause political figures to follow. The measures required to meaningfully impact AI outcomes require so much political will that extremely strong public opinion is required, and that extremely strong public opinion comes from a combination of real world impact and evidence(“AI took my job”) along with properly communicating the potential future and dangers (Like the content above). The more the public is on the side of an AI slowdown, the less impact a super PAC can have on politicians decisions regarding the topic (compare a world where 2 percent of voters say they support a pause on AI development to a world where 70 percent say they support it. In world 1 a politician would be easily swayed to avoid the issue by the threat of adversarial spending, but in world 2 the political risk of avoiding the issue is far stronger than the risk of invoking the wrath of the super PAC). This is not meant to diminish the very real harm that organized opposition can cause politically, or to downplay the importance of countering that political maneuvering in turn. Political work is extremely important, and especially so if well funded groups are working to push the exact opposite narrative to what is needed.
I don’t mean to diminish the potential harm this kind of political maneuvering can have, but in my view the future is bright from the safety activism perspective. I’ll also add that I don’t believe my view of “avoid labels” and your point about “standing proud and putting up a fight” are opposed. Both can happen parallelly, two fights at once. I strongly agree that backing down from your views or actions as a result of bad press is a mistake, and I don’t advocate for that here.
There’s a cottage industry that thrives off of sneering, gawking, and maligning the AI safety community. This isn’t new, but it’s probably going to become more intense and pointed now that there are two giant super PACs that (allegedly) see safety as a barrier to [innovation/profit, depending on your level of cynicism]. Brace for some nasty, uncharitable articles.
One such article came out yesterday; I think it’s a fairly representative example of the genre.
There’s a cottage industry that thrives off of sneering, gawking, and maligning the AI safety community. This isn’t new, but it’s probably going to become more intense and pointed now that there are two giant super PACs that (allegedly[1]) see safety as a barrier to [innovation/profit, depending on your level of cynicism]. Brace for some nasty, uncharitable articles.
I think the largest cost of this targeted bad press will be the community’s overreaction, not the reputational effects outside the AI safety community. I’ve already seen people shy away from doing things like donating to politicians that support AI safety for fear of provoking the super PACs.
Historically, the safety community often freaked out in the face of this kind of bad press. People got really stressed out, pointed fingers about whose fault it was, and started to let the strong frames in the hit pieces get into their heads.[2] People disavowed AI safety and turned to more popular causes. And the collective consciousness decided that the actions and people who ushered in the mockery were obviously terrible and dumb, so much so that you’d get a strange look if you asked them to justify that argument. In reality I think many actions that were publicly ridiculed were still worth it ex-ante despite the bad press.
It seems bad press is often much, much more salient to the subjects of that press than it is to society at large, and it’s best to shrug it off and let it blow over. Some of the most PR-conscious people I know are weirdly calm during actual PR blowups and are sometimes more willing than the “weird” folks around me to take dramatic (but calculated) PR risks.
In the activist world, I hear this is a well-known phenomenon. You can get 10 people to protest a multi-billion-dollar company and a couple journalists to write articles, and the company will bend to your demands.[3] The rest of the world will have no idea who you are, but to the executives at the company, it will feel the world is watching them. These executives are probably making a mistake![4] Don’t be like them.
With all these (allegedly anti-safety[1]) super PACs, there will probably be a lot more bad press than usual. All else being equal, avoiding the bad press is good, but in order to fight back, people in the safety community will probably take some actions, and the super PACs will probably twist any actions into headlines about cringe doomer tech bros.
I do think people should take into account when deciding what to do that provoking the super PACs is risky, and should think carefully before doing it. But often I expect it will be the right choice and the blowback will be well worth it.
If people in the safety community refuse to stand up to them, then they super PACs will get what they want anyway and the safety community won’t even put up a fight.
Ultimately I think the AI safety community is an earnest, scrupulous group of people fighting for an extremely important cause. I hope we continue to hold ourselves to high standards for integrity and honor, and as long as we do, I will be proud to be part of this community no matter what the super PACs say.
They haven’t taken any anti-safety actions yet as far as I know (they’re still new). The picture they paint of themselves isn’t opposed to safety, and while I feel confident they will take actions I consider opposed to safety, I don’t like maligning people before they’ve actually taken actions worthy of condemnation.
I think it’s really healthy to ask yourself if you’re upholding your principles and acting ethically. But I find it a little suspicious how responsive some of these attitudes can be to bad press, where people often start tripping over themselves to distance themselves from whatever the journalist happened to dislike. If you’ve ever done this, consider asking yourself before you take any action how you’d feel if the fact that you took that action was on the front page of the papers. If you’d feel like you could hold your head up high, do it. Otherwise don’t. And then if you do end up on the front page of the papers, hold your head up high!
To a point. They won’t do things that would make them go out of business, but they might spend many millions of dollars on the practices you want them to adopt.
Tactically, that is. In many cases I’m glad the executives can be held responsible in this way and I think their changed behavior is better for the world.
I do wish this was the case, but as I have written many times in the past, I just don’t think this is an accurate characterization. See e.g.: https://www.lesswrong.com/posts/wn5jTrtKkhspshA4c/michaeldickens-s-shortform?commentId=zoBMvdMAwpjTEY4st
I don’t think the AI safety community has particularly much integrity or honor. I would like to make there be something in the space that has those attributes, but please don’t claim valor we/you don’t have!
For context, how would you rank the AI safety community w.r.t. integrity and honor, compared to the following groups:
1. AGI companies
2. Mainstream political parties (the organizations, not the voters, so e.g. the politicians and their staff)
3. Mainstream political movements e.g. neoliberalism, wokism, china hawks, BLM,
4. A typical university department
5. Elite opinion formers (e.g. the kind of people whose Substacks and op-eds are widely read and highly influential in DC, silicon valley, etc.)
6. A typical startup
7. A typical large bloated bureaucracy or corporation
8. A typical religion e.g. christianity, islam, etc.
9. The US military
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
A lot of that is the result of agency, which is often good, but in this case a double-edged sword. Naive consequentialism and lots of intense group-beliefs make the appropriate level of paranoia when interfacing with the AI Safety community higher than with most of these places.
“Appropriate levels of paranoia when interfacing with you” is of course not the only measure of honor and integrity, though as I am hoping to write about sometime this week, it’s kind of close to the top.
On that dimension, I think the AI Safety community is below AGI companies and the US military, and above all the other ones on this list. For the AGI companies, it’s unclear to me how much of it is the same generator. Approximately 50% of the AI Safety community are employed by AI labs, and they have historically made up a non-trivial fraction of the leadership of those companies, so those datapoints are highly correlated.
This is a wild claim. Don’t religions sort of centrally try to get you to believe known-to-be-false claims? Don’t politicians famously lie all the time?
Are you saying that EAs are better at deceiving people than typical members of those groups?
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Can you be more specific about the way in which you think AI Safety folk are worse?
I agree that institutionally they are set up to do a lot of that, but the force they bring to bear on any individual is actually quite small in my experience, compared to what I’ve seen in AI safety spaces. Definitely lots of heterogeneity here, but most optimization that religions do to actually keep you believing in their claims are pretty milquetoast.
Definitely in-expectation! I think SBF, Sam Altman, Dario, Geoff Anders plus a bunch of others are pretty big outliers on these dimensions. I think in-practice there is a lot variance between individuals, with a very high-level gloss being something like “the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”.
Yeah, that’s a good chunk of it. Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here. Some of it is a competence side-effect. Paranoia becomes worse the more competent your adversary is. The AI Safety community is a particularly scary adversary in that respect (and one that due to relatively broad buy-in for something like naive-consequentialism can bring more of its competence to bear on the task of deceiving you).
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?[1]
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
(Is this the kind of thing you mean?)
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
Pretty much every day, I check “If someone had made this argument against [social media], would that have made me think that it was imperative to shut it down?”, about proffered anti AI arguments.
Also, come on, this seems false. I am pretty sure you’ve seen Leverage employees do this, and my guess is you’ve seen transcripts of chats of this happening with quite a lot of agency at FTX with regards to various auditors and creditors.
Yeah, that does sound roughly like what I mean, and then I think most people just drop the second part:
I do not think that SBF was doing this part. He was doing the former though!
My best guess you are doing a mixture of:
Indeed self-selecting yourself out of these environments
Having a too-narrow conception of the “AI Safety community” that forms a Motte where you conceptually exclude people who do this a lot (e.g. the labs themselves), but in a way that then makes posts like the OP we are commenting on misleading
Probably have somewhat different standards for this (indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception).
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
Is this not common in politics? I thought this was a lot of what politics was about. (Having never worked in politics.)
And corporate PR campaigns too for that matter.
I have been very surprised by how non-agentic politics is! Like, there certainly is a lot of signaling going on, but when reading stuff like Decidingtowin.org it becomes clear how little optimization actually goes into saying things that will get you voters and convince stakeholders.
I do think a lot of that is going on there, and in the ranking above I would probably put the current political right above AI safety and the current political left below AI safety. Just when I took the average it seemed to me like it would end up below, largely as a result of a severe lack of agency as documented in things like deciding-to-win.
Re corporate campaigns: I think those are really very milquetoast. Yes, you make cool ads, but the optimization pressure here seems relatively minor (barring some intense outliers, like Apple and Disney, which I do think are much more agentic here than others, and have caused pretty great harm in doing so, like Disney being responsible for copyright being far too long in the US because Disney was terribly afraid of anyone re-using their characters and so tainting Disney’s image).
Are you combining Venkatesh Rao’s loser/clueless/sociopath taxonomy with David Chapman’s geek/mop/sociopath?
(ETA: I know this is not relevant to the discussion, but I confuse these sometimes.)
Oh, oops, yep, I confused the two. I meant geek/mop/sociopath in the David Chapman sense. Thank you for the catch!
Huh. Seems super wrong to me fwiw. How would you rank AIFP on the list?
Very low, though trending a bit higher over time. The policy-focused playbook has to deal with a lot more trickiness here than AI-2027, and you have to deal more with policymakers and stuff, but currently y’all don’t do very much of the kind of thing I am talking about here.
I really appreciate your clear-headedness at recognizing these phenomena even in people “on the same team”, i.e. people very concerned about and interested in preventing AI X-Risk.
However, I suspect that you also underrate the amount of self-deception going on here. It’s much easier to convince others if you convince yourself first. I think people in the AI Safety community self-deceive in various ways, for example by choosing to not fully think through how their beliefs are justified (e.g. not acknowledging the extent to which they are based on deference—Tsvi writes about this in his recent post rather well).
There are of course people who explicitly, consciously, plan to deceive, thinking things like “it’s very important to convince people that AI Safety/policy X is important, and so we should use the most effective messaging techniques possible, even if they use false or misleading claims.” However, I think there’s a larger set of people who, as they realize claims A B C are useful for consequentialist reasons, internally start questioning A B C less, and become biased to believe A B C themselves.
Sure! I definitely agree that’s going on a lot as well. But I think that kind of deception is more common in the rest of the world, and the things that set this community apart from others is the ability to do something more intentional here (which then combined with plenty of self-deception can result in quite catastrophic outcomes, as FTX illustrates).
This is not good. Why should people run the risk of interacting with the AI safety community if this is true?
I do think it’s not good! But also, it’s an important issue and you have to interface with people who aren’t super principled all the time. I just don’t want people to think of the AI Safety community as some kind of community of saints. I think it’s pretty high variance, and you should have your guard up a good amount.
For those that rely on intelligence enhancement as a component of their AI safety strategy, it would be a good time to get your press lines straight. The association of AI safety with eugenics (whether you personally agree with that label or not) strikes me as a soft target and a simple way to keep AI safety as a marginal movement.
I think a good counter to this from the activism perspective is avoiding labels and producing objective, thoughtful, and well-reasoned content arguing your point. Anti-AI-safety content often focuses on attacking the people or the specific beliefs of the people in the AI safety/rationalist community. The epistemic effects of these attacks can be circumvented by avoiding association with that community as much as is reasonable, without being deceptive. A good example would be the YouTube channel AI in Context run by 80,000 Hours. They made an excellent AI 2027 video, coming at it from an objective perspective and effectively connecting the dots from the seemingly fantastical scenario to reality. That video is now approaching 10 million views on a completely fresh channel! See also SciShows recent episode on AI, which also garnered extremely positive reception.
The strong viewership on this type of content demonstrates that people are clearly receptive to the AI safety narrative if it’s done tastefully and logically. Most of the negative comments on these videos (anecdotally) come from people who believe that superintelligent AI is either impossible or extremely distant, not that reject the premise altogether. In my view, content like this would be affected very weakly by the type of attacks you are talking about in this post. To be blunt, to oversimplify, and to take the risk of being overconfident, I believe safety and caution narratives have the advantage over acceleration narratives by merit of being based in reality and logic! Imagine attempting to make a “counter” to the above videos trying to make the case that safety is no big deal. How would you even go about that? Would people believe you? Arguments are not won by truth alone, but it certainly helps.
The potential political impact seems more salient, but in my (extremely inexpert) opinion getting the public on your side will cause political figures to follow. The measures required to meaningfully impact AI outcomes require so much political will that extremely strong public opinion is required, and that extremely strong public opinion comes from a combination of real world impact and evidence(“AI took my job”) along with properly communicating the potential future and dangers (Like the content above). The more the public is on the side of an AI slowdown, the less impact a super PAC can have on politicians decisions regarding the topic (compare a world where 2 percent of voters say they support a pause on AI development to a world where 70 percent say they support it. In world 1 a politician would be easily swayed to avoid the issue by the threat of adversarial spending, but in world 2 the political risk of avoiding the issue is far stronger than the risk of invoking the wrath of the super PAC). This is not meant to diminish the very real harm that organized opposition can cause politically, or to downplay the importance of countering that political maneuvering in turn. Political work is extremely important, and especially so if well funded groups are working to push the exact opposite narrative to what is needed.
I don’t mean to diminish the potential harm this kind of political maneuvering can have, but in my view the future is bright from the safety activism perspective. I’ll also add that I don’t believe my view of “avoid labels” and your point about “standing proud and putting up a fight” are opposed. Both can happen parallelly, two fights at once. I strongly agree that backing down from your views or actions as a result of bad press is a mistake, and I don’t advocate for that here.
One such article came out yesterday; I think it’s a fairly representative example of the genre.