My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
A lot of that is the result of agency, which is often good, but in this case a double-edged sword. Naive consequentialism and lots of intense group-beliefs make the appropriate level of paranoia when interfacing with the AI Safety community higher than with most of these places.
“Appropriate levels of paranoia when interfacing with you” is of course not the only measure of honor and integrity, though as I am hoping to write about sometime this week, it’s kind of close to the top.
On that dimension, I think the AI Safety community is below AGI companies and the US military, and above all the other ones on this list. For the AGI companies, it’s unclear to me how much of it is the same generator. Approximately 50% of the AI Safety community are employed by AI labs, and they have historically made up a non-trivial fraction of the leadership of those companies, so those datapoints are highly correlated.
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
This is a wild claim. Don’t religions sort of centrally try to get you to believe known-to-be-false claims? Don’t politicians famously lie all the time?
Are you saying that EAs are better at deceiving people than typical members of those groups?
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Can you be more specific about the way in which you think AI Safety folk are worse?
Don’t religions sort of centrally try to get you to believe known-to-be-false claims?
I agree that institutionally they are set up to do a lot of that, but the force they bring to bear on any individual is actually quite small in my experience, compared to what I’ve seen in AI safety spaces. Definitely lots of heterogeneity here, but most optimization that religions do to actually keep you believing in their claims are pretty milquetoast.
Are you saying that EAs are better at deceiving people than typical members of those groups?
Definitely in-expectation! I think SBF, Sam Altman, Dario, Geoff Anders plus a bunch of others are pretty big outliers on these dimensions. I think in-practice there is a lot variance between individuals, with a very high-level gloss being something like “the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”.
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Yeah, that’s a good chunk of it. Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here. Some of it is a competence side-effect. Paranoia becomes worse the more competent your adversary is. The AI Safety community is a particularly scary adversary in that respect (and one that due to relatively broad buy-in for something like naive-consequentialism can bring more of its competence to bear on the task of deceiving you).
Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here.
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?[1]
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
(Is this the kind of thing you mean?)
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
Pretty much every day, I check “If someone had made this argument against [social media], would that have made me think that it was imperative to shut it down?”, about proffered anti AI arguments.
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?
Also, come on, this seems false. I am pretty sure you’ve seen Leverage employees do this, and my guess is you’ve seen transcripts of chats of this happening with quite a lot of agency at FTX with regards to various auditors and creditors.
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
Yeah, that does sound roughly like what I mean, and then I think most people just drop the second part:
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
I do not think that SBF was doing this part. He was doing the former though!
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
My best guess you are doing a mixture of:
Indeed self-selecting yourself out of these environments
Having a too-narrow conception of the “AI Safety community” that forms a Motte where you conceptually exclude people who do this a lot (e.g. the labs themselves), but in a way that then makes posts like the OP we are commenting on misleading
Probably have somewhat different standards for this (indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception).
indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here
Is this not common in politics? I thought this was a lot of what politics was about. (Having never worked in politics.)
Is this not common in politics? I thought this was a lot of what politics was about.
I have been very surprised by how non-agentic politics is! Like, there certainly is a lot of signaling going on, but when reading stuff like Decidingtowin.org it becomes clear how little optimization actually goes into saying things that will get you voters and convince stakeholders.
I do think a lot of that is going on there, and in the ranking above I would probably put the current political right above AI safety and the current political left below AI safety. Just when I took the average it seemed to me like it would end up below, largely as a result of a severe lack of agency as documented in things like deciding-to-win.
Re corporate campaigns: I think those are really very milquetoast. Yes, you make cool ads, but the optimization pressure here seems relatively minor (barring some intense outliers, like Apple and Disney, which I do think are much more agentic here than others, and have caused pretty great harm in doing so, like Disney being responsible for copyright being far too long in the US because Disney was terribly afraid of anyone re-using their characters and so tainting Disney’s image).
“the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”
Are you combining Venkatesh Rao’s loser/clueless/sociopath taxonomy with David Chapman’s geek/mop/sociopath?
(ETA: I know this is not relevant to the discussion, but I confuse these sometimes.)
Very low, though trending a bit higher over time. The policy-focused playbook has to deal with a lot more trickiness here than AI-2027, and you have to deal more with policymakers and stuff, but currently y’all don’t do very much of the kind of thing I am talking about here.
I really appreciate your clear-headedness at recognizing these phenomena even in people “on the same team”, i.e. people very concerned about and interested in preventing AI X-Risk.
However, I suspect that you also underrate the amount of self-deception going on here. It’s much easier to convince others if you convince yourself first. I think people in the AI Safety community self-deceive in various ways, for example by choosing to not fully think through how their beliefs are justified (e.g. not acknowledging the extent to which they are based on deference—Tsvi writes about this in his recent post rather well).
There are of course people who explicitly, consciously, plan to deceive, thinking things like “it’s very important to convince people that AI Safety/policy X is important, and so we should use the most effective messaging techniques possible, even if they use false or misleading claims.” However, I think there’s a larger set of people who, as they realize claims A B C are useful for consequentialist reasons, internally start questioning A B C less, and become biased to believe A B C themselves.
Sure! I definitely agree that’s going on a lot as well. But I think that kind of deception is more common in the rest of the world, and the things that set this community apart from others is the ability to do something more intentional here (which then combined with plenty of self-deception can result in quite catastrophic outcomes, as FTX illustrates).
I do think it’s not good! But also, it’s an important issue and you have to interface with people who aren’t super principled all the time. I just don’t want people to think of the AI Safety community as some kind of community of saints. I think it’s pretty high variance, and you should have your guard up a good amount.
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
A lot of that is the result of agency, which is often good, but in this case a double-edged sword. Naive consequentialism and lots of intense group-beliefs make the appropriate level of paranoia when interfacing with the AI Safety community higher than with most of these places.
“Appropriate levels of paranoia when interfacing with you” is of course not the only measure of honor and integrity, though as I am hoping to write about sometime this week, it’s kind of close to the top.
On that dimension, I think the AI Safety community is below AGI companies and the US military, and above all the other ones on this list. For the AGI companies, it’s unclear to me how much of it is the same generator. Approximately 50% of the AI Safety community are employed by AI labs, and they have historically made up a non-trivial fraction of the leadership of those companies, so those datapoints are highly correlated.
This is a wild claim. Don’t religions sort of centrally try to get you to believe known-to-be-false claims? Don’t politicians famously lie all the time?
Are you saying that EAs are better at deceiving people than typical members of those groups?
Are you claiming that members of those groups may regularly spout false claims, but they’re actually not that invested in getting others to believe them?
Can you be more specific about the way in which you think AI Safety folk are worse?
I agree that institutionally they are set up to do a lot of that, but the force they bring to bear on any individual is actually quite small in my experience, compared to what I’ve seen in AI safety spaces. Definitely lots of heterogeneity here, but most optimization that religions do to actually keep you believing in their claims are pretty milquetoast.
Definitely in-expectation! I think SBF, Sam Altman, Dario, Geoff Anders plus a bunch of others are pretty big outliers on these dimensions. I think in-practice there is a lot variance between individuals, with a very high-level gloss being something like “the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities”.
Yeah, that’s a good chunk of it. Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people’s beliefs in the same way as seems routine around here. Some of it is a competence side-effect. Paranoia becomes worse the more competent your adversary is. The AI Safety community is a particularly scary adversary in that respect (and one that due to relatively broad buy-in for something like naive-consequentialism can bring more of its competence to bear on the task of deceiving you).
I’ve been around the community for 10 years. I don’t think I’ve ever seen this?[1]
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should “count”? Am I just selected out of such conversations somehow?
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what’s up to volunteers, see what’s confusing, and adjust what we’re saying).
(Is this the kind of thing you mean?)
But, importantly, we’re clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I’m happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I’m not going to lead with that).
Pretty much every day, I check “If someone had made this argument against [social media], would that have made me think that it was imperative to shut it down?”, about proffered anti AI arguments.
Also, come on, this seems false. I am pretty sure you’ve seen Leverage employees do this, and my guess is you’ve seen transcripts of chats of this happening with quite a lot of agency at FTX with regards to various auditors and creditors.
Yeah, that does sound roughly like what I mean, and then I think most people just drop the second part:
I do not think that SBF was doing this part. He was doing the former though!
My best guess you are doing a mixture of:
Indeed self-selecting yourself out of these environments
Having a too-narrow conception of the “AI Safety community” that forms a Motte where you conceptually exclude people who do this a lot (e.g. the labs themselves), but in a way that then makes posts like the OP we are commenting on misleading
Probably have somewhat different standards for this (indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception).
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
Is this not common in politics? I thought this was a lot of what politics was about. (Having never worked in politics.)
And corporate PR campaigns too for that matter.
I have been very surprised by how non-agentic politics is! Like, there certainly is a lot of signaling going on, but when reading stuff like Decidingtowin.org it becomes clear how little optimization actually goes into saying things that will get you voters and convince stakeholders.
I do think a lot of that is going on there, and in the ranking above I would probably put the current political right above AI safety and the current political left below AI safety. Just when I took the average it seemed to me like it would end up below, largely as a result of a severe lack of agency as documented in things like deciding-to-win.
Re corporate campaigns: I think those are really very milquetoast. Yes, you make cool ads, but the optimization pressure here seems relatively minor (barring some intense outliers, like Apple and Disney, which I do think are much more agentic here than others, and have caused pretty great harm in doing so, like Disney being responsible for copyright being far too long in the US because Disney was terribly afraid of anyone re-using their characters and so tainting Disney’s image).
Are you combining Venkatesh Rao’s loser/clueless/sociopath taxonomy with David Chapman’s geek/mop/sociopath?
(ETA: I know this is not relevant to the discussion, but I confuse these sometimes.)
Oh, oops, yep, I confused the two. I meant geek/mop/sociopath in the David Chapman sense. Thank you for the catch!
Huh. Seems super wrong to me fwiw. How would you rank AIFP on the list?
Very low, though trending a bit higher over time. The policy-focused playbook has to deal with a lot more trickiness here than AI-2027, and you have to deal more with policymakers and stuff, but currently y’all don’t do very much of the kind of thing I am talking about here.
I really appreciate your clear-headedness at recognizing these phenomena even in people “on the same team”, i.e. people very concerned about and interested in preventing AI X-Risk.
However, I suspect that you also underrate the amount of self-deception going on here. It’s much easier to convince others if you convince yourself first. I think people in the AI Safety community self-deceive in various ways, for example by choosing to not fully think through how their beliefs are justified (e.g. not acknowledging the extent to which they are based on deference—Tsvi writes about this in his recent post rather well).
There are of course people who explicitly, consciously, plan to deceive, thinking things like “it’s very important to convince people that AI Safety/policy X is important, and so we should use the most effective messaging techniques possible, even if they use false or misleading claims.” However, I think there’s a larger set of people who, as they realize claims A B C are useful for consequentialist reasons, internally start questioning A B C less, and become biased to believe A B C themselves.
Sure! I definitely agree that’s going on a lot as well. But I think that kind of deception is more common in the rest of the world, and the things that set this community apart from others is the ability to do something more intentional here (which then combined with plenty of self-deception can result in quite catastrophic outcomes, as FTX illustrates).
This is not good. Why should people run the risk of interacting with the AI safety community if this is true?
I do think it’s not good! But also, it’s an important issue and you have to interface with people who aren’t super principled all the time. I just don’t want people to think of the AI Safety community as some kind of community of saints. I think it’s pretty high variance, and you should have your guard up a good amount.