Analogy: there are ways to phrase true facts that are predictably misleading to other people (about that fact and/or about those who choose to say that version of it). If you announce a fact phrased misleadingly, you are making other people’s models actively worse, and it is a poor defense to say that X is literally true the way you understand the words.
In this case, elevating Caroline to a position of power in Manifund will predictably cause other people to incorrectly conclude that our community does not think the FTX fraud was wrong[1]. This is a straightforward cost of the action, it’s legitimate to count it as a cost, and the value above replacement of Caroline as an employee seems unlikely to justify that cost.
In some cases, the cost of incorrectly updating other people is worth paying. But I have a hard time believing that Caroline’s value above replacement is worth it. (On top of other obvious costs like “a person guilty of major fraud, even if they are sincere about having reformed, is likely to have weaker ethical inhibitions than a replacement employee”.)
[1] In fact, I’ve heard that Austin does think FTX was not actually that bad, but I’m focusing on a hypothetical that’s more generous to him.
TBC my core objection is “Caroline is in the class of people for which we accurately expect a higher rate of fraud”, and I’m trying to explain why I think that that is sufficient for people to update on Austin’s fraud tolerance and to consider those updates in their decisions, even if a burning bush tells Austin privately that Caroline is an exception.
This seems much better. Saying something like “I’m not in a position to trust her after the major fraud, and your post is basically not giving me enough information to move past that prior, so basically it seems like you’re doing something insane and terrible and I will act accordingly” is much better than “what you’re doing looks bad to many people and that means it is bad, end of discussion”.
There’s something important missing in your version, though. You’re pointing to “I am evaluating Austin as doing something bad given the limited information that I have”, whereas Elizabeth seems to be getting at something more like “it is Austin’s job (as a member of this ecosystem) to give us more information when he does something like this; and he’s failing at his job”.
This is obviously trickier reasoning to use, because it’s easy to slide into things like “the fact that I don’t trust you is evidence that you’ve failed to invest sufficiently in giving me reason to trust you which is evidence that you’re untrustworthy”. But I think it’s also important reasoning—for example, when AI labs seem to be doing shady shit, we want to say not just “it seems to us given the information we have that you’re doing shady shit, which is bad”, but also “even if you’re not doing shady shit you’re being irresponsible by not giving outside observers enough information to be confident that you’re not doing shady shit”.
You’re pointing to “I am evaluating Austin as doing something bad given the limited information that I have”, whereas Elizabeth seems to be getting at something more like “it is Austin’s job (as a member of this ecosystem) to give us more information when he does something like this; and he’s failing at his job”.
I’m probably closer to the latter than Ben is but it’s not how I would phrase it and it’s a somewhat separate principle.
I wouldn’t use the naked phrase “it is Austin’s job to”. But I would say “if Austin doesn’t want to be judged as a central example of hiring a fraud felon, the burden is on him to make it sufficiently cheap to detect that Caroline is different (which may be impossible, and that’s not a failure of the observers).” The difference is that Austin is still allowed to decide hiring Caroline is worth the judgement it incurs. I’m just not sympathetic to the consequences, even if a burning bush really did vouch for Caroline.
even if you’re not doing shady shit you’re being irresponsible by not giving outside observers enough information to be confident that you’re not doing shady shit”
The legal term here is “demonstrable compliance”, and I think it’s really important. It’s not enough to know how to drive, you have to prove it to the state, because a system where everyone judges their own driving leads to too many deaths. It’s not enough to buy a ticket to the conference, you need to wear your badge so they can see you did so.
This decision is not only evidence that Austin has poor judgment, but also that he cannot be trusted with leadership roles in-general.
(Why? Well, as a result of this decision I expect I will no longer encourage people to donate to his projects relating to AI and the surrounding ecosystem[1], and I expect this is true of many other relevant people. Austin could have found this out ahead of time, but evidently didn’t think to check. This means that in future situations where he has power, he will likely continue to make decisions that will cause everyone to lose faith in him and not notice this fact until afterward, and this is a terrible property for someone with a leadership role in an ecosystem.)
Yeah, I agree that the “bad optics” frame is not good. But I also disagree with the way you’re framing it. For example, your paragraph in parentheses relies on the idea that Austin “didn’t think to check” and “[didn’t] notice this fact until afterwards”. It seems much more likely to me that Austin thought this outcome was plausible, but didn’t endorse making decisions by taking it into account.
And when you say “he will likely continue to make decisions that will cause everyone to lose faith in him and not notice this fact until afterward” as a reason for not giving him power, that’s also pretty exploitable reasoning, because you can levy a very similar critique against someone who is acting in high-integrity ways without caring if they e.g. get defunded in response. (Concretely, I can imagine EAs saying a pretty similar thing about Lighthaven—indeed, it’s not too far off the critique that Scott Alexander just made of hosting Slutcon.)
On a meta level, the thing I’m pushing back against is that it feels like you’re taking Austin’s trustworthiness to be a fact about him that you’re just gathering information about, rather than a property of the relationship between him and the community. The solution IMO is to take a stance that’s more like “Austin is not acting to uphold a high-trust relationship between him and the rest of the community—e.g. we had reasonable expectations which he didn’t meet”.
I (sadly) don’t have much time to spend on this thread; but I am here referring to communicating with your trusted allies, whereas I understand respectability politics to be about the signals you send to the broader public.
I guess, it still feels weird to speak in my own defense but I’ll try to do so here:
I did in fact check in with several people about the hiring announcement, including Oli, who I’m very grateful for about his criticisms. Possibly you meant I should have sampled more people before deciding to hire Caroline. I feel like the process I ran here was kind of reasonable—a 1 month work trial, another month incognito, and then a public announcement once we knew we would work well together. I tend to think doing things like this in public is better than in private.
I think “trust with leadership” is kind of an odd frame; I mostly don’t know that anyone’s eg hired me for an executive position; insofar as I have any leadership, I think I mostly built those through my own track record of work. I agree that donors and employees and partners and the general public should look at my actions now and reassess whether they want to continue working with me or Manifund; I’ve encouraged people to do so.
I appreciate that you think I’ve been a good partner to trade with! You and Lightcone broadly have been as well, and I try to share this fact where I can. I hope to conduct many more positive sum trades in the future.
I mostly wish you had publicly argued more for your belief that a convicted criminal basically didn’t do the crimes. I suspect this would have led me to trust you more (though more likely for you to find out your evidence & arguments were weak).
Perhaps you have and I didn’t see it? But either way it seemed crazy to me to not say something in your announcement post like “I understand to many people I respect, and to many people who use and support Manifund and Mox that I hired someone who is a major convicted felon to a relevantly similar role to where they committed their crimes, and that this seems like a straightforwardly bad decision, but FYI I actually think that they largely didn’t do the crimes they confessed to, so it’s not actually that bad” and (for bonus points) some reasons why you think that.
I would say (possibly with the benefit of hindsight) that a piece of writing more shaped like this would be appropriate if you wanted your allies to come to think this was a good decision.
Both the inferential gap here and also the quality-of-the-decision-gap are both far more substantial than usual.
I appreciate the pointer and feedback! In any case, I’m currently thinking pretty hard about whether the EA or AI safety community feels like the right set of allies for me and my work. More writeups to come, I suppose.
I let EA out of my heart a long time ago. I felt the culture was rotting and its leadership abandoning it back in ~2018; I had forbid myself to even look at the EA Forum almost a year before the FTX crash, to help me separate from it. Looking at some of the threads there about this decision of yours this week has just been opening up old traumas, and I quickly closed them again. I would encourage you to focus your efforts elsewhere, I see little good that will come of that place.
I don’t relate to the AI safety community in an especially similar way. It is one of the actors in the play of what is going on, like the news media, or China; with the main difference being that I am more a part of it. I have more ability to steer it, and more responsibility for its impacts.
Not that you asked, but I thought I’d mention it anyway, in case you were interested.
Could it be both “it seems like you’re doing something insane and terrible and I will act accordingly” and “it probably also seems that way to many other people, and my silence would look like an approval, so I need to distance myself from you visibly (instead of just ignoring you)”?
One thing is “I don’t trust you anymore” and the other thing is “you are burning social capital, so I am building a firewall between you and me”.
In this case, elevating Caroline to a position of power in Manifund will predictably cause other people to incorrectly conclude that our community does not think the FTX fraud was wrong[1]. This is a straightforward cost of the action, it’s legitimate to count it as a cost, and the value above replacement of Caroline as an employee seems unlikely to justify that cost.
Sure, but if we want to play the consequences game, being the kind of person who blocks or prevents promoting Caroline to a position of power in Manifund, even though they know (via some hypothetical means) that it will not make future thing FTX fraud less likely, therefore communicates that we do not care about what things actually cause fraud, it communicates that we care about covering our ass whether that helps with fraud or not.
This is a straightforward cost of the action. It’s legitimate to count it as a cost, and the marginal cost of signaling that, instead of just saying “this is bad because it makes fraud more likely” if and only if we believe that, seems likely to be greater than whatever thing you get by signaling affiliation with some generic “FTX was wrong” coalition beyond what is justified.
Like, I think the real analysis here is complicated. Going up a simulacra level in your analysis of a topic is a major major cost and should not be done lightly. Sometimes it’s the right call, but man, it seems to me like a more straightforward cost than the benefits of playing these signaling games marginally better.
If we want to get decision-theoretical about it, I view something on the order of FTX to be worthy of a grim-trigger policy. If you substantively aided the fraud, you should be barred from positions of power in the community thenceforth.
I agree with that this is a good prior given appropriate operationalization of “positions of power”, which is importantly not what we are discussing! I also think you are underestimating the opportunities people have for genuine redemption and putting people in positions of conditional power where their strengths are leveraged without needing to extend them much trust in domains where their judgement is weak.
E.g. Caroline ending up in a position of power as a result of becoming a really really good forecaster who lots of people defer to because she has an obviously strong forecasting track record, while people are appropriately hesitant to not give her unconditional power, seems like a fine outcome, and if anything it would be inappropriate for institutions like Metaculus to ban Caroline because of a consequentialist argument about her being publicly displayed as a strong forecaster being bad for the world by empowering her.
Surely you agree that the correct place to draw the line is short of a role where she could, if she chose, commit massive fraud again within the EA ecosystem!
Analogy: there are ways to phrase true facts that are predictably misleading to other people (about that fact and/or about those who choose to say that version of it). If you announce a fact phrased misleadingly, you are making other people’s models actively worse, and it is a poor defense to say that X is literally true the way you understand the words.
In this case, elevating Caroline to a position of power in Manifund will predictably cause other people to incorrectly conclude that our community does not think the FTX fraud was wrong[1]. This is a straightforward cost of the action, it’s legitimate to count it as a cost, and the value above replacement of Caroline as an employee seems unlikely to justify that cost.
In some cases, the cost of incorrectly updating other people is worth paying. But I have a hard time believing that Caroline’s value above replacement is worth it. (On top of other obvious costs like “a person guilty of major fraud, even if they are sincere about having reformed, is likely to have weaker ethical inhibitions than a replacement employee”.)
[1] In fact, I’ve heard that Austin does think FTX was not actually that bad, but I’m focusing on a hypothetical that’s more generous to him.
TBC my core objection is “Caroline is in the class of people for which we accurately expect a higher rate of fraud”, and I’m trying to explain why I think that that is sufficient for people to update on Austin’s fraud tolerance and to consider those updates in their decisions, even if a burning bush tells Austin privately that Caroline is an exception.
This seems much better. Saying something like “I’m not in a position to trust her after the major fraud, and your post is basically not giving me enough information to move past that prior, so basically it seems like you’re doing something insane and terrible and I will act accordingly” is much better than “what you’re doing looks bad to many people and that means it is bad, end of discussion”.
There’s something important missing in your version, though. You’re pointing to “I am evaluating Austin as doing something bad given the limited information that I have”, whereas Elizabeth seems to be getting at something more like “it is Austin’s job (as a member of this ecosystem) to give us more information when he does something like this; and he’s failing at his job”.
This is obviously trickier reasoning to use, because it’s easy to slide into things like “the fact that I don’t trust you is evidence that you’ve failed to invest sufficiently in giving me reason to trust you which is evidence that you’re untrustworthy”. But I think it’s also important reasoning—for example, when AI labs seem to be doing shady shit, we want to say not just “it seems to us given the information we have that you’re doing shady shit, which is bad”, but also “even if you’re not doing shady shit you’re being irresponsible by not giving outside observers enough information to be confident that you’re not doing shady shit”.
I’m probably closer to the latter than Ben is but it’s not how I would phrase it and it’s a somewhat separate principle.
I wouldn’t use the naked phrase “it is Austin’s job to”. But I would say “if Austin doesn’t want to be judged as a central example of hiring a fraud felon, the burden is on him to make it sufficiently cheap to detect that Caroline is different (which may be impossible, and that’s not a failure of the observers).” The difference is that Austin is still allowed to decide hiring Caroline is worth the judgement it incurs. I’m just not sympathetic to the consequences, even if a burning bush really did vouch for Caroline.
The legal term here is “demonstrable compliance”, and I think it’s really important. It’s not enough to know how to drive, you have to prove it to the state, because a system where everyone judges their own driving leads to too many deaths. It’s not enough to buy a ticket to the conference, you need to wear your badge so they can see you did so.
This decision is not only evidence that Austin has poor judgment, but also that he cannot be trusted with leadership roles in-general.
(Why? Well, as a result of this decision I expect I will no longer encourage people to donate to his projects relating to AI and the surrounding ecosystem[1], and I expect this is true of many other relevant people. Austin could have found this out ahead of time, but evidently didn’t think to check. This means that in future situations where he has power, he will likely continue to make decisions that will cause everyone to lose faith in him and not notice this fact until afterward, and this is a terrible property for someone with a leadership role in an ecosystem.)
These are both important things to discuss; nonetheless framing it primarily around “bad optics” and “PR” is a terribly exploitable tactic and known failure mode.
As an aside, Austin has been an excellent trade-partner, and I expect to still exchange money for goods and services with him.
Yeah, I agree that the “bad optics” frame is not good. But I also disagree with the way you’re framing it. For example, your paragraph in parentheses relies on the idea that Austin “didn’t think to check” and “[didn’t] notice this fact until afterwards”. It seems much more likely to me that Austin thought this outcome was plausible, but didn’t endorse making decisions by taking it into account.
And when you say “he will likely continue to make decisions that will cause everyone to lose faith in him and not notice this fact until afterward” as a reason for not giving him power, that’s also pretty exploitable reasoning, because you can levy a very similar critique against someone who is acting in high-integrity ways without caring if they e.g. get defunded in response. (Concretely, I can imagine EAs saying a pretty similar thing about Lighthaven—indeed, it’s not too far off the critique that Scott Alexander just made of hosting Slutcon.)
On a meta level, the thing I’m pushing back against is that it feels like you’re taking Austin’s trustworthiness to be a fact about him that you’re just gathering information about, rather than a property of the relationship between him and the community. The solution IMO is to take a stance that’s more like “Austin is not acting to uphold a high-trust relationship between him and the rest of the community—e.g. we had reasonable expectations which he didn’t meet”.
This seems so much more exploitable and closer to respectability politics than anything I said
I (sadly) don’t have much time to spend on this thread; but I am here referring to communicating with your trusted allies, whereas I understand respectability politics to be about the signals you send to the broader public.
I guess, it still feels weird to speak in my own defense but I’ll try to do so here:
I did in fact check in with several people about the hiring announcement, including Oli, who I’m very grateful for about his criticisms. Possibly you meant I should have sampled more people before deciding to hire Caroline. I feel like the process I ran here was kind of reasonable—a 1 month work trial, another month incognito, and then a public announcement once we knew we would work well together. I tend to think doing things like this in public is better than in private.
I think “trust with leadership” is kind of an odd frame; I mostly don’t know that anyone’s eg hired me for an executive position; insofar as I have any leadership, I think I mostly built those through my own track record of work. I agree that donors and employees and partners and the general public should look at my actions now and reassess whether they want to continue working with me or Manifund; I’ve encouraged people to do so.
I appreciate that you think I’ve been a good partner to trade with! You and Lightcone broadly have been as well, and I try to share this fact where I can. I hope to conduct many more positive sum trades in the future.
I mostly wish you had publicly argued more for your belief that a convicted criminal basically didn’t do the crimes. I suspect this would have led me to trust you more (though more likely for you to find out your evidence & arguments were weak).
Perhaps you have and I didn’t see it? But either way it seemed crazy to me to not say something in your announcement post like “I understand to many people I respect, and to many people who use and support Manifund and Mox that I hired someone who is a major convicted felon to a relevantly similar role to where they committed their crimes, and that this seems like a straightforwardly bad decision, but FYI I actually think that they largely didn’t do the crimes they confessed to, so it’s not actually that bad” and (for bonus points) some reasons why you think that.
I kind of wanted to, but some people including Oli thought I should not include FTX apologia in my hiring announcement. So, shrug.
I would say (possibly with the benefit of hindsight) that a piece of writing more shaped like this would be appropriate if you wanted your allies to come to think this was a good decision.
Both the inferential gap here and also the quality-of-the-decision-gap are both far more substantial than usual.
I appreciate the pointer and feedback! In any case, I’m currently thinking pretty hard about whether the EA or AI safety community feels like the right set of allies for me and my work. More writeups to come, I suppose.
I let EA out of my heart a long time ago. I felt the culture was rotting and its leadership abandoning it back in ~2018; I had forbid myself to even look at the EA Forum almost a year before the FTX crash, to help me separate from it. Looking at some of the threads there about this decision of yours this week has just been opening up old traumas, and I quickly closed them again. I would encourage you to focus your efforts elsewhere, I see little good that will come of that place.
I don’t relate to the AI safety community in an especially similar way. It is one of the actors in the play of what is going on, like the news media, or China; with the main difference being that I am more a part of it. I have more ability to steer it, and more responsibility for its impacts.
Not that you asked, but I thought I’d mention it anyway, in case you were interested.
Could it be both “it seems like you’re doing something insane and terrible and I will act accordingly” and “it probably also seems that way to many other people, and my silence would look like an approval, so I need to distance myself from you visibly (instead of just ignoring you)”?
One thing is “I don’t trust you anymore” and the other thing is “you are burning social capital, so I am building a firewall between you and me”.
Sure, but if we want to play the consequences game, being the kind of person who blocks or prevents promoting Caroline to a position of power in Manifund, even though they know (via some hypothetical means) that it will not make future thing FTX fraud less likely, therefore communicates that we do not care about what things actually cause fraud, it communicates that we care about covering our ass whether that helps with fraud or not.
This is a straightforward cost of the action. It’s legitimate to count it as a cost, and the marginal cost of signaling that, instead of just saying “this is bad because it makes fraud more likely” if and only if we believe that, seems likely to be greater than whatever thing you get by signaling affiliation with some generic “FTX was wrong” coalition beyond what is justified.
Like, I think the real analysis here is complicated. Going up a simulacra level in your analysis of a topic is a major major cost and should not be done lightly. Sometimes it’s the right call, but man, it seems to me like a more straightforward cost than the benefits of playing these signaling games marginally better.
If we want to get decision-theoretical about it, I view something on the order of FTX to be worthy of a grim-trigger policy. If you substantively aided the fraud, you should be barred from positions of power in the community thenceforth.
I agree with that this is a good prior given appropriate operationalization of “positions of power”, which is importantly not what we are discussing! I also think you are underestimating the opportunities people have for genuine redemption and putting people in positions of conditional power where their strengths are leveraged without needing to extend them much trust in domains where their judgement is weak.
E.g. Caroline ending up in a position of power as a result of becoming a really really good forecaster who lots of people defer to because she has an obviously strong forecasting track record, while people are appropriately hesitant to not give her unconditional power, seems like a fine outcome, and if anything it would be inappropriate for institutions like Metaculus to ban Caroline because of a consequentialist argument about her being publicly displayed as a strong forecaster being bad for the world by empowering her.
Surely you agree that the correct place to draw the line is short of a role where she could, if she chose, commit massive fraud again within the EA ecosystem!
Definitely! I don’t know whether that’s true about her current position and think the correct place to draw the line is a bunch earlier than that.