I probably don’t understand the risks! Like I have some similar example institutions in mind, but none that are super similar to what I’m doing or which have consequences as bad as you are implying, so I assume maybe there’s a significant history that you are aware of and which I have missed. What examples do you have in mind?
I plan to write my idea in much greater detail later before actually launching it. More information now about the risks could be convenient if it shows how it is a waste of time to pursue, though alternatively if I disagree I might address the disagreements in later writeups.
One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
Like in the context of the FTX situation, a proposal that I’ve discussed with a number of people is, “Fuck, man, why did people trust Sam? I didn’t trust Sam.” “What we should have done is,we should have just created a number, like, there’s 10 cards and these are the ‘actually trustworthy people’ and ‘high-integrity people’ cards, and we should have given them to 10 people in the EA community who we actually think are [highly] trustworthy and high-integrity people, so that it actually makes sense for you to trust them. And we should just have [high-legibility] signals of trust and judgment that are clearly legible to the world.”
To which my response was “Lol, that would have made this whole situation much worse.” I can guarantee you that if you [had] handed a number of people—in this ecosystem or any other ecosystem—the “This person has definitely good judgment and you should trust what they say.” [card. Then] in the moment somebody has that card and has that official role in the ecosystem, of course they will be [under a] shitton of adversarial pressure, for [them to] now endorse people who really care about getting additional resources, who really care about stuff.
...
And then I’m like, “Well, there are two worlds. Either nobody cares about who you think is high-integrity and trustworthy, or people *do* care and now you’ve made the lives of everyone who you gave a high-integrity / trustworthy card a lot worse. Because now they’re just an obvious giant target, that if you successfully get one of the people of the high-integrity, high-trustworthy cards to endorse you, you have free reign and now challenging you becomes equivalent to challenging the “high-integrity, [high-trust] people” institution. Which sure seems like one of the hardest institutions to object to.
And I think we’ve seen this in a number of other places… There was a specific board set up by the Center for Applied Rationality, [where] CFAR kept being asked to navigate various community disputes, and they were like, “Look, man, we would like to run workshops, can we please do anything else?” And then they set up a board to be like, “Look, if you have community disputes in the Bay Area, go to this board. They will maybe do some investigative stuff, and then they will try to figure out what should happen, like, do mediation, maybe [speak about] who was actually in the right, who was in the wrong.”
And approximately the first thing that happened is that, like, one of the people who I consider most abusive in the EA community basically just captured that board, and [got] all the board members to endorse him quite strongly. And then when a bunch of people who were hurt by him came out, the board was like, “Oh, we definitely don’t think these [people who were abused] are saying anything correct. We trust the guy who abused everyone.”
Which is a specific example of, if you have an institution that is being given the power to [blame] and speak judgment on people, and try to create common knowledge about [what] is trustworthy and what is non-trustworthy, that institution is under a lot of pressure...
[We] can see similar things happening with HR departments all around the world. Where the official [purpose] of the HR department is to, you know, somehow make your staff happy and give them [ways to] escalate to management if their manager is bad. But most HR departments around the world are actually, like, a trap, where if you go to the HR department, probably the person you complained about is one of the first people to find out, and then you can be kicked out of the organization before your complaint can travel anywhere else.
It’s not true in all HR departments but it’s a common enough occurrence in HR departments that if you look at Hacker News and are like, “Should I complain to HR about my problems?”, like half of the commenters will be like, “Never talk to HR.” HR is the single most corrupt part of any organization. And I think this is the case because it also tends to be the place where hiring and firing [decisions get] made and therefore is under a lot of pressure.
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
My crux around this is something like; conflict resolution is an iterated and adversarial game, where failure can cost a lot. “Creating an institution to work on [uncovering injustices and predators]” looks to me a lot like “Creating an institution to keep secrets on computers connected to the internet.” You don’t just have to outsmart the basic problem, you also need to outsmart everyone who has an incentive to subvert your system. I don’t think that’s an impossible challenge but it’s harder than it looks, and it’s harder than it looks in part because some people are actively trying to obscure the ways in which it’s hard. The system can even look like it’s working fine right up until it tries to tackle something important, in the same way that a substitution cypher will look like it’s working fine right up until I try to store a bunch of bank account information with it.
There’s also a dynamic of something like… this is one of those issues where being too interested in the problem is correlated with being bad at solving it. Obviously you have to compromise on this a little or these kinds of things never get done, but if someone’s only qualification is interest I think the EV is very negative.
That sounds really discouraging, so I want to tell @tailcalled: I think it’s great you care about people are want to prevent them from being hurt. I think the easiest, least risky way to do that is to create abundance so people have less dependence on any one entity and are thus less vulnerable. The more parties being thrown by people who aren’t creeps (or harboring creeps), the easier it is to avoid the parties that are. So I’d encourage you to start by building socially, rather than investigation.
the median outcome for projects like this is doing far more harm than good
you haven’t given any indication you understand the risks, much less are likely to beat them.
I probably don’t understand the risks! Like I have some similar example institutions in mind, but none that are super similar to what I’m doing or which have consequences as bad as you are implying, so I assume maybe there’s a significant history that you are aware of and which I have missed. What examples do you have in mind?
I plan to write my idea in much greater detail later before actually launching it. More information now about the risks could be convenient if it shows how it is a waste of time to pursue, though alternatively if I disagree I might address the disagreements in later writeups.
One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
I plan on doing that.
I think that that Habryka podcast has a lot of potential for projects, it just needs a wide variety of people to build off of it.
Haven’t downvoted but was considering it.
My crux around this is something like; conflict resolution is an iterated and adversarial game, where failure can cost a lot. “Creating an institution to work on [uncovering injustices and predators]” looks to me a lot like “Creating an institution to keep secrets on computers connected to the internet.” You don’t just have to outsmart the basic problem, you also need to outsmart everyone who has an incentive to subvert your system. I don’t think that’s an impossible challenge but it’s harder than it looks, and it’s harder than it looks in part because some people are actively trying to obscure the ways in which it’s hard. The system can even look like it’s working fine right up until it tries to tackle something important, in the same way that a substitution cypher will look like it’s working fine right up until I try to store a bunch of bank account information with it.
It seems like you aren’t noticing some skulls.
There’s also a dynamic of something like… this is one of those issues where being too interested in the problem is correlated with being bad at solving it. Obviously you have to compromise on this a little or these kinds of things never get done, but if someone’s only qualification is interest I think the EV is very negative.
That sounds really discouraging, so I want to tell @tailcalled: I think it’s great you care about people are want to prevent them from being hurt. I think the easiest, least risky way to do that is to create abundance so people have less dependence on any one entity and are thus less vulnerable. The more parties being thrown by people who aren’t creeps (or harboring creeps), the easier it is to avoid the parties that are. So I’d encourage you to start by building socially, rather than investigation.