indeed, a thing I’ve updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
Do you have an example of this? (It sounds like you think that I might be participating in this dynamic on one side or the other.)
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don’t really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).