I think it’s not a stable equilibrum to have one Evals Orgs. A) even if it was a good idea, I don’t see how we can get from-here-to-there on coordinating on a monopolistic Evals Org. B) I think having Healthy Competition is good and you should avoiding structuring your ecosystem around a monopoly if you can help it. There are just a lot of benefits of having multiple Evals Teams competing to develop better methods.
So I see the goal here as “design an incentive system such that the competition is good, instead of bad.”
It’s possible that competitions similar to the Audit Games concept would fit in here.
I think it’s not a stable equilibrum to have one Evals Orgs. A) even if it was a good idea, I don’t see how we can get from-here-to-there on coordinating on a monopolistic Evals Org. B) I think having Healthy Competition is good and you should avoiding structuring your ecosystem around a monopoly if you can help it. There are just a lot of benefits of having multiple Evals Teams competing to develop better methods.
So I see the goal here as “design an incentive system such that the competition is good, instead of bad.”
It’s possible that competitions similar to the Audit Games concept would fit in here.