First of all thank you for your post, it’s very thorough :)
While I want to reread it in case I missed any arguments for this, the main issue I usually have with these trust webs is the propensity for the creation of echo chambers: by relying only on those you trust and who they trust, you might filter out others opinions not because they are less valid, but because you disagree on some fundamental axioms. Have you given any thought on how to avoid echo chambers in these webs of trust?
Best, Miguel
I think it’s possible to build a Goodharts example on a 2D vector space.
Say you get to choose two parameters x and y. You want to maximize their sum, but you are constrained by x2+y2=2 . Then the maximum is attained when x=y=1. Now assume that y is hard to measure, so you use x as a proxy. Then you move from the optimal point we had above to the worse situation where x=√2, but y=0.
The key point being that you are searching for a solution in a manifold inside your vector, but since some dimensions of that vector space are too hard or even impossible to measure, you end up in sub optimal points of your manifold.
In formal terms you have a true utility function u(v) based on all the data v you have, and a misaligned utility function u′(v′) based on the subspace of known variables v′, where u′ could be obtained by integrating out the unknown dimensions if we know their probability distribution, or any other technique that might be more suitable.
Would this count as a more substantive assumption?
Best, Miguel
Edit: added the “In formal terms” paragraph