The Exploitability/Explainability Frontier
Epistemic status: quick insight that seems true in many cases
Your ability to notice that something is wrong trades off against your ability to convince others that said thing is, in fact, wrong. Suppose you discover a way in which the mainstream narrative is wrong in whatever field—maybe a famous math paper has an embarrassing typo, making the conclusion false. It would be very easy to convince others of this, if you knew. However, it’s very difficult to discover such mistakes, because they’re very rare. The flip side also seems true—if it’s fairly easy to discover that the mainstream narrative on something is wrong, then it’s likely to be extremely difficult to convince others of this. Politics is a great example of this; convincing others that your views are correct is nearly intractable, and often far more difficult than figuring out the correct view on specific issues.
To the extent something is easy to convince everyone of, once discovered, it’s likely to have already been discovered and therefore become mainstream. Society is hard to exploit in that sense—if it’s easy to discover and easy to spread, it would have already happened.
If you want to change society, you can move along the Exploitability/Explainability Frontier, either spending your time and energy looking for exploits that are relatively easy to explain, or spend it trying to explain exploits that were relatively easy to find. These are two distinct sets of research problems, and it seems worth distinguishing between the two.
Would it be, though? Math (anything proof related really) has results that can be checked, which is great. But basically no mathematician writes proofs using fully formal languages and automated theorem prover. And from what I’ve seen from researcher in formal methods in my lab, whenever you’re trying to formalize a proof from a published paper, there’s probably a tiny detail missing or a typo. But it’s actually pretty rare that “what the mathematicians meant” was wrong. (It happens sometimes). And if it’s a long paper, the burden of showing that the proof doesn’t pan out might be far from trivial.
So I believe you might be short selling the challenge in explaining difficult ideas.
My point is that it’s rare and therefore difficult to discover.
The kinds that are less rare are easier to discover but harder to convince others of, or at least harder to convince people that they matter.
I was drawing off this example, by the way: https://econjwatch.org/articles/recalculating-gravity-a-correction-of-bergstrands-1985-frictionless-case
A 35 year old model had a simple typo in it that got repeated in papers that built on it. Very easy to convince people that this is the case, but very difficult to discover such errors—most such papers don’t have those errors so you need to replicate a lot of correct papers to find the one that’s wrong.
If it’s difficult to show that the typo actually matters, that’s part of the difficulty of discovering it. My point is you should expect the sum of the difficulty in explaining and the difficulty in discovery to be roughly constant.
I don’t see why it would be roughly constant. I don’t even see why it would tradeoff for your examples: in the maths case, you could say that if you understood everything clearly about the proof, the typo and its consequence, you could convince most mathematicians of it. But for your political examples, finding the correct view seems way harder than to find a typo in a math proof and convincing others is way more difficult than for the maths typo.
Assume you’re at the frontier of being able to do research in that area and have similar abilities to others in that reference class. The total amount of effort most of those people will put in is the same, but it will be split across these two factors differently. The system being unexploitable corresponds to the sum here being constant.
There can be examples where both sides are difficult, which are out of the frontier.
Re politics, there are some issues that are difficult, some issues that are value judgments, and some that are fairly simple in the sense that spending a week seriously researching is enough to be pretty confident of the direction policy should be moved in.