Design policy to be testable

My father in law is an expert on legislation. He’s travelled around the world advising countries on how to craft laws and constitutions.

One of the projects he’s working on at the moment is advising on a bill to promote integrated education of different cultural and religious groups in Northern Ireland.

He was discussing the policy with me the other day, and was saying that he thinks it will be highly effective: “It’s impossible for a preacher to stand up and say that all catholics/​protestants are evil, when you’ve grown up in the same class as them and simply know they’re not.”

I agreed the reasoning made sense, but was somewhat skeptical it would necessarily have the desired effect. I can easily think of just so stories where the opposite happens.

For example, kids naturally form cliques. And protestant kids are more likely to form cliques with other protestant kids, and catholic kids with catholic kids. So they might both grow up ‘knowing’ that the other group were very cliquey and not at all interested in being friends.

So I asked, how do they plan to test whether it worked?

He looked at me for a few seconds, and then said that he supposed you could look at statistics such as rates of conscription to the IRA.

“Wait a minute, ” I replied, “do you mean to say that they haven’t designed up front criteria for how to measure how effective this policy is? Then how on earth will they know if it’s successful?”

Unfortunately, but not unsurprisingly, the idea of crafting policy in such a way that it can be easily testable just doesn’t seem to exist!

I know that we’re probably never going to have futarchies, or even deregulate healthcare. But I think the idea of designing policy to be testable is something that should be pretty uncontroversial (at least if you’re not a politician).

I would love to live in a world where any bill required the following 5 items:

  1. A description of how they’re going to measure the impact of the bill.

  2. A prediction distribution for the impact of the bill.

  3. A level at which they’d consider the bill a success, and a level at which they’d consider it a failure.

  4. A description of how they’re going to measure how bad things would be conditional on the bill not passing.

  5. A prediction distribution for this value.

I’m not even asking for these values to be used explicitly in any way, (e.g. I’m not suggesting we automatically fire lawmakers who propose bills which end up underachieving).

I think this will have a number of benefits:

Firstly it’s impossible to evaluate a policy without at least implicitly considering the expected impact of the policy, vs the impact of not passing the policy. Since this is so critical for judging policy it should be made explicit, so that disagreements can be far more effective. Instead of saying it’s a bad policy, you could distinguish between

  1. I don’t think the thing the policy is trying to achieve is worthwhile.

  2. I disagree with your predictions.

  3. I don’t think your expected impact is worth the cost.

Which can lead to far more fruitful discussion.

The second benefit is that when it’s a requirement for bills to describe how their impact is going to be measured, hopefully that’ll encourage them to design the bill in such a way that this can be done easily and accurately. Ideally we’d have a non partisan group of scientists whose sole purpose is to evaluate whether this is a sufficiently unbiased instrument for measuring the impact.

This will be enormously valuable in helping design future policy when you can accurately see which past policies worked well and which flopped, both in the same country and others.

Finally this gives us a way to evaluate lawmakers. I would expect institutions to spring up which rate lawmakers (and governments) on how effective both their policies and their predictions were, and this will hopefully aid voters in making accurate decisions rather than just going off gut feelings.