I Want Governmental A/B Testing

In voting in the giant San Francisco November 2010 ballot (six pages; there were something like 24 propositions), I noticed something: basically every prop, and every other political choice, seemed like it was going to have effects too complicated to actually predict.

What led me to this conclusion was that for basically all of them, you had at least one putative effect asserted by one side, where the other side asserted that the exact opposite result would occur. In general, people agreed that the intended effect of a given prop was a good one, the disagreement was on what would actually happen.

On top of that, neither side could usually offer a compelling reason why their version of the future was most likely. I mean "compelling" here in the scientific sense, like "9 out of 10 fundamentally similar regions (link to research showing similarity) that enacted such legislation (link to the research showing that the various versions are actually comparable) saw benefit X".

This matches human intuition, I think: individual humans are usually hard for other people to predict, and groups essentially impossible. Add in the complexities of a mostly efficient economy, and I think no-one really believes that correctly predicting the effect of legislative changes is possible.

None the less, the lack of information annoys me. What I choose to vote for feels so random: I don't know what's going to happen, and neither does anybody else. So, here's a idea on how to fix it. It's fraught with implementation problems, but maybe it could help in some cases, and that's better than nothing.

I propose that where the outcome is uncertain, we engage in legislative A/B testing.

"A/B Testing" is a process used in marketing to choose between various presentation options. As an example, you might send out a donation mailing, and send it to half the people in orange envelopes, and half the people in yellow envelopes. You then use the frequency of response in each group to inform future decisions. It might turn out that people in New York respond better to yellow, but people in Miami don't seem to care.

If someone is proposing that a particular tax will provide money for street repairs without any economic hardship, and everyone agrees that this would be good, and that this particular tax might do the job, why not just test it? Apply it to half the area in question (with the lines drawn to split socioeconomic status evenly), define a particular measure of success, and see how it goes?

The important part here is "measure of success". In this case it might be "no more than a 10% downturn in new businesses relative to the other half" and "50% more finished street repairs than the other half". This does two things:

  1. It clearly defines a goal we can all support, so we remember we're all on the same side of wanting a better future
  2. It gives simple end conditions: simply say that if those results have not been achieved after a year or whatever, the law is automatically removed.

If it seems to work, roll it out to the whole region. If not, roll it back. No acrimony, no arguing, just "does this actually help?".

Show PHP error messages