I am in the process of finishing up a cost-benefit analysis on the value of water quality programs in Ohio. The statewide policy alternatives I am considering are based on the H2Ohio program, which is currently focused in a handful of northwest Ohio counties. Today I wanted to briefly talk about the challenges that come with trying to scale up pilot programs and how policymakers should think about expanding successful programs.
First, let’s recognize why we have pilot programs in the first place. The goal of a pilot program is to test the effectiveness of a proposal before investing a large amount of resources should it fail for some reason. If the trial run is successful, then the next step is to scale up the program to serve a larger population.
However, pilot programs are often specifically designed to succeed in a way that is harder to replicate on a larger scale. As the economist John List points out in an episode of the Freakonomics Radio podcast, “after researchers in the social sciences do an efficacy test, they forget to tell everyone else that it was an efficacy test.”
This is not to say that it is wrong for pilot programs to try to achieve good results. It is hard to imagine a researcher getting funding for some intervention where implementation is shoddy and where they plan to target the people least likely to benefit. However, as programs grow it becomes more difficult to ensure implementation fidelity and proper targeting.
Limiting participation in programs to only those who will receive the greatest benefits also might lead to the most efficient outcome, but it might not be the most equitable outcome. There is no right or wrong way to trade off equity and efficiency. Understanding that the two sometimes compete when resources are limited is the essence of good analysis and policymaking.
In the context of the H2Ohio program, one specific alternative I am considering is making the whole state eligible for the current program. Currently, the program is only open to counties in Northeast Ohio which have large concentrations of farmland and are part of the Lake Erie Basin. Because the rest of the state does not have the same contributions to Lake Erie algal blooms, this program likely won’t have exactly the same results in other counties beyond the current “pilot” counties.
These insights about problems common to scale-up can inform the sensitivity analysis phase of our cost-benefit analyses. As analysts, we can project what might happen under different conditions, say if scale-up works swimmingly or if scale-up falls far short of the impacts of the pilot program. Exploring different scenarios, especially the ones where things are not quite as effective as they were in the pilot program, often lead to better insights and better decisions.