In this installment of a continuing series on the steps in Eugene Bardach’s A Practical Guide for Policy Analysis: the Eightfold Path to More Effective Problem Solving, we address what Bardach calls “the hardest step” of policy analysis: projecting outcomes.
Projecting outcomes is difficult for two reasons. First, it requires us to predict the future, which we are often very bad at. Second, it requires us to confront the eternal problem of the forecaster: optimism. Lastly, politics often drives people to overstate confidence in their predictions.
Bardach begins by saying that projection of outcomes should extend the logic of common sense. This means using social science to create multiple models of the world, tempering it with facts on the ground and using metaphors to understand a complex world and communicate how it works.
Important when projecting outcomes to choose a base case to compare the policy to. This base case could be anywhere from the present situation to the future situation without the policy in place. It could incorporate future conditions under business as usual, changes that would occur if some policy were adopted, or even results of a certain policy option. The point is to provide a reasonable baseline scenario to compare the policy you are analyzing against.
In order to give actionable and credible information to policymakers, dare to make magnitude estimates. It is often not useful enough to give the direction of the results of a policy: an analyst must go beyond this to say how much the status quo will change under adoption of the policy. While a point estimate can be helpful here, a range will be more intellectually honest in the vast majority of analyses.
An analyst should also pay attention to trends, since trends might be the basis of projections. The analyst should pay attention to the past versus likely future scenarios to see if they line up with one another and should also make sure that cyclical trends are acknowledged so that short-term trends are not overextrapolated into long-term trends.
One tool for the analyst in projecting outcomes put forth by Bardach is break-even analysis. He advocates the usefulness of this technique because break-even estimates can shrink uncertainty. He suggests using break-even analysis to locate the point of minimum effectiveness given the costs. He also suggests applying this analysis to points in the process to see how effective those points need to be to give good results and to see how likely it is these results will come about and estimating probability of failure. All these approaches improve the credibility of policy analysis.
Bardach further urges analysts to try sensitivity analysis. Techniques such as Monte Carlo Analysis and Long Term Analysis can help determine how robust the analysis is and how likely a scenario will change based on assumptions made.
Important in projections is to confront the optimism problem. Bardach suggests doing this by “scenario writing”: imagining scenarios where things don’t go well and projecting outcomes under those scenarios. Analysts can also brainstorm undesirable side effects like moral hazard, overregulation, rent seeking, and how policies impact other policies. Optimism can pose risks, overlooking the harm policies can do to certain people, so addressing it is vital to ethical policy analysis.
Another factor to consider when projecting outcomes is the emergent-features problem. Sometimes policies can interact with other factors to create new outcomes that are hard to predict. This is a difficult problem, but one that can be ameliorated at least on the margins by playing out how major actors are likely to react to new policies.
A way to organize complex information when projecting outcomes is to construct an outcomes matrix. A table that lists policy options down one axis and criteria down another axis can be a useful tool for helping you as an analyst and the policymaker to understand the tradeoffs inherent in choice of different policy options.
We like to think of policies as transportable from content to context, but policy contexts differ. Just because something works in rural Ohio doesn’t mean it will work in urban Copenhagen. Paying attention to where studies are conducted and how comparable the context is can lead to better policy analysis.
Lastly, the analyst must setup for the next step. Using an outcomes matrix or otherwise making it clear what tradeoffs policymakers will be dealing with will help make the step of confronting trade-offs much easier.