When making an estimate of a cost, economic benefit, or some other important policy impact, a policy analyst is carrying out a very difficult task. She is trying to put numbers to things that we often don’t see as quantifiable. Inevitably, any analysis will lean on a range of assumptions in order to get from abstract idea to a number.
But what happens when we change these assumptions? And what happens if the empirical evidence we use is a little bit off from reality? This is where the policy analyst employs the important tool of sensitivity analysis.
Sensitivity Analysis is the process of estimating how assumptions included in an analysis impact the uncertainty around the findings of the analysis. By conducting sensitivity analysis, we can get an idea of how precise our findings are. We can also report these findings as a range so we don’t oversell the precision of our findings.
But how do we conduct sensitivity analysis? There are a few different ways to do this, but the most common approaches are what we call “partial sensitivity analysis,” “worst- and best-case analysis,” “breakeven analysis,” and “Monte Carlo Simulation.” Below are some explanations of the methods for conducting sensitivity analysis as laid out in the Ohio Handbook of Cost-Benefit Analysis.
Partial Sensitivity Analysis
Partial sensitivity analysis is in some ways the most basic of sensitivity analysis techniques. This technique is carried out by taking one key input and varying it to see how it impacts the results of the study. By showing how one factor impacts the outcome of a study, a policymaker can understand the risks involved in relation to a key factor.
An example is an input like the “value of a statistical life,” a valuation that has a range of different values depending on what agency is carrying out the analysis. In a study that includes an important input with varying valuations like the value of a statistical life, showing the net present value of the program under different assumptions for the value of a statistical life gives an insight into the variability of the results.
Worst- and Best-Case Analysis
Let’s move on from varying one input to varying multiple inputs. Sometimes when we conduct a cost-benefit analysis or another policy analysis, we may come up with a result that is very optimistic or very pessimistic. In this situation, we can conduct sensitivity analysis to test how reliant our results are on our assumptions.
To carry out this form of a sensitivity analysis, an analyst takes all the inputs and sets them to the most optimistic or pessimistic reasonable assumptions. This allows her to communicate to a policymaker what the policy’s outcomes would look like if all her assumptions are pessimistic and what they would look like if all her assumptions were optimistic. This process also allows her to test her assumptions and see if changing them impacts the ultimate findings of the analysis, determining if it has net costs or benefits under all circumstances or if it ends up above or below water depending on assumptions.
Breakeven Analysis
Some analyses are fixated particularly on finding if a policy has a positive net present value or a benefit-cost ratio above one. In these situations, it is useful to see how much assumptions need to change in order to see when benefits and costs “break even.”
Breakeven analysis is the process of varying assumptions to see where costs would equal benefits. This gives policymakers an understanding of how much the assumptions need to vary from their expectation for the policy’s benefits to exceed costs.This can be modified to broader policy analytic techniques if an approach is created to measure impact against each other: see what needs to be changed in order for two alternatives to be basically the same from an analytic perspective.
One way breakeven analysis can be useful is if the results are particularly robust. For instance, if a policy has a positive net present value unless the average wage for people carrying out the program exceeds $1 million per year, then that would suggest confidence that the net present value of the program is indeed positive.
Monte Carlo Simulation
Sometimes called the “gold standard of sensitivity analysis,” Monte Carlo Simulation is a more complex sensitivity analysis technique that requires data analysis software. The essence of a Monte Carlo simulation is to generate a large number of possible outcomes by varying all the assumptions in the analysis. Using these outcomes, confidence intervals for cost-benefit outcomes can be estimated. Advanced Microsoft Excel users can execute a Monte Carlo simulation with a little bit of help from Google.
Even conducting a simple partial sensitivity analysis can provide useful insights for a policymaker. The point of sensitivity analysis is to estimate precision of estimates, and using any of the above techniques makes an analysis more complete.