In policy analysis, the primary goal is often to make a best guess as to what the impacts of a potential proposal are. Because this work is often on a pretty short timeline, the end result is often a single estimate for the effects of some policy, with much of the behind the scenes work failing to make it into the headlines.
In my opinion, the biggest downside of focusing on point estimates as the main result of interest is that they don’t communicate anything about how likely the result actually is. This is a question that can be answered by a well done sensitivity analysis.
Through sensitivity analysis, we get the opportunity to test the boundaries of our results. This allows us to see what happens in the best and worst case scenarios.
I would argue that this information is far more important than a point estimate when it comes to making informed policy decisions. Policymakers should be thinking about questions like what the probability is that we see an effect size of at least 10%, or how likely is it that an intervention breaks even.
These questions require a greater deal of probabilistic thinking which can be challenging. Still, these are the insights that can tend to lead to better decision making. A point estimate is a good talking point, but understanding uncertainty is the key to evaluating risk correctly.
Sensitivity analysis is also an important way to check and make sure results are credible. Point estimates can sometimes be misleading when the variance of an estimate is extremely high. Just because a point estimate is the best guess does not necessarily mean that the actual outcome of a policy intervention is likely to be close to it.
On the other hand, sensitivity analysis can help make the case for programs that have lower variance as well. Consider an example from Washington State’s Institute for Public Policy involving cognitive behavioral therapy (CBT).
One program is targeted at youth in state institutions, the other at moderate-to-high risk adults in the criminal justice system. The first program has a net present value of over $16,000 compared to just under $800 for the second. However, the first program is only expected to have positive value 68% of the time compared to 98% for the second.
Maybe the effect size of the first program is worth the added risk, or perhaps policymakers are more willing to take that risk with children compared to with adults. Either way, by understanding the full range of outcomes policymakers can make smarter decisions about how to allocate limited resources.
These are just a few reasons why policymakers should care more about sensitivity analysis than point estimates when evaluating the work of analysts. Point estimates are still important, and the fact that they are easier to communicate to the public should not be underestimated.
Still, in order to make the best possible decisions, policymakers need to understand uncertainty. It will lead to more effective decisions, and better outcomes for society as a whole.