I am currently working on a cost-benefit analysis on water quality in Ohio. After spending two weeks preparing and thinking about this current cost-benefit analysis, I am starting assembling some preliminary models. This step is often known as projecting the outcomes and is almost always the most difficult part of a cost-benefit analysis.
One of the main challenges when trying to project outcomes is fully understanding what sort of assumptions we as analysts have to make in order to make projections at all. Our assumptions define how we think about our analysis and they shape our results. Assumptions cannot be avoided in any sort of analysis, but as analysts it is our job to be fully aware of which ones we make.
Generally speaking, each additional assumption makes the analysis easier to perform and to understand, but it makes it more difficult to generalize the results. A common misconception is that having stronger assumptions makes analysis less accurate or less meaningful, but sometimes making additional assumptions actually improves our models.
This tradeoff is extremely common and is even described mathematically by the bias-variance tradeoff. In statistical terms, we evaluate estimators using a criteria called mean squared error which can be written as the sum of an estimator's variance and its squared bias. Decreasing one of those quantities often means increasing the other. In our context adding more assumptions simplifies the models and reduces the variance, but because we are assuming more things we may be adding some bias to our results. It is ok to keep some bias, especially if we understand that it exists and it lowers the variance of our estimate making it more useful.
At the beginning of my model building, I chose to make as many assumptions as I had to in order to get some preliminary results. It may not make sense to assume that certain things will remain constant well into the future, but because this is an iterative process it is important to get a working model to build off of. Building a simple model can give you a general idea of potential results and can give you an answer which then you can build from by refining your model.
Making assumptions is also useful because they help you identify what things you need to learn more about. After going through and making all of the first models, it is important to ask which assumptions can I potentially get rid of. Does removing these assumptions make the model sufficiently more generalizable?
As analysts, it is our job to make clear that predicting the future is extremely difficult. I once heard someone say that making predictions is like trying to drive a car by only looking in the rearview mirror. Sometimes trying to work with real data forces us into making strong assumptions. Still, by using the best available methods and fully understanding the ramifications of the assumptions we make, we can help policy makers decide on the best course of action with better information than they would have on their own.
This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.