As policy analysts, we are often at the mercy of the research others have done in order to estimate some outcome. Rarely in this job do we have the time to sit down and research a problem the same way an academic does. Instead, we focus on finding creative ways to take the research that others do and apply it to the context that we are interested in understanding.
Unfortunately, academic research doesn’t always agree with itself.
Sometimes it’s a small difference, and the final results don’t really change all that much. But sometimes other research is directly contradictory, and our outcome will depend heavily on what estimate we choose to use.
In the case of a small difference, we could do something like use the average effect and wait until sensitivity analysis to explore the full range of outcomes. In the case where there are contradictory results, we need to make a decision.
Anytime we make a decision like this, we need to be careful to fully understand what the implications are and communicate them effectively. Here are a few things I try to think about when I am presented with conflicting information.
What context is most similar?
The first place I always begin when thinking about what research I want to use in my own projects is how similar is the context of my situation. It is more reasonable to think that the estimates others have come up with will more closely hold if we change fewer things about the situation in which we are applying them.
For example, when working on a project estimating the cost of climate change in Pennsylvania, it would be much easier to use climate research from places with similar climates today. Ideally, we’d want studies that look at climate change in Pennsylvania specifically, but if someone measured the cost of climate change in Ohio that would still be a useful piece of research.
Conversely, it would probably be incorrect to use estimates for the cost of climate change measured in Brazil. Even though those researchers might have come up with a very detailed causal equation that neatly ties increased temperatures to monetary losses, we know for a fact that the underlying assumptions about the climate are different there than in Pennsylvania.
Which estimate is the easiest to translate?
Another important consideration is how much work is it going to take to manipulate someone else's results and make them usable. I don’t necessarily mean how much effort it takes (though time is a limited resource and should always be considered) but rather how many steps and assumptions are required to use someone else’s result.
Each time we inflate a value, adjust for regional differences in prices, or change units, we introduce opportunities for estimates to become less meaningful. Some of these are more important than others, changing from pounds to kilograms for instance should not interfere with anything.
—
These are just two things to consider when deciding what estimate to include in a study. Ideally, during the sensitivity analysis phase we can explore and report how these different estimates would impact the results.