Last year, I wrote an article on problem definition, the first step of Eugene Bardach’s Eightfold Path to more effective problem solving. Problem definition is the first step in a good policy analysis, but when we think of what it is that makes up policy analysis, the second step, to assemble some evidence, feels like the beginning of the project.
The heart of assembling evidence is assembling data on the topic at hand, which consists of learning facts relevant to the problem you have defined. When data has meaning, we call it information, because it informs our understanding of a problem. Information then becomes evidence when it becomes valuable to people who are trying to understand a problem.
Bardach urges analysts to think before you collect data. He stresses that data collection can often feel productive when it is not: pulling together a bunch of data that does not inform or provide evidence for the analysis is ultimately not time well spent.
Economy is an important element of policy analysis. Since policy analysis usually occurs under a time limitation, making sure time is well spent is a valuable skill for a policy analyst. Economizing the evidence assembly step means understanding the value of data (how likely it is to become information and evidence) and understanding the utility of the data (how hard it is to collect it and how much time it will take). It also means leaning on educated guesses to guide your analysis: having good instincts to know what kind of data you may need to solve a problem can save a lot of time in this phase of analysis.
One key way data can be found is through review of the available research. Looking at professional journals, particularly policy and economics journals, can yield valuable evidence at this step. One source I go back to over and over again is the Washington Institute for Public Policy’s benefit-cost database.
Bardach also suggests analysts survey “best practices,” or look at how policymakers in other jurisdictions have solved this problem before. Note that just because a policy is being used other places does not mean it will work for your policymaker (or that it’s working at all!). Despite this, policymakers have noted in surveys that they highly value information about what other jurisdictions are doing. This comparative information can be valuable in a policy analysis on its own.
One way to deal with a unique problem is to use analogies. Bardach talks about how a policy analysis on merit pay in the public sector could use data from merit pay studies in the private sector. He also talks about how an analysis of how the state can discipline incompetent attorneys could draw from studies on how states discipline incompetent physicians. If you can’t find exactly the policy you’re trying to study, find policies like them and use them as a guide for gathering data and information.
Requests for data can be difficult. If you want data that has bureaucratic barriers, you may need time to get it. That is why you should start early when searching for evidence. The longer you wait, the less access to data you will have for your analysis.
Assembling evidence can often have a political component to it. People could be trying to protect a program or change it while you are trying to understand it. An analyst must touch base, gain credibility, broker consensus, and work with people in order to get access to the data they need and make it information and evidence that informs the policymaking process.
Lastly, don’t commit yourself to answers at the start of the analysis. Free the captive mind. Contact people you may disagree with and get data from them because they might send you in directions you don’t expect to go.
Assembling of data can be a fuzzy step and ultimately is one of the most iterated steps in the Eightfold Path. But doing it well means being critical, openminded, and efficient with your time. A good analyst knows how to do all these things, which is why assembly of evidence is so key to good policy analysis.