I started working at Scioto Analysis in September of 2022, which means I am approaching the end of my third full year with the practice. When I started, I was fresh out of grad school and notably had never done any policy analysis before. As I’ve mentioned before, I graduated with a degree in statistics, from a program that had a pretty heavy academic lean to it.
It was like going to school to learn all about the inner workings and uses of power tools, then getting a job as a carpenter. I had a deep understanding of the tools of policy analysis, but I didn’t yet know how to apply them.
As the end of the year is approaching, I wanted to take some time to reflect on these first three years and some of the lessons I’ve learned about being a good policy analyst.
Policy analysis is about problem solving
In Eugene Bardach’s A Practical Guide to Policy Analysis, the first step on the eightfold path laid out by the author is problem definition. We start this way because we understand that the world is not currently perfect, and it is the role of the public sector to help us move towards a more perfect world. This is different from the outlook of people in my grad school program, who largely wanted to pursue statistics to further our collective understanding of the field.
I want to be clear, I believe that what academics are doing is extremely important. We need people to push the frontiers of our knowledge and to try to improve our understanding of the world.
However, the insights gained from academia need to be applied to the real world for them to improve outcomes for people, and that is a job not always well suited for academics themselves. When we as analysts begin with a problem that we need to solve, it gives us a different lens to approach the findings generated by academics.
Policy analysis is client driven
An important memory I have from grad school was from a group project in my class called Statistical Consulting. Our goal was to do some analysis for a client that had a bunch of data and didn’t have the technical skills to analyze it.
My group was a mix of masters and PhD students, and one PhD candidate was insistent that we include a plot in our presentation showing how we found the optimal penalty for our lasso regression.
I suggested that we shouldn’t include this plot, because although it shows that our methods were valid, it didn’t mean anything to the client. I ended up being overruled by the group, and when we eventually gave our presentation we got the feedback that our presentation was too technical and we should have focused more on the results.
That kind of information is better suited to an appendix. Those that want to replicate that work or anyone who wants to ensure that we are checking all the necessary technical boxes should have access to that, but the information that matters is what is going to help someone make a better decision, and super technical things don’t make the cut.
Policy analysis should be collaborative
One of the main reasons I was drawn to statistics in the first place was I could not make a decision about what interested me the most. I generally like learning about different topics, and I thought that statistics was broad enough that it could enable me to do research on all sorts of topics, from biology to political science.
Policy analysis has a similar broad range of applications. My first ever project with Scioto Analysis looked at water quality in Ohio. The second project looked at the Genuine Progress Indicator, a broad economic replacement for Gross Domestic Product.
While this breadth of subject matter satisfies my desire to learn about different topics, it also largely means I’m not an expert in the thing I’m analyzing. I’m an expert in statistics and how to do policy analysis, but we often benefit from talking to people with more subject matter expertise.
Fortunately for us at Scioto, many of our clients are subject matter experts who are looking for assistance with policy analysis and statistics. That naturally leads to a collaborative process which leads to a better final product. When this doesn’t happen though, it is often smart to seek out people with subject matter expertise to help provide context.
—
It’s an exciting time in the world of policy analysis. The availability of good data and the abundance of statistical tools is making it more accessible than ever. The increased supply of analysis is being met with a rising demand, which is leading to better decisions being made by policymakers at all levels of government. At Scioto Analysis, we’re going to keep searching for ways to improve our analysis, as well as ways to get our information to policymakers more easily.
