At Scioto Analysis, our mission is to provide policymakers and policy influencers with evidence-based analysis of pressing public problems. For us, this almost always means analyzing data and finding ways to quantify impacts.
However, there are still times when data isn’t quantifiable but analysis is still helpful. For example, when we want to draw insights from written comments on a survey.
In these instances, there are two main options we have as analysts. First, we can try to quantify these abstract data points. The main issue with this is that quantifying inherently means simplifying the data. When done well, key points can be preserved, but some amount of nuance is always lost when quantifying qualitative data.
The other option we have is to perform qualitative analysis.
As someone who went to school specifically for statistics, I am much more comfortable with quantitative analysis compared to qualitative analysis. It is possible even to take qualitative data and manipulate it into quantitative data that you can analyze.
That being said, the methods used to encode qualitative data are quite robust and can provide extremely useful information to analysts.
But, the more policy analysis work I do the more I have come to value the insights that qualitative analysis can provide when conducted well. Additionally, I think the additional cost that often comes with quantitative analysis is not always worth the additional insights it provides.
For example, we’ve recently been working on a project trying to better understand the impacts that climate change will have on health equity by midcentury. One of the main areas of focus for this project is on climate change related mental health issues.
Mental health is a notoriously difficult subject to quantify. Because there is such a wide spectrum of outcomes and there is not any tangible way to measure people’s experiences, quantitative analysis of mental health is almost always limited in its ability to give analysts useful information.
For this project in particular, a quantitative estimate of how climate change would affect mental health was unnecessary. There is a great deal of qualitative evidence from mental health professionals about how climate change affects mental health.
To gather some expert opinions on the topic, I spoke with a few mental health professionals about the subject. The context they were able to provide helped us convey the main message of the report without needing to get bogged down with math that would ultimately be unhelpful to the project as a whole.
One final consideration when comparing quantitative and qualitative analysis is who the intended audience of a study is. Quantitative methods can produce extremely robust results for use in tasks like budgeting by professionals who use math from day to day, but it might not be worth doing if you need to explain how you encoded peoples' written comments and performed a proportional odds logistic regression to a group that doesn’t have any background in statistics.
In this case, it might be more helpful to those people if instead of throwing hard-to-interpret regression results at them, you instead drew some qualitative insights that get the same point across.
Overall, I generally believe that quantitative analysis provides better insight into many questions than qualitative analysis. However, the more work I do in policy analysis, the more I have come to appreciate what qualitative analysis can provide. Understanding the pros and cons of both methods is important for anyone who wants to improve decision making in the public sector.