When I tell people I do policy analysis, one of the most common questions I get is this: “Do politicians even care about what the research says?”
It’s a fair question, and usually I respond with anecdotes from history or from my personal experience that show that (at least sometimes!) policymakers do listen to research.
New research published in the National Bureau of Economic Research’s Working Paper series is now lending experimental credence to this claim. Researchers at Harvard, Columbia, UC Davis, and Innovations for Poverty Action collaborated with the National Confederation of Municipalities in Brazil to run two large experiments on policymakers’ demand for and use of policy analysis.
The first experiment involved 900 municipal-level officials across 657 municipalities in Brazil and was conduced at a series of Nation Confederation of Municipalities meetings across the country. The study gave local officials surveys on tablets with information on early childhood education programs, which are funded and administered at the municipal level in Brazil. The officials were then given lottery entries for a trip to Boston and a private tour of Harvard University. They were given the option to “purchase” information on the impact of early childhood education programs on outcomes like test scores and cognitive skills down the road by trading in these lottery entries for the results of peer-reviewed studies.
So did the local officials play along, trading their opportunity for a trip to Boston for information on the impact of early childhood education? Actually, they did. The average official was willing to trade a number of lottery entries for the trip, on average amounting to a value of $36 US. This means that officials, learning about a topic they did not seek out, were willing to pay the equivalent of $36 on average to find out more information on that topic. The study also found that policymakers were willing to pay more for the results of studies with larger sample sizes and changed their beliefs about the impact of early childhood in line with the research presented after exposure to results.
The second study randomly invited 881 Brazilian mayors via email and text message to a session on reminder letters for tax compliance at a large National Confederation of Municipalities meeting. Tax compliance is an important issue in Brazil where some estimate that about 20% of property taxes go unpaid every year. Reminder letters are also a low-cost and administratively simple policy to implement with a strong evidence base of its effectiveness, making it a strong candidate for a policy.
Of the invited mayors, 38% attended the session.[1] An additional 1% of uninvited mayors at the conference attended the open session as well. At the session, an instructor presented on policy impact, cost-effectiveness, and impact evaluation research around taxpayer reminder letters, then gave an overview of the literature, a list of characteristics of effective letters, and sample letters. Both treatment and control municipalities were interviewed 15 to 24 months to see if they were using reminders for tax compliance more than a year after the conference.
It turns out that the session had a significant impact on participants compared to the control group. Mayors who attended the session were 33% more likely to be sending tax compliance reminder letters 15-24 months out than mayors that did not attend the session. They also were more likely to have beliefs in line with the research 15-24 months than control mayors who did not attend the session.
There are reasons to still be skeptical. Maybe Brazil has a different political environment than the United States (though their most recent national election suggests the two countries have at least one thing in common) that makes them more amenable to research than the United States. Maybe these issues just work because they’re below the “policy waterline” as we’ve wrote about here before.
Despite these objections, this study does show one thing: there are certainly times when research and analysis matters and policymakers actually want it. More sources of rigorous, nonpartisan analysis will only improve policymaking at all levels of government, including in states and localities.
[1] Notably, younger and college-educated mayors were 7 to 15 percentage points more likely to attend the session.