What happens when policy analysts disagree?

In public policy analysis, competition can lead to higher-quality analysis. The best way to spur competition between analyses is not completely clear, however.

Often, at the state level, budgeting is driven by a competition of projections between the executive and legislative branches.

For instance, in the state of Ohio, the Governor’s Office of Budget and Management makes projections for what tax revenues for the state will be over the upcoming year. The Legislative Service Commission, the research arm of the Ohio General Assembly, makes their own estimates.

When I worked for the legislature, I was told legislative leaders usually used the Office of Budget and Management’s projections rather than the projections from the Legislative Service Commission. This was because the projections from the Legislative Service Commission tended to be more conservative than the projections from the Office of Budget and Management, even though the Office of Budget and Management projections were usually less accurate.

An alternative to presenting two separate forecasts and having policymakers decide which forecast to use is to create some sort of structure for creating consensus between forecasters.

The state of New York, for instance, has a system for creating a consensus report for its budget forecast between its executive and legislative branches. The state holds a consensus budgeting conference and members of the legislature and executive branches agree on a forecast based on testimony.

The essential tradeoff policymakers have when making policy that is relevant to policy analysts is the tradeoff between time and information. From the analyst perspective, the tradeoff comes from giving up the amount of analysis a policy analyst is able to do. But from a policymaker perspective, they often need to take competing opinions and decide which one to follow.

One way to think about this problem is to think about the policymaking process as one that encompasses policy analysis. Policymakers (those who make the decisions about policy) hire policy analysts to understand policy and to give them information on the impacts of public policy. It is then up to policymakers to confront tradeoffs between different public policy goals (like effectiveness, efficiency, and equity of competing policy options) and to make a decision.

If a policymaker has competing sources of information, which can be thought of as competing policy analysts, then the policymaker has to find a way to evaluate these policy analysts against one another.

One way a policymaker can evaluate policy analysts is through ethos heuristics. Basically this is asking the question “who do I trust the most?” This is driven by perceptions of objectivity and prestige of analysts. It also may be driven by whose “side” the analyst is on: a legislator is more likely to trust a legislative analyst than an executive office analyst, all other things being equal.

Another way is through a process heuristic. The example of the “use the conservative number” heuristic used by the Ohio General Assembly is an example of this. This can be useful if you have a good heuristic to fall back on that makes sense with the policy goal.

Consensus budgeting provides a different approach. It leans on our intuitions about “wisdom of the crowd,” hoping that good analysts will be able to fall back on reason and information when given the choice. Some states are deploying this approach with promising results.