How do you know if your strategic plan is working?

If you are a nonprofit executive, you are certainly familiar with the idea of a “strategic plan.” A group of board members get together, sometimes for a day or a weekend, to chart out a plan for the organization for the next few years.

While strategic planning has become a profession in itself, what comes after the strategic plan can be where things break down for an organization. And this can happen even if the organization is successful at executing all the strategy decided upon during strategic planning. This is because strategic planning tends to focus on what an organization hopes to accomplish. If an organization wants to know if their activities are actually bringing those outcomes to fruition, they need to get into the business of evaluation.

But how do we go about evaluating our success at executing a plan? Here is one four-step framework you can use for evaluating your organization’s success in light of their strategic plan.

Step 1: Translate strategic goals into measurable outcomes

Quantify, quantify, quantify. It is easy to walk away from strategic planning jazzed about the potential future outcomes, but if nothing can be measured from those efforts a few years later, then how do you know that you did anything in the first place?

I am going to use an example of a hypothetical nonprofit whose board has decided that “improving housing stability” will be the focus of its strategic plan. What are a few ways that this strategic goal can be translated into measurable outcomes?

The economist in me focuses on housing cost burden: take the population the nonprofit is serving and see how many people the organization can get to where their housing costs are less than 30% of their total income. You could also use outcomes like reducing rates of homelessness, reducing evictions, reducing nonpayment of rent, or increasing length of stay at a given home.

Sometimes these are laid out in a strategic plan. Often they are not. If they are not, it may take collaboration between board members, executives, and clients to decide exactly which quantitative measures best capture the goals and resources of the organization. An organization working to reduce housing cost burden could undertake very different activities than an organization working to reduce homelessness.

Step 2: Map activities, outputs, and outcomes

For those of you who are familiar with logic modeling, this is probably ringing a bell to you. A good way to then connect your organization’s work to your strategic plan’s goals is to map activities to outputs then eventually outcomes.

Activities are the specific programs that are carried out by an organization. Outputs are the direct results of those activities. Outcomes are the broader impacts that come about from outputs that are not directly controlled by the organization but that the organization wants to bring into being.

So for instance, our hypothetical nonprofit could have adopted the goal of promoting housing stability, then the executive director could have determined that “reduce the number of people experiencing homelessness” is a good measurable outcome that follows the goals of the board. The organization then undertakes an activity of providing permanent supportive housing to homeless residents of a target community. The output would be the number of people housed by the program. The outcome of interest would be the number of previously-homeless people who then do not experience homelessness over the upcoming year.

As you can see, the activities of the organization bring about outputs, which then bring about outcomes that tie into the organization’s strategic plan.

Step 3: Estimate impact

It would be a mistake at this point to say that we have now determined how successful the organization has been at carrying out the strategic plan. This is because no nonprofit provides services in a vacuum. Ultimately, the goal of a nonprofit organization is not to just be associated with outcomes, they want to make these outcomes happen.

Let’s make this a little more concrete. So say our hypothetical nonprofit program director was choosing people to stay in its permanent supportive housing program. She finds a group of people who have above-average income who recently lost their homes to a hurricane. They are living in fairly nice hotels right now, subsidized by support from their local government. You would imagine once these people get housing again, they will likely be housed continuously for the next year. If she brings those people into her permanent supportive housing, the outcome of homelessness will not show up, but it likely had little to do with their actions. These people would probably have overcome their homelessness anyway.

This is why approaches like randomized controlled trials and quasiexperimental methods are so vital to social programs. An effective nonprofit goes beyond outcomes to determine the real outcomes brought about by their work.

Step 4: Use evidence to improve decisions

If you have done step 3, you now have an incredible tool: knowledge of what does (or does not) advance the goals of your strategic plan. With this information, you can tune up programs to make them more effective, make the case to funders about programs that are already successful at a smaller scale and could use more resources to scale up, or reallocate funds from programs that do less to advance the goals in your strategic plans to goals that are working more.

Consider our hypothetical nonprofit. If they evaluate their permanent supportive housing program and find that it reduces homelessness by 25% against a control group of similarly-situated people, the executive director may want to put more resources into the permanent supportive housing program than on more treatment-focused programs that aren’t showing much success in reducing homelessness.

A good strategy for nonprofits is to put a lot of evaluation dollars into new programs, less into programs that are tested already on a smaller scale, and a minimum amount of evaluation on large programs that have been well established. This ensures that programs with strong evidence are scaled up and new approaches are tested for effectiveness against the goals of the organization.

Make evaluation a part of your strategic plan

Ultimately, a strategic plan without evaluation is a plan without teeth. What is measured is what matters, and if activities are living in a world divorced from their outputs, outcomes, and the reality on the ground, then the strategic plan is an aspirational document and a wasted Saturday for busy board members. On the other hand, an executive director can take a strategic plan, turn it into measurable outcomes, map activities onto outputs then onto those outcomes, find the real impact of those activities, then use them to advance the plan her board of directors set in motion. And that is what I would call an effective strategic plan.

How are data centers impacting Cincinnati, Cleveland, and Columbus?

A couple of weeks ago, Scioto Analysis published a cost-benefit analysis of data center policies in Ohio. We found that data centers in Ohio produced a net social benefit of $62 billion between 2013 and 2026. Between 2027 and 2034, however, we project that they will produce a net social benefit of only about $2.4 billion. This is partly because data centers have strong early employment effects: a county gains most of its jobs when its first data centers are developed, and later ones add relatively few. At the same time, newer data centers are more energy intensive, so electricity use grows faster than the number of facilities. As electricity demand grows, so do the strain on the grid and the emissions from power generation. That raises costs for ratepayers and harms public health.

We estimate that 178 data centers currently operate in Ohio. If all planned projects come online by 2034, that number will rise to 294. Many of these facilities are concentrated in particular parts of the state. To better understand the benefits and costs that different areas of Ohio will see from data centers over the next several years, I wanted to look at our cost-benefit results for major metropolitan areas across Ohio.

Columbus MSA

The Columbus Metropolitan Statistical Area consists of ten counties:

• Delaware County

• Fairfield County

• Franklin County

• Hocking County

• Licking County

• Madison County

• Morrow County

• Perry County

• Pickaway County

• Union County

Across those ten counties, there are 106 existing data center facilities and 47 planned or proposed data center facilities between now and 2034. We estimate that these 106 data centers consume about 37 million megawatt-hours of electricity per year, an average of 350,000 annual megawatt-hours per facility. This amount of electricity could power around 3.4 million homes per year, nearly four times the number of households in the Columbus Metropolitan Statistical Area. 

By 2034, we estimate that data centers in the Columbus Metropolitan Statistical Area will consume about 89 million megawatt-hours of electricity per year, an average of 580,000 annual megawatt-hours per facility. This amount of electricity could power over nine times the number of households in the Columbus Metropolitan Statistical Area.

With no policy intervention, we estimate that data centers within the Columbus Metropolitan Statistical Area have generated about $18 billion in net social value since 2013, mostly due to around 30,000 jobs supported. However, between now and 2034, we estimate that data centers within the Columbus Metropolitan Statistical Area will generate about $2 billion in net social loss, mainly due to higher electricity prices and emissions associated with  electricity generation.

If future data center projects are banned within the Columbus Metropolitan Statistical Area, we estimate the net social value of data centers in the region to be about $8.1 billion between now and 2034. If future data center projects are required to generate electricity behind-the-meter, with 90% renewables and 10% natural gas, we estimate the net social value of data centers in the region to be about $8.2 billion between now and 2034, slightly higher than a ban on construction.

The table below shows the future impacts of data centers in the Columbus Metropolitan Statistical Area under each scenario.

Cincinnati MSA

The Cincinnati Metropolitan Statistical Area consists of fifteen counties, five in Ohio. There are also ten counties in the Cincinnati Metropolitan Statistical Area in Indiana and Kentucky, but we didn’t track data centers in counties outside of Ohio for this study.

• Brown County

• Butler County

• Clermont County

• Hamilton County

• Warren County

Across those five counties, there are 24 existing data center facilities and 3 planned data center facilities between now and 2034. We estimate that these 24 data centers consume about 6.1 million megawatt-hours of electricity per year, an average of 250,000 annual megawatt-hours per facility. This amount of electricity could power around 570,000 homes per year, over half the number of households in the Cincinnati Metropolitan Statistical Area. 

By 2034, we estimate that data centers in the Cincinnati Metropolitan Statistical Area will consume about 9.7 million megawatt-hours of electricity per year, an average of 360,000 annual megawatt-hours per facility. This amount of electricity could almost power every home in the Cincinnati Metropolitan Statistical Area.

With no policy intervention, we estimate that data centers within the Cincinnati Metropolitan Statistical Area have generated about $21 billion in net social value since 2013, mostly due to around 24,000 jobs supported. Between now and 2034, we estimate that data centers within the Cincinnati Metropolitan Statistical Area will generate about $11 billion in net social value. The employment benefits from data centers in Cincinnati will slow down, but the utility and public health related costs aren’t as bad in Cincinnati as they are in Columbus due to slower data center growth.

If future data center projects are banned within the Cincinnati Metropolitan Statistical Area, we estimate the net social value of data centers in the region between now and 2034 will increase to $12 billion. If future data center projects are required to generate electricity behind-the-meter, with 90% renewables and 10% natural gas, we estimate the net social value of data centers in the region to be about the same as a ban on construction, about $12 billion between now and 2034.

The table below shows the future impacts of data centers in the Cincinnati Metropolitan Statistical Area under each scenario.

Cleveland MSA

The Cleveland Metropolitan Statistical Area consists of six counties:

• Ashtabula County

• Cuyahoga County

• Geauga County

• Lake County

• Lorain County

• Medina County

Across those six counties, there are 12 existing data center facilities and 1 planned data center facility for 2028. We estimate that these 12 data centers consume about 440,000 megawatt-hours of electricity per year, an average of 36,000 annual megawatt-hours per facility. This amount of electricity could power around 40,000 homes per year, only around 4% the number of households in the Cleveland Metropolitan Statistical Area. 

With no policy intervention, we estimate that data centers within the Cleveland Metropolitan Statistical Area have generated about $21 billion in net social value since 2013, mostly due to around 20,000 jobs supported. Between now and 2034, we estimate that data centers within the Cleveland Metropolitan Statistical Area will generate about $12 billion in net social value. 

The Cleveland Metropolitan Statistical Area only has one planned data center facility between now and 2034; however, this singular data center facility is massive, so its impacts are meaningful.

If future data center projects are banned within the Cleveland Metropolitan Statistical Area, we estimate the net social value of data centers in the region between now and 2034 will decrease to slightly below $12 billion. If future data center projects are required to generate electricity behind-the-meter, with 90% renewables and 10% natural gas, we estimate the net social value of data centers in the region to increase to about $13 billion between now and 2034.

The table below shows the future impacts of data centers in the Cleveland Metropolitan Statistical Area under each scenario. 

What does this mean for Ohio?

If we add up the future net present values from the Columbus, Cincinnati, and Cleveland metropolitan statistical areas, we get about $21 billion in net social value over the next several years. This is much higher than the $2.4 billion net present value we calculate for the state of Ohio as a whole. This finding suggests that outside of Ohio’s three largest metropolitan areas, the future costs of data centers exceed the future benefits.

The main reason behind this is that the majority of employment benefits from data centers in Ohio are felt in Ohio’s three largest metropolitan statistical areas. According to Yue and Zeng in 2026, our source for the employment impacts of data centers, the statistically significant employment benefits of data centers are concentrated in metropolitan areas.

However, it’s also possible that employment benefits are overestimated in these core metropolitan areas. As we discuss in our full cost-benefit analysis, employment benefits are highly uncertain, while emissions costs are much more certain.

In areas where data center growth is slow over the next several years, it doesn’t appear that the costs begin to outpace the benefits of data centers. In metropolitan areas like Cleveland or Cincinnati where there are only a handful of planned data center projects, the sustained employment benefits continue to outpace the costs from electricity consumption and emissions. 

On the other hand, in metropolitan areas that are experiencing rapid growth in data centers, like Columbus, we predict that the costs from electricity consumption and emissions will begin to outpace data center benefits. Local policymakers may want to consider policies such as requiring behind-the-meter electricity generation with a high share of renewable energy sources to minimize the costs of future data centers while maintaining the benefits.

What do you do when there are data or methodological limitations in a policy analysis?

While I was still in school, I worked as a guide on wilderness expeditions. As part of that job, I was a certified Wilderness First Responder. In that training there is a mantra about working from “ideal to real.” We’d love to work in sterile, safe environments all the time but if someone needs help while sitting in the mud during a rainstorm that's where you need to help them. Policy analysis is similar.

One of the challenges of policy analysis is its reliance on external factors for its success. Things like demographic data, effect size estimates, or methodological frameworks are all prerequisites for a strong policy analysis. While it would be extremely nice to live in an ideal world where all the data and methods are well established and credible, our actual job is to provide information in the real world when conditions are often subpar.

There are many times when analysts don’t have all the ideal prerequisites they need for an analysis. Maybe a dataset doesn’t include the population we are interested in, or maybe the policy we are studying has never been implemented in the state where we are working. 

These limitations don’t mean we can’t provide useful information or that an analysis is impossible. Situations where we have to work around these kinds of limitations are fairly frequent. Instead, we have to think carefully about what the available evidence can tell us and what assumptions are needed to fill in the gaps. 

What can we do when we don’t have the right data?

Data concerns are always at the front of mind when conducting a policy analysis. All statistical models are threatened by the “garbage in, garbage out” principle. Understanding how to work around these limitations is a key part of quantitative policy analysis.

One way this manifests is a situation where we have good data on the implementation of a policy in one context (say a neighboring state), but we lack the information needed to estimate what the impact will be in the context we are studying.

Imagine we are studying a rental assistance program that has been implemented in a different city. The city we are studying is smaller than the city with the existing program, so we lack some of the requisite data needed to help us estimate uptake. As a starting point, we could compare the number of renter households in each city and scale the program size accordingly. If our city has half as many renter households, we might initially estimate that a similar program would have half as many participants.

This simple model requires a lot of underlying assumptions to hold. There must be a similar share of renter households who would qualify for and participate in the program. These households must have characteristics that imply that they would experience similar outcomes. If we have information on things like differences in income or housing cost burdens, we might be able to refine that estimate. If we don’t, we need to acknowledge that our estimate relies on those populations being sufficiently similar.

If there is no suitable data available and there isn’t a credible way to adjust an estimate from another context, then we might have to rely on describing outcomes qualitatively. When providing a qualitative description, it is important to distinguish between an outcome we have evidence to expect and one that is simply possible. We might be able to explain how rental assistance could help households avoid eviction without estimating how many evictions it would prevent. That still provides useful information, as long as we explain the evidence supporting that relationship and why we cannot confidently estimate its size.

What if the available methods don’t answer our question?

As analysts, we try not to reinvent the wheel as often as possible. If a method has been shown to work well before, our first instinct is to replicate and adapt rather than search for a novel approach. Established methods give us a starting point for organizing an analysis and understanding its limitations. However, we still need to check whether their requirements and assumptions fit the question we are trying to answer.

There are of course situations where there aren’t existing methods that directly address our question. When this happens, we have to find a creative way to understand the question at hand.

It can often be helpful to start thinking through the problem logically. Say we don’t know how this rental assistance will impact health outcomes directly. We could perhaps get around this limitation by estimating the impact it has on net household incomes, and then making the connection between higher net household incomes and better health outcomes.

Why is it important to explain our assumptions?

Every time we work around one of these limitations in data or methodology, we increase the number of assumptions we are making with our models. While this is not inherently bad, it does expose us to more variability and it requires us to be more detailed about the assumptions we make and what their implications may be.

It is essential that we make all of these choices visible. It should be readily apparent in the narrative of a report what parts of the model depend on our assumptions. It can also be helpful in some situations to explain qualitatively why we believe our assumptions are reasonable and where we have less confidence.

This matters because policymakers may reasonably disagree with an assumption while still finding the analysis useful. If they can see how we arrived at a result, they can better judge whether it applies to the decision they are facing. An estimate without that explanation can appear more certain than the evidence actually supports.

How can sensitivity analysis help?

Disclosing an assumption is an important first step, but we also want to know how much it matters. Sensitivity analysis allows us to change assumptions and see what happens to our results.

Returning to the rental assistance example, we could estimate outcomes assuming the program is less effective than the study suggests, then compare those results with our main estimate. We could also test different participation levels or program costs. 

Note that when we do sensitivity analysis we aren’t just seeing what happens if our model is wrong. The goal is to understand how our assumptions affect our conclusions. When we say that a policy is more or less effective, that should always be based on the best available evidence about the variation in outcomes.

Sometimes these changes have little effect on the overall conclusion. Other times, a relatively small adjustment changes whether a policy’s estimated benefits exceed its costs. Both findings are useful. The first suggests that the conclusion holds across the conditions we tested, while the second tells policymakers that the decision depends heavily on something we don’t know with much confidence. Sensitivity analysis cannot make weak evidence stronger or account for every possible uncertainty. Its goal is to communicate the uncertainty in our results. 

As with wilderness medicine, policy analysis is about working from ideal to real.  It would be so much easier if there was always high-quality data and highly rigorous methodological frameworks to fall back on, but often there aren’t. If we want policymakers to have better information to inform their decisions, then we need to be able to work with what we have. There are situations where these concerns make some aspects of an analysis impossible, but our goal should always be thinking about “ideal to real.” What questions can we answer with the data and methods we have available?

What kind of analysis could I get for $40,000?

Often, governments, nonprofits, foundations, and associations have budgets to answer important questions but don’t know exactly what kind of study they need to answer them. I use $40,000 as an example of a typical analysis because it is enough to answer a significant policy question but requires an agency or organization to make decisions about what sort of analytical approach they want to use.

This article explores four examples for what a policy analysis firm could do with $40,000. These are illustrative scopes, not fixed-price packages. But these are all approaches we have taken with $40,000 or less to great success on a range of different topics.

Option 1: Understand the economic impact of an industry or project

Are you asking a question like “how much does our industry contribute to a local economy?” If so, economic impact analysis could be the right approach for you.

Economic impact analysis is an approach to economic analysis that focuses on jobs, earnings, economic output, tax revenue, and other activity associated with a given industry or project. A $40,000 economic impact analysis project could pay for an analysis of an industry including its impact on regional Gross Domestic Product, jobs supported, earnings supported, and tax revenue generated.

This kind of study could include those results, a methodology explaining how the results were calculated, analysis of the results, and a report ready to be distributed to the public. This gives the client evidence to inform economic development and investment decisions and information about how to explain the impact of an industry to policymakers. A limitation of this approach is that this kind of analysis rarely can establish what would happen if an industry or project didn’t exist and it generally does not analyze the social impacts of policies.

Option 2: Determine whether social benefits of a policy outweigh social costs

If you are asking a question like “would expanding this program generate more benefits than it costs?,” then a cost-benefit analysis may be what you are looking for.

Cost-benefit analysis is the process of assessing all the social impacts of a policy, quantifying those impacts, then monetizing them to see what society’s aggregate willingness to pay for the intervention is. An exploratory cost-benefit analysis using existing data sources can be done for $40,000.

In a cost-benefit analysis, we can estimate what the benefits or costs of a policy are, the net benefits of a policy, the benefit-cost ratio, and conduct sensitivity analysis to see how the assumption about the policy affect its results. Clients can use this to learn whether a policy creates social benefits and under what conditions it does. Some limitations of this approach are that cost-benefit analysis often lends itself to analyzing specific policies and that extensive primary data collection is not usually possible with this approach.

Option 3: Compare policies for fighting a public problem

If you are asking a question like “what should we do to most effectively solve this problem?,” then policy analysis could be the approach for you.

Formal policy analysis compares policy options against consistent evaluative criteria. Policy analysis begins with defining a problem rigorously then moves into assembling evidence on the problem, constructing options to ameliorate it, defining criteria, projecting outcomes on those criteria, then confronting tradeoffs and telling the story.

A policy analysis generally includes an assessment of each policy option, a comparison of how policies bring about certain outcomes, identification of tradeoffs between policies, all written into a report on the findings. Clients walk away from a policy analysis with evidence to help policymakers choose among competing policy approaches. One limitation is that a $40,000 project likely has to rely on existing high-quality research to make estimates about a given policy since extensive data collection is difficult to do with a budget like this. 

Option 4: Benchmark a community against its peers

If you are more interested in questions of comparison like “how does our community compare to other communities and where are we falling behind,” then community benchmarking may be what you are looking for.

Benchmarking uses a series of indicators to assess how a community compares to others in community conditions and performance. A $40,000 community benchmarking report could include choosing comparison communities, collecting data, and compiling data into a digestible and sharable report.

A report like this could include the indicators themselves, trend analysis, visualizations, and analysis of which indicators make a given community unique. This can be helpful to clients who want a clearer picture of local conditions for the purpose of strategic planning, making decisions about investments, and evaluation of the community in the future. A limitation of this approach is that with this kind of budget, we usually have the ability to talk about what a community looks like without the ability to say why it looks like this.

It’s also worth mentioning that benchmarking doesn’t have to just be communities. It can also apply to organizations.

Which type of analysis should I choose?

The type of analysis an organization should choose is based on the decision you are trying to make.

If you are trying to understand the economic activity associated with an industry, organization, or project, economic impact analysis is the best tool for you. If you are trying to understand if the economic benefits of a policy will outweigh its economic costs, cost-benefit analysis is the best type of analysis for you. If you are trying to evaluate the relative benefits of competing approaches to solving a problem, policy analysis is the best tool for you. If you need to understand the conditions, trends, or performance of your community versus others, data analysis and benchmarking will be the best approach for you.

These are analytical archetypes. Different analyses can combine techniques from different analytical frameworks. But limited budgets tend to limit the ability of answering all of these questions, so prioritizing which questions are most important to you and your organization is key to doing a thorough analysis with a given budget. Your most important job is to find out which question you most need answered, then analysts can help you understand the best way to go about answering them.

A $40,000 budget can support a range of different kinds of helpful analyses. There is no universally correct way to allocate those funds. It all depends on the question you are trying to answer. The most useful engagement for a policy analysis firm is when they can work with a client who needs evidence to help make decisions. If you want to learn more about costs associated with policy analysis, check out another blog I wrote on the topic. And if you are trying to figure out what kind of analysis you need, email me. I’d be happy to have a conversation with you about the question you are trying to answer and what can be answered with your budget.

Majority of Ohio economists agree that vehicle miles traveled fee could increase road financing efficiency

In a survey released this morning by Scioto Analysis, 16 of 31 economists agreed that a vehicle-miles-traveled fee would be a more efficient way to fund Ohio's roads than the motor-fuel tax.

Earlier this month, Governor Mike DeWine signed House Bill 519, which halts collection of Ohio’s gasoline and diesel fuel tax through January 2, 2027. Ohio’s fuel tax is 38.5 cents per gallon of gasoline and 47 cents per gallon of diesel fuel. Gas prices have since decreased by 35 to 47 cents per gallon across the state. Ohio’s fuel tax is mostly used to construct and repair roads.

16 of 31 economists agreed that as fuel efficiency and electric-vehicle adoption increase, a vehicle-miles-traveled fee would be a more efficient way to fund Ohio's roads than the motor-fuel tax. According to David Lindequist of Miami University, “Mileage fees reflect road use more directly than fuel taxes and they would also include electric vehicles. They could lead to more efficiency assuming collection costs remain manageable and emissions are addressed separately.” Of the remaining economists, 11 disagreed and 4 were uncertain. David Brasington of the University of Cincinnati was uncertain, explaining, “If you use GPS tracking of Ohio drivers, it would be feasible, but people would probably have big privacy concerns. Other ways of implementing a miles-driven system have high costs of administering the system compared to a gas tax.”

18 of 31 economists disagreed that Ohio's annual registration fee on electric vehicles is an efficient way to make up for lost fuel tax revenue. According to Jonathan Andreas of Bluffton University, “This lump-sum fee provides zero disincentive to drive more. It also overcharges low-mileage drivers and undercharges high-mileage ones. Anyhow, a typical EV produces a fraction of one thousandth as much road damage as a fully-loaded semi-truck, so most of the cost of road maintenance should be borne by heavy vehicles like trucks which would make our overall transportation system more efficient by moving more freight to railroads who already pay for all of their own (rail)road maintenance unlike the trucking industry which is heavily subsidized by taxpayers for their road usage.” Of the remaining economists, 8 agreed and 5 were uncertain. 

17 of 31 economists agreed that Ohio should rely more heavily on user fees than general tax revenue to finance highways. Roman Sheremeta of Case Western Reserve University indicated, “People who use roads more, and heavier vehicles that damage them more, should pay more.” Of the remaining economists, 9 disagreed and 5 were uncertain. Bill LaFeyette of Regionomics disagreed, explaining, “As electrical vehicles proliferate and fuel efficiency improves, gas taxes move further away from the total needed to preserve our deteriorating roads. Roads benefit everybody, not just those who drive on them.”

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists. Individual responses to all surveys can be found here.

What Makes Public Revenue Sustainable?

Last year, Scioto Analysis was hired by the City of Columbus and Franklin County to make revenue projections for the Funding Review Advisory Committee. This committee was a city/county effort to make recommendations for the two local governments to finance human services, tourism, and arts programs through midcentury.

Earlier this month, I spoke to the Columbus City Council on the work we did, explaining our approach and how the recommendations should be used by the Council. During this testimony, Councilmember Lourdes Barroso de Padilla asked me a question about sustainability of funding streams. She noted that the City's hotel tax had originally been intended for a certain purpose but that over time, it came to be used for a range of other purposes as the City identified new community needs. Here, she talked about a key tension in municipal finance: how do you ensure revenue streams are sustainable over time?

What does “sustainable revenue” really mean?

“Revenue sustainability” can seem like an abstract concept, but I will offer three cornerstones of a sustainable revenue stream. First, there is adequacy: does this revenue stream provide enough revenue for the identified need? Second, there is stability: will the revenue stream bring in a predictable amount of revenue each year? Lastly, there is growth: will the revenue stream keep pace with future needs.

One reason broad-based taxes like income and sales taxes are favored by economists is that they tend to meet all three of these criteria. A large amount of revenue can be adequately collected, they tend to be stable year-to-year, and collections grow with the economy.

This also helps understand some of the appeal of user fees like gasoline taxes used for maintaining roads. Because people who buy gasoline use roads, a properly-set gasoline tax will adequately fund road maintenance, will be stable relative to road usage, and will grow as road use grows. This relationship, though, it becoming more fraught with the increasing popularity of fuel-efficient and electric vehicles.

An example of a revenue stream running into problems with this framework is Cuyahoga County’s cigarette tax. While the tax once adequately financed arts, people in the arts community are now starting to sweat as smoking rates decline, creating a growth problem for the fund. Using these funds for programs like smoking cessation programs would track growth of the revenue stream more closely.

Can multiple revenue sources make revenue more sustainable?

Another question Councilmember Barroso de Padilla asked was around the number of revenue streams needed for sustainable funding of community priorities. She tapped into an important insight here: relying on a single funding source can lead to problems in funding stability. Columbus hotel/motel tax collections fell from approximately $23.7 million in 2019 to $10.1 million in 2020, a decline of roughly 57%. Putting together a package of revenue streams is one way to make revenue more sustainable. This is very similar to individual investing: governments generally do better when they diversify their revenue portfolio. 

Diversification does not guarantee revenue stability, however. Sometimes revenue streams will dip at the same time. For instance, say a city council adopted a hotel tax, a rideshare fee, and a car rental fee at the same time. This would be three different revenue streams that would likely take hits at the same time. Think of 2020 again: if people aren’t staying at hotels, they’re probably not using Uber and renting cars as much as well. 

How can governments protect dedicated funding?

A big issue Councilmember Barroso de Padilla raised was how a dedicated fund (in her example, the city hotel tax) can continue to support its original purpose.

Ultimately, this is not an economic question, but a political one: how can a fund be structured in a way that protects it from changing political priorities? Governments have a few options for doing this. Governments can earmark funds for certain purposes: write into the law that a certain percentage of revenue needs to go to certain purposes. They can do this with an allocation formula that puts a certain percentage of revenues toward certain projects. They can make a fund a dedicated fund that goes toward certain purposes. And they can conduct periodic reviews like that which the Funding Review Advisory Committee did to review how funds are used and whether they are still sustainably meeting community needs.

There is a tension here. Recipients of funds, whether they are agencies or community partners, benefit from reliable public commitments. On the other hand, policymakers need flexibility to respond to emerging priorities. Because of this tension, there is no “right” way to design a funding stream.

How should policymakers evaluate revenue options?

With so many options to finance public services before them, it can be difficult for policymakers to think through which options to pick. I will put forth some criteria for policymakers here.

First, consider the sustainability of the revenue option. This means not only evaluating the adequacy of revenue collections (“how much will this raise?”) but also their stability (“how volatile are these collections year-to-year?”) and their growth (“will this continue to meet needs a decade from now?”).

Then, consider the economic impacts of the revenue option. How efficient is the policy, meaning how much will the revenue option distort economic decisions, and what economic costs will those distortions create? Who will pay for it: will it fall more on residents of your community or other communities? And how regressive will the revenue option be: how is the tax burden distributed across income groups, particularly as a share of household income?

Lastly, determine what administrative considerations need to be taken into account. Will this revenue option be costly to administer? What rules will need to be put in place around collection and disbursement? Is it even possible to collect revenue this way?

Our work with the Funding Review Advisory Committee was mainly focused on the question of adequacy: we were trying to figure out how much each revenue option could raise. But the range of considerations I lay out here is key for a policymaker trying to build a sustainable revenue model for public needs for their city.

Revenue sustainability is about building a system of revenue that adequately funds public needs, is stable from year-to-year, and grows at the same rate the public needs grow. The first year of revenue from a new revenue stream can give you some insight into its adequacy, but sustainability and growth can only be known from years. Ultimately, building a system around the financing mechanism that ensures sustainability while still leaving the flexibility necessary for governing is the puzzle policymakers need to put together as they try to design a sustainable funding system.

The challenge of categorizing people

Earlier this week, I attended the Minnesota Evaluation Association’s annual conference. In the morning, I had a conversation with a few other attendees about some of the challenges of performing quantitative analysis in a way that accurately reflects the groups we are trying to better understand.

At the heart of this conversation was how we as statisticians assign categories to certain questions and what the implications of those choices are. Grouping people together helps us identify patterns, but those same groupings can obscure important differences between the people we are studying.  

Categorical variables

At the heart of many statistical analyses is the question “how is X related to Y?” How do poverty rates differ across racial groups, for example? What happens, though, when we dig into the categories we use to answer that question?

Categorical variables allow us to organize observations into groups. Instead of looking at each person’s exact age, we might compare children, working-age adults, and older adults. We may choose to do this when looking at age depending on our specific research question, but for non-numeric variables we are often forced into studying them categorically.

One of the key problems with discrete categorization is that certain people might be grouped together on one measure like race or age, but that might not be a good representation of that individual. As an example, someone who is 64 might have more in common with someone who is 65 than with someone who is 25, even though our categories might place the first two people in separate groups. The boundary is useful for organizing an analysis, but it doesn’t necessarily represent a meaningful change in someone’s circumstances.

Tradeoffs with sample size

One way to address this problem is to use more specific categories. If a broad group contains people with very different experiences, we could divide it into smaller groups and analyze those separately. However, we face a tradeoff as we go down this path. The more specific our model, the less generalizable it becomes.*

Imagine we are using a survey to study employment barriers in a particular city. We might have enough responses to estimate an overall employment rate with reasonable confidence, but only a handful of responses from a particular immigrant community. The experiences of those respondents are important, but a few people’s answers may not give us a reliable estimate for the entire community.

In these situations, we sometimes have to combine categories or leave a separate estimate out of a report. This can be especially frustrating when the smaller population is the one we are most interested in understanding. A dataset can be large enough to answer questions about the broader population while still being insufficient for the central question of our project.

The important distinction is that we are leaving out an estimate because we cannot support it with enough confidence, not because the population is unimportant. We should make that limitation explicit. Otherwise, readers might interpret a group’s absence from a report as evidence that it faces no particular challenges.

Depending on the project, we might address this by combining multiple years of data, collecting additional responses from that community, or using interviews to better understand people’s experiences. Each approach has tradeoffs. Combining years makes it harder to describe current conditions, while interviews provide depth without necessarily telling us how common an experience is.

Tradeoffs with race data

At Scioto Analysis, we rely on the Census Bureau’s race categories for our analyses. Because we rely so heavily on the American Community Survey and other Census Bureau products, this is the only option we have. Even in situations where we collect other data, we often need to compare it to estimates from the American Community Survey, so we still need to use the same categories. 

The Census Bureau explains that its race data are based on self-identification and that its categories reflect social definitions of race rather than biological distinctions. These categories provide a common framework for analysis, but they cannot fully describe someone’s identity or lived experience.

Two people who identify with the same broad racial category might have different cultural backgrounds or speak different languages. Putting those people into the same bucket can hide those differences, which might be relevant to whatever policy decision we are studying.

At the same time, broad categories can still be relevant to understanding how people are treated. People with different backgrounds might encounter similar discrimination because others perceive them as belonging to the same racial group. For a policy question about unequal treatment, that shared experience may be important even when the category tells us relatively little about their individual identities. However, someone’s self-reported race does not directly tell us how other people perceive them.

The question, then, is whether the categories available to us capture the differences that matter for the policy we are studying. If we are evaluating access to a public program, language or immigration history might help explain barriers that a broad racial comparison alone would miss.

Quantitative analysis requires us to simplify complicated realities. Our models should increase our understanding of a topic, but we know that it will never be perfect. When dealing with categorical variables, especially with race data, we should be aware of the underlying assumptions that we are making by assigning categories, and we should be careful that those assumptions balance the tradeoff between understanding the data we have and preserving generalizability for the wider population.

* To see this tradeoff clearly, consider the trivial example where every individual is assigned their own category. Our analysis will give us plenty of information about each individual in our sample, but there is no way to make claims about people outside our sample, since their categories are by definition not included in the study sample.

What is “housing mobility” and why is it important?

Last week, I attended Housing Mobility 2026, a two-day conference hosted by Mobility Works in Cleveland. The first day was a training institute for housing mobility staff, especially workers from various local housing authorities around the country. The second day was a gathering of housing mobility practitioners from around Ohio and a presentation about what their work looks like.

I have two main takeaways from the conference. First, many of the people who work at local housing authorities care deeply about the families they serve, but many of them are overloaded. Housing authorities don't always have the time or resources to help each family as much as they would like. Second, many organizations outside of housing authorities are doing important work in the housing space. This work ranges from buying lower-cost market-rate housing to keep it affordable, to helping families find new homes in new neighborhoods and supporting them for several years after they move.

What is housing mobility?

Housing mobility programs help families with Housing Choice Vouchers, often called Section 8, move to neighborhoods with lower poverty rates and better schools. This assistance can include help with the housing search, moving costs, landlord outreach, and support after a move. Housing mobility programs often aim to improve outcomes for children, especially safety and education.

A voucher alone often isn’t enough for a family to move to a new neighborhood. Families face several other barriers to moving. Families may not know much about neighborhoods outside of their own, and searching for housing takes time that working parents might not have. Security deposits, application fees, and moving costs add up quickly. Moving to a new neighborhood might also mean leaving behind family, friends, and other support networks, especially if reliable transportation is a concern for families. Additionally, in most of Ohio, landlords may refuse to accept tenants with vouchers, though some Ohio cities have banned such “source of income” housing discrimination. Housing mobility programs try to address these barriers by helping families search for housing, covering upfront costs, and building relationships with landlords.

One study of housing mobility programs looked at the Moving to Opportunity program, an experiment the Department of Housing and Urban Development began in 1994. The program offered families living in high-poverty public housing vouchers and counseling to help them move to lower-poverty neighborhoods. Children whose families moved before they turned 13 were more likely to attend college, and they earned 31%, or $3,500, more per year in their mid-twenties than children whose families did not receive a voucher. As adults, they were also more likely to live in better neighborhoods and less likely to become single parents. Children who were older than 13 when their families moved did not see the same gains, which suggests that how long a child spends in a better-resourced neighborhood matters.

Last year, Scioto Analysis conducted a cost-benefit analysis of a Moving to Opportunity program in Ohio. We modeled an expansion of Families Flourish, a program in Columbus that gives families rental support, relocation help, and life coaching. We estimated a net present value of $149 million, or about $3 in benefits for every $1 in costs. Most of these benefits come from higher lifetime earnings for children, reduced crime, and fewer emergency room visits. Even with a higher 7% discount rate, the program produces $1.70 in benefits for every $1 in costs. 

What does housing mobility look like in Ohio?

About one in three Ohio households rent their home, slightly less than at the national level, and Ohio’s median rent is about $1,090 per month. According to the National Low Income Housing Coalition, Ohio has about 422,000 extremely low-income renter households, but there are only 37 affordable and available rental homes for every 100 extremely low-income renter households. 73% of these households spend more than 50% of their income each month on housing.

Organizations across Ohio are working on these problems from different angles. The Cuyahoga Metropolitan Housing Authority is one of the housing authorities in the United States Department of Housing and Urban Development’s Community Choice Demonstration, a national study testing whether mobility services help voucher families move to higher-opportunity areas. Early results from the Community Choice Demonstration across all sites found that 24% of families offered mobility services moved to an opportunity area within a year, compared to 4% of families who did not receive these services.

In central Ohio, Families Flourish, the program our cost-benefit analysis was based on, gives families three years of rental support along with life coaching to help them move to higher-opportunity neighborhoods. The program has served 130 families since it began, and reports that participants' incomes rose by an average of 58%, or $17,000.

Other organizations focus on housing supply. Columbus Housing Enterprise buys naturally occurring affordable housing, or older rental housing that is affordable without a subsidy, to keep it from becoming unaffordable. In Northeast Ohio, FutureHeights in Cleveland Heights and LakewoodAlive in Lakewood work to keep older homes in good condition. For example, FutureHeights focuses on finding older homes that would otherwise be demolished and improving them to act as affordable housing. LakewoodAlive helps many older homeowners who might otherwise not be able to afford improvements and repairs to their homes.

Housing mobility and housing supply programs approach the same problem from different directions. Mobility programs help individual families move to higher-opportunity neighborhoods. These programs are especially useful for voucher holders and for families who lack the time and resources to search for new housing independently. Compared to mobility programs, supply programs work at a larger scale by building, preserving, or repairing affordable homes, often in higher-opportunity neighborhoods. However, supply programs do not change which neighborhoods a family can access, and they don’t necessarily reserve affordable housing for low-income households. Often, housing supply programs are most useful for low- to moderate-income families who can afford rent prices close to the market rate.

Housing mobility could be an overlooked tool for economic mobility in Ohio. Children who move to better-resourced neighborhoods at a young age are more likely to go to college and earn more as adults. Investment in housing mobility could be an area ripe for policymakers looking to improve long-term outcomes for Ohio families.

Scioto Analysis releases report on human development in Ohio

This morning, Scioto Analysis published a report on human development in Ohio. We estimate Ohio’s human development index to be 88.3, which is 1.5 points lower than the national index of 89.8. Human development measures a community’s ability to provide key fundamental needs to its residents across three areas:

• Health. Ohio residents have an average life expectancy of 77.5 years.

• Standard of Living. Ohio residents have a mean individual income of $49,000.

• Education. An Ohio six-year old can expect to complete 14.6 years of schooling over her lifetime.

Human development is highest in the suburbs of Ohio’s largest cities. The area with the highest human development index is a group of suburbs near Cincinnati made up of Loveland, Montgomery, and Forestville, with a human development index of 96.5. Suburbs east of Cleveland and north of Columbus are also among the highest areas for human development in Ohio.

Human development is lowest in core urban neighborhoods and some rural areas. Urban Dayton has the lowest human development index in the state at 82.4. Central Toledo, East Cleveland, West Columbus, and Scioto, Adams, and Pike Counties are also among the lowest areas for human development in Ohio. 

Human development also differs across demographic groups. White Ohio residents are expected to live four years longer and earn about $18,000 more per year than Black Ohio residents. This is not mirrored in education, though. The gap between White and Black Ohio residents is less than one year, both in adult average years of schooling and in children’s expected years of schooling.

While Ohio has very high human development overall, human development estimates differ across regions and demographic groups. The full report can be accessed here.

How long does a policy analysis take?

One of the most important questions we have to answer when we are proposing a project to a client is “how long will it take?” This one question has major implications for project costs, it can inform other parts of a client's policy strategy, and it is the most important part of determining what a project’s scope will be.

Because this question is so important, I wanted to spend some time discussing it from the perspective of an analyst. In this blog post, I talk about why timelines are important, how we build our timelines at Scioto Analysis, and what happens when unforeseen circumstances cause timelines to change.

Why is timeliness so important in policy analysis?

Unlike academic research where the primary goal is to increase our understanding of a topic, the goal of policy analysis is to inform policy decisions. In some cases where a policy analysis is purely exploratory, this doesn’t make much of a difference. However, when there is an active policy proposal being considered, or if there is a date set for a vote on a policy, analysts can sometimes be under immense time pressure. The deadline is often determined by the policy process, regardless of how much time we might want to spend on the research.

This creates a tradeoff that all policy analysis must reckon with: longer projects may lead to more confidence in results, but shorter projects leave more time for results to be communicated and digested by decision makers. There is almost always another question we could investigate or another assumption we could test. The challenge is deciding when that additional research would meaningfully improve a decision and when it would simply delay getting useful information to the people who need it.

How Scioto Analysis handles timelines

While every project is unique, there are general principles that we have developed over our years of experience that help us when determining how long we should spend on a project. A useful timeline needs to account for the entire process, from gathering data to reviewing the results and explaining what they mean. Finishing the calculations is only part of finishing the analysis.

As an example, we tend to begin almost all of our cost-benefit analyses with a 12-week timeline in mind. Based on our experience, this gives us enough time to research a topic in sufficient detail, while still prioritizing getting results out quickly enough to allow time for the information to be communicated and absorbed. That starting point gives us something concrete to discuss with a client as we determine what their particular project requires.

When we are developing project timelines with clients, we try to strike a balance between time and depth. We know approximately how much time is needed to perform each part of a policy analysis sufficiently, while our clients know when they need a report completed and how detailed they want it to be. Identifying the most important questions early helps us build a scope that fits the time available. We try to balance these needs so that we are providing our clients with as much information as they need, within a timeline and budget that works for them. This not only creates the most useful reports for our clients, but also ensures that the policymakers who may make decisions based on this research have enough time to fully understand the analysis we provide.

What happens if there is an unforeseen complication?

While careful planning and preparation can reduce the risk of delays in a project, there always arise some moments where unforeseen circumstances crop up. Maybe the dataset you planned on using wasn’t available for all the years you needed it, maybe the whole project team caught the flu and can’t work for a week. When this happens, there are often two main options:

The first option is to extend the project timeline. This can work for those types of projects that are more exploratory or don’t have hard external deadlines that must be met. This option should only be considered if there is enough time for the audience of the report to receive and digest the information even if they receive it later. A policy analysis doesn’t do any good if policymakers only learn about it after they’ve made their policy decision!

The second option is to reduce the project scope. If there are extraneous parts of a project that the analysts and clients agree are not likely going to lead to a significantly different outcome, or if there are sections that are not directly related to the main policy question, then it could be more valuable to get the audience of a report less information in a more timely manner. The key is to preserve the work needed to answer the central question credibly. A narrower analysis can still be useful, provided we clearly explain what it covers and what remains unanswered.

Because Scioto Analysis is a small team, this risk is front of mind whenever we create a project timeline. As a result, we always like to include an extra flex week into our timelines in order to account for any delays that might crop up. For most projects, this ends up being an extra week of editing and polishing that makes the final reports even stronger, but it has come in handy for addressing project delays as well.

One of the advantages of being a small team is that we can be more flexible and adjusting to these challenges is easier than it might otherwise be. We have the ability to change around our internal workflows as needed to respond to these types of challenges. Because of this, we are extremely proud of the fact that in our nearly eight years of operation, we have not missed a major project deadline. Meeting those deadlines matters because our work is most valuable when it reaches decision makers while they still have time to act on it.