Scaling Programs: Trying to Find the Mountain in the Molehill

I am in the process of finishing up a cost-benefit analysis on the value of water quality programs in Ohio. The statewide policy alternatives I am considering are based on the H2Ohio program, which is currently focused in a handful of northwest Ohio counties. Today I wanted to briefly talk about the challenges that come with trying to scale up pilot programs and how policymakers should think about expanding successful programs.

First, let’s recognize why we have pilot programs in the first place. The goal of a pilot program is to test the effectiveness of a proposal before investing a large amount of resources should it fail for some reason. If the trial run is successful, then the next step is to scale up the program to serve a larger population. 

However, pilot programs are often specifically designed to succeed in a way that is harder to replicate on a larger scale. As the economist John List points out in an episode of the Freakonomics Radio podcast, “after researchers in the social sciences do an efficacy test, they forget to tell everyone else that it was an efficacy test.” 

This is not to say that it is wrong for pilot programs to try to achieve good results. It is hard to imagine a researcher getting funding for some intervention where implementation is shoddy and where they plan to target the people least likely to benefit. However, as programs grow it becomes more difficult to ensure implementation fidelity and proper targeting. 

Limiting participation in programs to only those who will receive the greatest benefits also might lead to the most efficient outcome, but it might not be the most equitable outcome. There is no right or wrong way to trade off equity and efficiency. Understanding that the two sometimes compete when resources are limited is the essence of good analysis and policymaking.

In the context of the H2Ohio program, one specific alternative I am considering is making the whole state eligible for the current program. Currently, the program is only open to counties in Northeast Ohio which have large concentrations of farmland and are part of the Lake Erie Basin. Because the rest of the state does not have the same contributions to Lake Erie algal blooms, this program likely won’t have exactly the same results in other counties beyond the current “pilot” counties. 

These insights about problems common to scale-up can inform the sensitivity analysis phase of our cost-benefit analyses. As analysts, we can project what might happen under different conditions, say if scale-up works swimmingly or if scale-up falls far short of the impacts of the pilot program. Exploring different scenarios, especially the ones where things are not quite as effective as they were in the pilot program, often lead to better insights and better decisions.

Four issues voters care about this election season

With elections coming tomorrow, reporters across the state are asking voters what issues matter to them. I saw a recent survey done by 10TV  that caught my eye. 

This survey was a quick, small survey that likely had a convenience sample, but it still gives us some perspective on what voters are talking about. Asking respondents to rank twenty issues, four rose to the top. Looking at it and comparing it to the conversations I have with friends and the news coverage I see, I can see why voters have these on their mind as they prepare to go to the polls this week.

Cost of Housing

As far as places to live in the United States, Ohio is relatively affordable. According to Zillow data, the typical home price in the state is about $210,000—about $100,000 less than the country as a whole. 

That being said, prices are still going up. Median sales prices are up 7.1% over the last year, meaning you’d have to pay about $15,000 more today for a house than you would a year ago. And these are changes that go beyond Ohio’s big cities—small cities like Lancaster, Harrison, and Zanesville are among the top 10 cities in Ohio with fastest growing sales prices, all experiencing price increases of over 30% in the past year.

Home buyers are getting a reprieve with prices starting to slow, but this is the same time the federal reserve is increasing interest rates, making it harder to buy a home. State- and local-level policymakers elected this week will have to decide whether they will promote affordable housing and construction of new housing or allow housing prices to continue to rise.

Abortion

In the wake of the U.S. Supreme Court’s decision to overturn Roe v. Wade, states across the country are restricting abortion in new ways. Governor DeWine signed a ban on abortions after six weeks of pregnancy, which is currently on hold by court order.

The evidence is clear that bans on abortion are much less effective at keeping people safe and reducing the need for abortion than other strategies like increasing access to contraceptives. Policymakers elected this week will decide whether to promote these tools or continue to use abortion bans as a political cudgel.

Cost of food

About one in ten Ohio households are food insecure and about one in twenty are very food insecure, both higher than the national average. Food insecurity is most prevalent among low-income households, single women with children, and black households.

Food insecurity often revolves around lack of access to income, but there are educational interventions that can help reduce food insecurity as well. Policymakers elected this week will have the opportunity to tackle food insecurity through interventions like low-income tax credits and extending the state SNAP-Ed program.

Cost of health care

According to the Health Policy Institute of Ohio, Ohio ranks 47th among the states and D.C. when it comes to health value. Adverse childhood experiences, disparities in health outcomes, and low public health infrastructure investment has led to these problems.

Policymakers elected this week will have opportunities to support programs that prevent health problems before they become expensive private and public liabilities.

These are only a few of the issues that Ohioans are focused on this election season. Let’s hope voters choose the right leaders who have the courage to take on issues like these rather than just talking about them before election day.

This commentary first appeared in the Ohio Capital Journal.

What should a policy analyst know about the normal distribution?

Last year, Scioto Analysis conducted a policy analysis to evaluate alternatives to reduce carbon emissions in the state of Ohio. In order to test our models, we conducted a Monte Carlo simulation, the “gold standard” of sensitivity analysis in cost-benefit analysis and a tool we often employ to see what range of possible outcomes.

Below are the Monte Carlo simulation results for a strong cap-and-trade program. For this Monte Carlo simulation, we ran hundreds of thousands of simulations of alternate policy scenarios, randomly generating different social cost of carbon estimates, discount rates, and price elasticities of demand for electricity. You can see the results of the Monte Carlo simulation below.

If you’ve taken a statistics class, you’re probably familiar with the shape of this distribution. It is one of the most important shapes in statistical analysis and one we end up using a lot when we’re modeling policy outcomes: the normal distribution.

The normal distribution, often called a bell curve because of its shape, is one of the most universally recognized statistical concepts. It is intuitive, broadly applicable, and useful in simplifying complex concepts. Here we will briefly discuss a few important characteristics of the normal distribution and why they are important. 

Parameterization

The parameters of a statistical distribution are the things you need to know to fully understand it. For example, if we consider some binary outcome like flipping a coin (called a Bernoulli distribution), the only parameter we need to know to fully understand the range of outcomes is the probability of an event happening.

The normal distribution is special because it has two parameters, its mean and its variance. We can always calculate the mean and variance of observed data, but the fact that these tell us everything we need to know about the distribution of unobserved data from the same distribution is extremely powerful. 

Symmetry and Outliers

Two other properties we will discuss together are the facts that the normal distribution is symmetric. This means that we should expect to observe values above and below the mean with the same likelihood, and that big outliers are extremely uncommon such that the we should only observe a value only four standard deviations above or below the mean is about 0.001% of the time. These two characteristics often shape how we think about applying the normal distribution to real data.

Consider the distribution of incomes in the US. We know that a few small outliers skew the distribution heavily to the right which makes fitting these data to the normal distribution difficult. If we just calculate the observed mean and variance of all individuals in the US, we would expect there to be extreme outliers in the negative direction as well.

The Central Limit Theorem

Arguably the most important concept in statistics, the central limit theorem is certainly the most useful application of the normal distribution. There is a lot of rigorous math that we will skip over here, but in short the central limit theorem tells us that if we repeatedly take random samples from a population, those sample means will be approximately normal. 

Going back to the income example, if instead of measuring using the mean and variance of all incomes to approximate a normal distribution we took 500 random samples (with replacement) and calculated the means of all of those, we would find that those sample means did in fact follow a normal distribution quite well. 

The normal distribution gives us a way to mathematically describe what we expect to happen with lots of unobserved data quite well. Understanding it at a surface level is valuable for policy analysts and policy makers since it so often works its way into our assumptions, whether we realize it or not.

Conversion therapy bans could prevent hundreds of youth suicides over the next decade

Last Monday, the Akron City Council voted to ban conversion therapy in the city, making it the eleventh city in Ohio to do so. 

The pseudoscientific practice referred to as “conversion therapy” encompasses counseling aimed at children focused on changing sexual orientation. The practice has been condemned by the American Medical Association, the American Counseling Association, the American Academy of Pediatrics, and the American Association for Marriage and Family Therapy.

The Akron ordinance cited a 2019 study by the UCLA Williams Institute that found LGBT+ youth exposed to conversion therapy were twice as likely to consider and attempt suicide than those who hadn’t. Conversion therapy made worldwide news when transgender Ohio teenager Leelah Alcorn posted her suicide note on the social media site Tumblr in 2014, explaining how conversion therapy led to her death.

I still find myself surprised this is a practice that occurs in Ohio today. I’ll admit, I sometimes even think “is this a real problem or are these sorts of ordinances empty gestures?” I have been saddened to find that this problem is indeed very real.

In a survey conducted as a part of Kent State University’s LGBTQ+ Greater Akron Community Needs Assessment, 30 of 701 respondents reported they had received conversion therapy at some point. According to the UCLA Williams Institute, there are 72,000 LGBTQ+ Ohioans age 13-17. 

If Ohio rates of conversion therapy reflect the responses in this survey, that means nearly 3,000 Ohio teenager have been exposed to conversion therapy. The Trevor Project’s 2022 National Survey on LGBTQ Youth Mental Health found 18% of LGBTQ teens made a suicide attempt in the past year. 

If teens exposed to conversion therapy have roughly double the suicide attempt rate of the general population, this means over 1,000 LGBTQ teens in Ohio who have been exposed to conversion therapy attempt suicide every year. If the death rate of suicide attempts reflects the national average, this means 43 LGBTQ teens in Ohio who are exposed to conversion therapy die of suicide every year.

If conversion therapy doubles the suicide attempt rate for youth, this means banning the practice statewide could save over 200 youth lives from suicide in Ohio over the next decade.

Bans on conversion therapy have been enacted so far in the cities of Akron, Athens, Kent, Cincinnati, Cleveland, Cleveland Heights, Columbus, Dayton, Lakewood, Reynoldsburg, and Toledo. The largest cities without bans right now are Parma, Canton, Lorain, Hamilton, and Youngstown, each with total populations over 60,000 people.

City-level bans only go so far, though. Bans can be ducked by people providing conversion therapy across jurisdictional lines only a short drive away, making patchwork municipal bans less effective than a statewide ban. 21 states and the District of Columbia have now enacted statewide bans on conversion therapy, mainly concentrated in the northeast and west. This includes moderate political states like New Hampshire, Pennsylvania, Utah, and Virginia. 

The only state with a conversion therapy ban in the Midwest currently is Illinois, though it is the region of the country most blanketed with city-level bans. If state lawmakers want to reduce teen suicides in Ohio, they have a strong option in front of them.

This commentary first appeared in the Ohio Capital Journal.

Discounting in Cost-Benefit Analysis

This series of blog posts are about my first cost-benefit analysis with Scioto Analysis. As I am going through this process, I am writing about the challenges I come across and how I have been thinking about them. This week, I wanted to write about discounting, why it is important, and when it should be used.

Discounting is the process we use to estimate the difference in benefits policies create in the present from benefits they create in the future. Discounting is a cornerstone practice of good cost-benefit analysis.

Which would you rather have, $100 today or $100 in 10 years? It is pretty easy to understand why $100 in 10 years is the worse of these options. If you get hit by a bus tomorrow, that $100 later will be pretty useless. You could also invest your $100 today and end up with $200 in ten years. Having money now is better than having money later. 

When conducting cost-benefit analysis, discounting is extremely important because many impacts calculated incur short-term costs and long-term benefits. Appropriate discounting of future benefits is an essential step in determining the true value of a proposal from a social perspective.

In the context of my current cost-benefit analysis, I have spent a lot of time thinking about whether or not I should include future values. For context, the program I am analyzing is essentially government subsidies to encourage farmers to use less phosphorus fertilizer which in turn leads to less phosphorus runoff into Ohio’s waters.

The justification for including future impacts is that improved water quality is valuable year after year. However, there are two major assumptions that we must consider. First, the subsidy is a one-time investment meaning that all of our costs are accounted for in the present. Second, because the reduction in phosphorus is the result of using less fertilizer, we assume that going forward there are no new reductions in phosphorus. 

Whether or not discounting future benefits is appropriate depends on how we choose to monetize benefits. In this case, we model the benefits of this program as people’s willingness to pay for cleaner water. This means that the marginal decrease in phosphorus is valuable. We could discount the marginal increase in recreational use, but we should expect that benefit to be captured in the current willingness to pay for a marginal increase in water quality today. 

Choosing not to include future benefits in this analysis makes the most sense in this context, but as a policy analyst it is important to fully understand the implications of that decision. In this case, it leads to a more conservative estimate of the benefits of the program, which can be a useful direction to err.

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Introducing Scioto’s Interactive Budget App

At Scioto Analysis, we believe in making information as accessible as possible in order to inform policymakers and the public. In that spirit, we are proud to unveil our first interactive budget app! This app uses data from the Ohio Legislative Service Commission about the state’s forecasted budget for fiscal year 2023. 

In its current state, you can use this app to see how changing revenues and expenditures affects the state’s budget surplus and see the tradeoffs for yourself. With this app, you can effectively create your own state budget! Want to increase spending on public programs? Here you can see some of the different possible ways of financing it.

You can either use the tool below or you can open it up in its own window by clicking here.

In the future, we plan to include more options so you can see the effects of policy options ranging from specific proposed tax cuts to pie-in-the-sky proposals like universal basic income. If you have any feedback or suggestions feel free to email michael@sciotoanalysis.com.

In the meantime, get budgeting!

Social vs. Private Costs in Cost-Benefit Analysis

I am currently working on a cost-benefit analysis on policy options to improve water quality in Ohio. At this point in the process, I’ve spent a few weeks thinking about how to approach this question and even have some preliminary models built out. There is one specific part of these models that has made me stop and think and I want to talk about it briefly. 

All of the policy alternatives I am considering for this cost-benefit analysis involve some amount of government spending to improve agricultural practices on private farms. Essentially, the government subsidizes certain farm practices that while slightly more expensive, can improve water quality and provide benefits for people who use lakes and streams. The question then for the analyst is what should count between public costs, private costs, and private time and labor.

The case for including all these impacts in this analysis is that under most circumstances, the subsidy won’t cover all of the upfront costs associated with the program, and instead farmers would be spending a smaller amount in the short term to get additional long-term benefits in the form of reduced spending on fertilizer. If farmers are trading short-term costs for long-term benefits, then the model should try to capture that. The case against including both measures relies on the fact that the program of interest is voluntary and farms that participate in it are profit maximizing firms. 

Consider for a moment that this cost-benefit analysis did not involve any government subsidy, and we were just concerned about farmers implementing these practices themselves. In this case, we would include the upfront costs in our model. With government subsidies, the intuition might be to just subtract the subsidy from the total cost in order to avoid double counting. 

However, acknowledging the fact that in order to spend money the government needs to tax its citizens, the marginal excess burden of taxation is the social cost of a subsidy (read more in the Ohio Handbook of Cost-Benefit Analysis). In short, we are measuring the lost economic activity that comes with raising taxes, not the transfer of funds from the government to a farm, which is itself a net zero transfer payment. 

If farmers are choosing to enter into this program, then there should not be any lost economic activity. The upfront costs of this program are just a transfer of funds from the farmer to the scientists they need to perform soil tests. Whatever portion of these funds comes from the subsidy is largely irrelevant, because the only economic loss is through the marginal excess burden of taxation. If this was a mandatory program, then we would consider the lost value of however else the farmers would have spent their time because it would be mandated changes that would not have happened otherwise. Voluntary compliance suggests the costs and benefits are internalized by the farmer. 

This detail is an illustration of the fact that cost-benefit analysis is not a tool for measuring private costs, but rather social costs. Without additional information about value added or lost from transferring funds between parties, there should not be an economic effect. 

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Economists weigh in on proposal to abolish subminimum wage for workers with disabilities

In a survey published by Scioto Analysis this afternoon, economists in Ohio called into question the current tiered minimum wage system that allows for employer to pay lower wages to workers with disabilities.

Ohio Representatives Brigid Kelly and Dontavius Jarrells have put forth a bill to eliminate the current lower minimum wage for persons with disabilities. This lower wage threshold was originally put in place to encourage employment of people with disabilities. Advocates for its abolition argue the lower minimum wage has not yielded the results originally intended.

A majority of respondents to the Economic Experts Panel survey conducted last week said abolishing the lower minimum wage would reduce poverty. Some who agreed did so with the qualification that the overall poverty impact would be modest. Among those skeptical of a poverty reduction, economists cited the potential unemployment impact of a higher minimum wage and the small scale of the policy.

A majority of respondents also said the policy would not likely have negative human capital ramifications for people with disabilities. Economists who believed this would not hurt human capital development said that the current policy has not been shown to improve human capital for workers. Those who were uncertain said that employment loss could hinder human capital development for workers.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.

Conducting Cost-Benefit Analysis: The role of assumptions

I am currently working on a cost-benefit analysis on water quality in Ohio. After spending two weeks preparing and thinking about this current cost-benefit analysis, I am starting assembling some preliminary models. This step is often known as projecting the outcomes and is almost always the most difficult part of a cost-benefit analysis. 

One of the main challenges when trying to project outcomes is fully understanding what sort of assumptions we as analysts have to make in order to make projections at all. Our assumptions define how we think about our analysis and they shape our results. Assumptions cannot be avoided in any sort of analysis, but as analysts it is our job to be fully aware of which ones we make. 

Generally speaking, each additional assumption makes the analysis easier to perform and to understand, but it makes it more difficult to generalize the results. A common misconception is that having stronger assumptions makes analysis less accurate or less meaningful, but sometimes making additional assumptions actually improves our models. 

This tradeoff is extremely common and is even described mathematically by the bias-variance tradeoff. In statistical terms, we evaluate estimators using a criteria called mean squared error which can be written as the sum of an estimator's variance and its squared bias. Decreasing one of those quantities often means increasing the other. In our context adding more assumptions simplifies the models and reduces the variance, but because we are assuming more things we may be adding some bias to our results. It is ok to keep some bias, especially if we understand that it exists and it lowers the variance of our estimate making it more useful.

At the beginning of my model building, I chose to make as many assumptions as I had to in order to get some preliminary results. It may not make sense to assume that certain things will remain constant well into the future, but because this is an iterative process it is important to get a working model to build off of. Building a simple model can give you a general idea of potential results and can give you an answer which then you can build from by refining your model.

Making assumptions is also useful because they help you identify what things you need to learn more about. After going through and making all of the first models, it is important to ask which assumptions can I potentially get rid of. Does removing these assumptions make the model sufficiently more generalizable?

As analysts, it is our job to make clear that predicting the future is extremely difficult. I once heard someone say that making predictions is like trying to drive a car by only looking in the rearview mirror. Sometimes trying to work with real data forces us into making strong assumptions. Still, by using the best available methods and fully understanding the ramifications of the assumptions we make, we can help policy makers decide on the best course of action with better information than they would have on their own.

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Is it time for all Ohio children to get free meals at schools?

Last month, the Ohio State Board of Education officially recommended that the state of Ohio use American Rescue Plan Act dollars to provide free breakfast and lunch to Ohio students through the end of the year.

The federal government has been providing free lunch to children from low-income families since 1946, when President Harry Truman signed the National School Lunch Act into law. The program was expanded in 1966 through the Child Nutrition Act, which added breakfast and summer meals to the program.

The federal free school lunch and breakfast program is one of the largest antipoverty programs in the United States. In a study Scioto Analysis released last year, we estimated from American Community Survey and Current Population Survey data that more than 8,800 Ohioans were pulled out of poverty in 2018 by the federal free school lunch and breakfast program. 

Nationally, the impact is even larger. The Census Bureau estimates 300,000 Americans were pulled out of poverty by school lunch and breakfast programs in 2020, and an additional 3.2 million were pulled out of poverty by a combination of SNAP food assistance and free school lunch.

Free lunch has a big impact on the children who receive them. Half of the food consumption for these children comes from free or reduced-priced meals. Recent expansions in availability of free lunch has also meant better outcomes for children. Recent studies of expansions of free school lunch programs suggest the expansions have led to increases in math scores for students, especially among elementary school students and Hispanic students. These expansions have also led to decreases in suspensions among white elementary-school age boys.

School lunch programs have been expanding for years. In 2011, schools in Illinois, Kentucky, and Michigan began piloting a program that would make all children in low-income districts eligible for free lunch. By 2019, two-thirds of all low-income schools across the country were providing free lunch meals to their children.

The pandemic brought a seismic sea change to this landscape. The United States Department of Agriculture, which administers the free school lunch and breakfast programs, suspended all eligibility requirements for free and reduced meals, making free lunch universal with a single administrative change.

Last summer, this expansion lapsed, and Ohio reverted to a limited system of provision of free and reduced priced lunches.

Five states—California, Maine, Massachusetts, Nevada, and Vermont—have passed legislation providing no-cost meals to all students. Pennsylvania recently joined this list, passing legislation making breakfast free in schools.

It seems that the biggest reason free school lunch and breakfast has not been expanded even more aggressively in the United States is because of simple budget constraints. In the particular policy the state Board of Education calls for, American Rescue Plan dollars have a number of different alternate uses. I have not done an analysis of all possible uses for these funds, but I have to imagine this would be a use that would yield relatively high economic and equity benefits.

One potential objection to universal school meal programs is that they would mainly benefit middle-and upper-income households and do little for children from low-income households since low-income households are likely already covered by the current program. The benefits of administrative simplicity, though, have already led to universal provision in low-income schools. Providing this benefit to all children and then clawing back costs through progressive income taxation is likely a more efficient program design than a system focused on splitting hairs around eligibility.

The pandemic changed a lot of assumptions about the U.S. safety net. Maybe a permanent change could come to the program: maybe all children in Ohio should get free meals at school.

This commentary first appeared in the Ohio Capital Journal.