What’s the most amazing thing about evaluation?
Is it that it lets us know if something works or not? That’s pretty amazing, no doubt. Understanding if a program has the effects people claim it does is a huge part of understanding the value of the program.
Is it that evaluation helps us understand how much a program works? This is almost more amazing than knowing if the program works. For instance, maybe a graduation-focused tutoring program works because it increases the number of people who graduate each year at a high school by one. Another program is found to work by increasing the number of people who graduate by ten. Knowing how much a program works can be even more valuable than just knowing if it works in the first place.
So what’s even better than knowing both if a program works AND how much a program works? Knowing how much a program works—in respect to its resources. Maybe the second tutoring program above graduates 10 new people at a cost of $10,000. But the first tutoring program graduates 1 at $100. This means program 1 is ten times as cost effective as program 2. While the first program may be more effective on an absolute basis, the second is more effective per dollar…which can be very important for a cash-strapped school.
So if a school was running tutoring program 2 for ten people and it was costing them $10,000 but then switched those ten people into tutoring program 1, the school would save $9,000…and still graduate the same number of students.
Theoretically, the school could have evaluated both programs for up to $9,000, implemented the recommendations that came from that evaluation, and ended up on top, the evaluation effectively paying for itself.
In the evaluation world, this concept is called “cost-free evaluation.” If evaluation focuses on this question of cost-effectiveness, it can be a strong tool for saving money and effecting better outcomes at the same time.
In Results for America’s book Moneyball for Government, members of the Bush and Obama administrations put forth a number of recommendations for improving the efficiency and effectiveness of government by being more results-oriented and understanding the interaction between costs of programs and results brought about by those programs.
One of their recommendations is for every federal agency to have 1% of their discretionary budget set aside for evaluation. This means that they can focus their evaluation efforts on programs that may be expensive or ineffective and potentially make that money back for the American people. Then money can be spent on better programs or even rebated back with less spending overall.
There are of course wrinkles to evaluation. Just because a program is cost-effective when small does not mean it will be cost-effective as it grows. Often programs have trouble when they try to scale up. But this is why evaluation needs to occur during the scale-up process, too. If anything, evaluation could be more important here because there are more resources to be lost here. But the promise rings true: evaluation, when done right, can be free. And that is truly amazing.