In May of this year, I wrote a blog post talking about the unintended consequences of taking standardized test scores out of the college admission process. In that post, I highlighted a paper written by economists from the Federal Reserve Bank of Philadelphia.
Many people who advocate for colleges and universities to not look at test scores in their admission process point to the fact that test scores are not a perfect indicator of future academic success, and they could be biased in favor of students who have more resources for expensive practice materials and tutors.
However, this paper found that in practice, test scores could reduce bias in the admission process. This is because when schools did not have access to the information about test scores, they instead relied on information that introduced even more bias such as whether or not a student has family members who attended the school.
I thought about this paper this week when I was reading a new article in the Journal of Public Economics talking about the impact of using test scores to hire teachers in Colombia. This paper focuses on a new merit-based hiring system for teachers nationwide. The goal of the program was to increase the quality of teachers, which in turn should lead to better student achievement.
Despite the well-intentioned goals of this policy, the authors of the study found student performance actually declined in the wake of its implementation. According to the authors, students' test scores dropped by 8.2% of a standard deviation and both college enrollment and graduation rates decreased significantly after merit-based hiring was implemented.
This finding parallels the earlier debate about college admissions: test scores, while useful in some contexts, are not a perfect measure of quality. Just as test scores in college admissions do not fully capture a student's potential for success, teacher test scores do not completely reflect their effectiveness in the classroom. The Colombian policy overemphasized one narrow measure—cognitive ability—at the expense of other important factors, such as teaching experience.
The decline in student test scores and matriculation outcomes can be partly explained by an influx of inexperienced teachers driven by merit-based hiring. The share of teachers with little to no experience in Colombia increased from 10% to 30%. The literature shows that teacher quality tends to be lower during the first five years of teaching, and this reform exacerbated that issue by bringing in a large number of new teachers.
This case study highlights an important lesson: teaching experience matters for student outcomes. While cognitive ability is important, it is only one component of what makes a teacher effective. Experienced teachers have had time to develop classroom management skills, learn how to adapt their teaching methods to different student needs, and build relationships with their students—all of which are critical to fostering a productive learning environment.
The study offers an important policy takeaway for education systems worldwide: teacher hiring criteria need to be multifaceted. Instead of relying on a single metric like test scores, hiring systems should consider a broader set of criteria, including both ex ante (before hiring) and ex post (after hiring) measures of teacher effectiveness. Experience, interpersonal skills, and classroom performance are equally valuable indicators that should inform not only hiring but also decisions about retention and promotion.
Just as in the case of college admissions, where removing test scores has been found to inadvertently increase bias by relying on less objective measures, focusing too narrowly on test scores in teacher hiring can also lead to unintended consequences. This suggests that policymakers should aim for a balanced approach that integrates multiple dimensions of teacher quality, combining cognitive abilities with real-world teaching experience and performance data.
Ultimately, the lesson from both college admissions and teacher hiring reforms is clear: while test scores provide valuable information, they are far from perfect. Effective policy design requires a comprehensive approach that considers all elements of a person's abilities, whether they are a student or a teacher. Only by doing so can we create systems that both promote fairness and improve outcomes.