One of the most important limitations of any single research study is that it only truly represents the data that the researcher used. When it comes to extrapolating any results to new data or in a new context, some studies are better than others.
Studies that include techniques like randomized control trials or causal inference methods are better than straightforward observational studies in this regard, but no one study is perfect.
As evidence of this fact, different researchers often ask the same question but end up finding different answers. This doesn’t mean that everyone is wrong, just that we live in an uncertain world and small changes in the inputs to a research project can have big effects on the outcomes.
But, as anyone who understands statistics knows, taking the average of repeated samples is one of the most effective ways to find the true average value of something. If we consider each individual study of the same question as a sample, then it follows that by averaging the results of all the studies we can more accurately approximate the truth.
We call this process “meta-analysis.” Meta-analysis is the systematic approach of analyzing the variable results of many studies of the same question. In many ways, meta analysis is very similar to policy analysis. The goal is to synthesize as much information as you can on one topic to find a single answer.
However, because it is a scientific tool, in order for something to truly be considered a meta-analysis it needs to meet certain standards above just comparing the results of similar studies.
First, meta-analysis requires a complete literature review of the topic of interest. Often researchers performing a meta-analysis will define a search criteria in advance. For example, they might limit themselves to every paper written about the value of recreational fishing in the last 30 years.
Exactly how to define a good literature search is a topic of open discussion–there is no “industry standard” for what constitutes a good literature review. Some researchers advocate for including every possible study to avoid some selection bias, while others might selectively exclude studies whose methods might have been questionable. Ultimately, the literature review should focus on gathering as much information as possible on the research question at hand.
Once you have a body of research to analyze, the next step is to record some of the key characteristics of each study. Most modern meta-analyses use meta regression techniques to control for key differences between studies. Some examples of variables that get recorded are the year the data was collected, the type of statistical model used, or the nation of origin of the study.
Often it is best practice for multiple people to perform the last step simultaneously. This way, they can make sure that their results are free from an individual's bias. If two researchers read the same paper, and come to different conclusions about what characteristics to record, then they know to go back and take a closer look.
Another important consideration for researchers is publication bias. Publication bias stems from the idea that academics and journal editors have very little incentive to publish papers that don’t find any new interesting results. This is a problem because it is still important for the broader understanding of a subject to test a hypothesis and find out that we were wrong. It just doesn’t make for good reading.
In the context of meta analysis, publication bias might result in our estimates being biased to be larger and less variable. There are statistical and graphical checks researchers can perform to check for publication bias, but there is no single method to be certain.
When done correctly, a meta-analysis can synthesize an entire field of research into a much more digestible and applicable format. There is a lot of work that goes into it and there are many pitfalls along the way, but the reward is certainly worth it: we get one step closer to the truth.