When studies collide confusion can reign

It is all too common to hear news outlets touting the latest study such as one recent report: “Drinking a glass of red wine is just as good as spending an hour at the gym.” Or perhaps you’ve received Facebook posts about studies that seem to contradict one another. For example, different studies have shown that drinking more coffee has positive health benefits, including reversing liver damage, helping prevent colon cancer and decreasing the risk of endometrial cancer, but other studies have shown that drinking coffee may have negative health effects such as an increased risk of miscarriages, sleep disruptions, and increased blood sugar levels.

What, then, should we do — drink more coffee hoping for the best? Stop drinking coffee just to be safe? Ignore scientific studies because they are often contradictory? Before you do decide to act based on what you’ve heard from a recent study, it is important to understand a study’s methods: how it was designed and conducted. Variations in study designs and in the quality of studies can impact the results. When studies “collide” and we’re faced with contradictory scientific evidence, here are a few things to consider.

First, find out how the researchers defined their key concepts. If a study claims that drinking coffee can prevent colon cancer, one question is, “How much coffee must be consumed to see this positive effect?” Researchers should clearly define their key concept — “coffee consumption” — by indicating how many cups of coffee were consumed on a daily or weekly basis and even what kinds of coffee (e.g., caffeinated/decaffeinated) were consumed. Different findings may arise if, say, a study included people who drank one cup of coffee per day and another study included people who drank one cup per week. It may not be appropriate to compare two studies that defined their key concepts differently.

Second, find out details regarding the study sample. Typically, researchers cannot study every relevant, eligible person. So they must select a subset of people — their sample. To know what a study’s findings mean, we must know who (or what) the researchers used in their sample. For example, that “red wine equals an hour at the gym” study mentioned above was actually based on an experiment with rats!

Unfortunately, those details may not be reported in a news headline/story and we are left to presume that such findings are relevant to human beings. Animal studies are vitally important for research, but findings based on a rodents should not be assumed to be valid for humans. Furthermore, even if a study includes humans, it is important to know who those humans are. Researchers should specify what their sample “looks like.” If, for example, a study only includes freshmen from Mercer University, the results may be valid and valuable. But it limits the researchers’ ability to generalize to different people, settings, or times. A result that occurs in one group in one particular study may not be applicable to others.

Also, it is important to understand that scientific knowledge often advances through small, incremental steps. So don’t be surprised if new findings contradict what we once thought was “true.” Typically, scientists are not able to answer a research question that is “100 percent true” for all people, in all situations, in all times. It is the job of researchers to use scientific methods that will advance knowledge in what is currently unknown. As consumers of research, we must understand how studies are conducted so that we can make more informed decisions.

This story was originally published April 22, 2017 at 1:23 PM with the headline "When studies collide confusion can reign."