In 2012, Dr. John Ioannidis, the chairman of disease prevention at Stanford, published a study titled “Is Everything We Eat Associated With Cancer?” He and a co-author randomly selected 50 recipes from a cookbook and discovered that 80 percent of the ingredients — mushrooms, peppers, olives, lobster, mustard, lemons — had been linked to either an increased or a decreased risk of cancer in numerous studies. In many cases a single ingredient was found to be the subject of questionable cancer claims in more than 10 studies, a vast majority of which “were based on weak statistical evidence,” the paper concluded.
Nutrition epidemiology is notorious for this. Scientists routinely scour data sets on large populations looking for links between specific foods or diets and health outcomes like chronic disease and life span. These studies can generate important findings and hypotheses. But they also have serious limitations. They cannot prove cause and effect, for example, and collecting dietary data from people is like trying to catch a moving target: Many people cannot recall precisely what they ate last month, last week or even in the past 48 hours. Plenty of other factors that influence health can also blur the impact of diet, such as exercise, socioeconomic status, sleep, genetics and environment. All of this makes the most popular food and health studies problematic and frequently contradictory.
In one recent example, an observational study of thousands of people published in The Lancet last year made headlines with its findings that high-carb diets were linked to increased mortality rates and that eating saturated fat and meat was protective. Then in August, a separate team of researchers published an observational study of thousands of people in a related journal, The Lancet Public Health, with contrasting findings: Low-carb diets that were high in meat increased mortality rates.
“You can analyze observational studies in very different ways and, depending on what your belief is — and there are very strong nutrition beliefs out there — you can get some very dramatic patterns,” Dr. Ioannidis said.
He and other experts have called for reform in nutrition science. They say that researchers should publicly register their study protocols beforehand to eliminate data dredging, share their raw data to increase transparency, focus on large randomized controlled trials to produce better results, and refrain from slicing and dicing large observational data sets into multiple papers that magnify weak findings.
Experts say that the problem extends to science journalists as well: Many reporters are encouraged to produce articles that get lots of clicks. That is another reason researchers and universities feel pressure to put out studies and news releases with exaggerated findings.
“I would say that we’re all drinking from the same well, and we’re all contributing to poisoning the water,” said Ted Kyle, an obesity expert who runs a health site called ConscienHealth. “At every step along the way there are folks who are culpable. I would suspect that we’re all complicit.”
