Natural Foods Merchandiser

Bridging the research credibility gap

To promote credibility and trust in the supplements aisle, retailers must discriminate between real science and pseudoscience.

"Let's take an example of the scientific gold standard. Without doubt, sufficient folic acid in your diet lowers your risk of having a baby with certain birth defects," says Elizabeth Whelan, Sc.D., president of the American Council on Science and Health. "So what elements here make up this gold standard? Years of research, hundreds of thousands of participants, hundreds of studies, all published in peer-reviewed journals.

"Then there's the lower level of gold standard: dozens of epidemiological studies in peer-reviewed journals suggest a diet rich in calcium might reduce your risk of colon cancer. It's not all wrapped up, but it looks good.

"From there, you step down to almost everything else—like whether St. John's wort helps depression. There's no vast body of evidence to show that effect whatsoever." And this last level, says Whelan, describes 95 percent of manufacturer-cited studies.

Distrust messengers
"Beware news stories linking supplements to diseases or curative effect because they're regularly dismissed the next week," Whelan says. "Daily, in the news, there's a link about some item causing or preventing disease. If a marketer picks up on these stories and sells something based on those transient claims—which vary and are isolated—then he has no credibility at all."

"Studies cited in releases or media are mere snapshots of the full study," says Anthony Almada, M.Sc., NFM's science adviser and founder of IMAGINutrition, a nutrition technology think tank based in Laguna Niguel, Calif.

Are natural-health media more credible? "Both are of little value," Almada says. "They're only roadmaps pointing to your destination—the full-length journal paper."

Almada recommends retailers read only complete studies. However, full-text studies often cost a fee. Check for links to journals offering free access. And if the journal you want is not listed?

Demand assistance
"Call the manufacturer citing a study, using their toll-free number. Ask them to e-mail or fax you the complete study," says Almada. "Ask them to highlight dose, brand, whether that's their actual product, how often taken and whether researchers were economically involved.

"And don't settle for the abstract. Statements in the abstract often don't appear in, or aren't justified by, the actual study.

"If you make claims based on abstracts, you can end up looking like a fool. Demand the full study. If you don't get it promptly, well, I'd inform the company that you're going to stop accepting their products."

Discount epidemiology
Single epidemiological studies are seldom useful, Almada says.

"To me, an epidemiological study is something to discuss with your friends—but to actually put faith in it that it works? Completely misguided," he adds. "Tests of epidemiological studies often end up proving the opposite."

"First, was it an epidemiological study done on humans? The vast majority are rodent studies," Whelan says. "Second, was it a large or small study?" For instance, the Harvard University Nurses' Health Study had more than 120,000 participants—a huge, and impressive benchmark.

If epidemiological studies found a correlation between wearing skirts and getting breast cancer, would that mean skirts cause this disease? No. People who wear skirts are generally women—and, most cases of breast cancer involve women. Don't interpret "correlation" to mean cause.

Assess sources
"Ask who these scientists are," says Whelan. "Are they involved in selling supplements? Has the study been published in a peer-reviewed journal? If it wasn't, it should certainly be viewed with strong skepticism. If it was, it could be one aberrant study—sound science is a collection of studies."

Question significance
"Put it into context," Almada says. "Say a supplement causes a blood pressure drop of four points, and the 'Results' section of the study reports that's statistically significant. Is that medically or clinically significant, if my blood pressure is 149? No, it has no significance for consumers."

Ignore percentages
"Expressions of risk as a percentage change—say, 400 percent lower risk—usually come from advertisers, and should be ignored," Almada says. "Only actual risks count, not percentage changes. If risks before and after supplement use were infinitesimally small, it's totally misleading to state that difference in terms of percentage reduction in risk."

Whelan agrees: "You get huge percentage reductions in risk by changing just one or two units."

Double-check blinding
Manufacturers wine and dine some researchers, who don't realize how they're being influenced subconsciously. Even when independent researchers assign subjects to "active" or "placebo" groups, selection bias can creep in. For reliability, studies should be double-blinded.

But even double blinding can be flawed. How can you judge? Read the study's 'Methods.' "[Also], studies often have an accompanying editorial by someone not involved in the study that critiques the soundness of the study," says Almada.

Scrutinize ingredients
Manufacturers often cite studies on one specific biological item to sell their brand—which contains a different variation. "If studies show health benefits from a particular brand, retailers should only recommend that one brand," Almada says.

Check previous studies
"When you have industry-funded researchers, they'll typically exclude mention of studies that don't conform to what they're trying to communicate," says Almada. "Take conjugated linoleic acid: More studies have shown it doesn't work, but people get all excited by one positive study. You must check all previous studies on the issue and whether they corroborate or contradict this one study."

How? The 'Introduction' and 'Discussion' sections of good journal studies provide perspective by describing related previous research. Online journals often provide links to earlier studies.

If retailers simply cannot understand scientific literature, Whelan has a suggestion: "Approach people in nutrition at local universities and say, 'We've been told about this study: Where does it fit into the general understanding of things?'"

Think long-term
"There's some conflict of interest [in dismissing pseudoscience] because retailers want to increase sales," Whelan says.

Want the complete list of tips for distinguishing sound studies from pseudoscience? See our online checklist at
But store owners should think long-term, suggests Almada: "What's the tenor of media perspective about studies cited to sell products? That most don't work, may be unsafe and often don't contain what's on the label. Consumers read that. The retailer has to make a decision: Does he truly want, over a two-to-five-year period, to emerge as an evidence-based authority? "Many retailers are evangelists: They stock products on faith, because somebody said they help—but they can't point to any fact to say that. Less than one out of every 1,000 supplement products has been proven to work.

"When you purchase a drug, you have confidence it'll work," says Almada. "If retailers approach that level of confidence by imparting evidence-based wisdom—fact, not faith—they'll win respect of healthcare practitioners in the area." And with that, an increase in customers in the long run.

Michael Downey is a Toronto-based freelance writer.

The second in a series of three articles.
Click here for part I.
Click here for part III.

Natural Foods Merchandiser volume XXVIII/number 2/p. 34, 38

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.