In 2010, ingredient supplier Applied Food Science conducted a study in India on whether a green coffee bean extract could reduce body weight and body fat. The six-week study on 16 overweight subjects found an incredible 17-pound reduction in weight, 6.4-pound reduction in body mass index, and 4.44 percent reduction in body fat. The results were so impressive that Dr. Oz touted green coffee bean extract on his daytime show. The green coffee bean extract boom was on.
Until it wasn't. In 2014, the Federal Trade Commission sued Applied Food Science for violating the FTC Act, which prohibits "unfair or deceptive acts or practices in or affecting commerce." The company settled for $3.5 million.
The FTC said there were "design and methodological flaws" in the study, including an egregious act of the researchers in India repeatedly altering the subjects' weight and other data. In other words, the too-good-to-be-true study results were in fact just that - too good to be true. But that's not all.
The study was retracted. The study was on a different population (Indians) with a different diet and lifestyle than that for which the product was going to be marketed (Americans). The study was conducted over only six weeks. The sample size was a mere 16 subjects. And Applied Food Science contracted third-party researchers to publish the results without verifying the data that was collected.
There was no shortage of missteps in the infamous case, but this is hardly the only mistake companies make with conducting research studies to write marketing claims and boost sales.
"Historically, our industry has brought products to market based on testimonials until the regulatory bodies,such as the FTC, began taking actions against certain products," said Bibiane Zakaria, director of sales and client services for KGK Synergize, a contract research organization (CRO) that conducts clinical trials for clients. "The right way is to do the science first and let the findings guide what you can say on your product. Our industry does not invest what's needed or required when researching their products. I have definitely seen an improvement over the last four years, but we still have a long way to go."
Bad science abounds
Often, a company will pirate science that other companies conduct on similar or even the same ingredient, sometimes without a close look at that study. Even when research studies are well designed and the results are positive, marketers still make claims that do not realistically align with what the research says. Beyond that, too many companies will release a product with the same ingredient but not the same dosage that the research has shown to be efficacious.
About 10 percent of companies exhibiting at Natural Products Expo West and East run afoul of New Hope standards, which are crafted to align with federal regulations around proper claims-making rules. (Disclosure: NBJ is part of New Hope.) The New Hope Standards Department carefully vets every exhibitor at Natural Products Expos and every advertiser for any New Hope publication or website for the claims they make.
"If we can set you straight before you take your claims public, that definitely decreases the chances of you hearing from the FDA or FTC," said Michelle Zerbib, New Hope standards director. "Very few exhibitors get upset about our work. The majority greatly appreciate the service we provide."
Zerbib said that there are four common missteps companies tend to make:
• Taking pilot study data and extrapolating that to the general population
• Taking study claims for an individual ingredient and extrapolating that to the finished product
• Inserting "pixie dust" quantities of ingredients and making claims based on higher ingredient levels instead of what is actually in their product.
• Making drug claims – supplements cannot claim to treat, prevent or cure any
disease condition. That is strictly under the purview of pharmaceuticals.
"It happens a lot," Zerbib said. "Our primary goal is education so companies know the correct way to make claims in a regulated environment. We will also sometimes advocate that companies search for service providers that can help them bridge the divide between research and marketing claims."
Ultimately, there is a significant gap within companies regarding the marriage of science, regulatory and marketing, according to Risa Schulman, Ph.D., founder and president at Tap-Root, consultancy guiding companies on substantiating claims and appropriate communications. "The regulatory team, the R&D team and legal each have an expertise and they duke it out to create a label," she said. "Research is concerned with fact; marketing is concerned with story. But what falls through the cracks is the accuracy of the science, which is often understated or overstated."
The result, she added, is store shelves filled with too many products based on medium science.
"Everyone's going after the same claims with the same family of ingredients," said Schulman. "There's some percentage of products that are well-researched. And then there's the middle 40 percent of products that are the same stuff and they're trying to differentiate, and how do you do it? This one's more bioavailable, one will say. They're looking for something to talk about, and sometimes it's lame."
And this leads to the related problem of making claims with irrelevant consumer-resonant end points. Say you have an omega-3 DHA and you have a study that shows it increases blood levels of DHA twice as high as conventional fish oil. Sounds good, right? But what if you have a study showing increased DHA levels in the membranes of the red blood cells twice as high as fish oil. Better, right? But what if you have a study showing it decreases psoriasis and other skin conditions? That's what consumers really want – they want their visible or noticeable health condition solved.
"You should want to focus on patient-relevant endpoints," said Anthony Almada, CEO of Vitargo Global Sciences. "You can talk all you want that a probiotic changes the composition of gut flora. Who cares if you still have diarrhea?"
An idea for a proper inflammation-based study - as might be conducted with current ingredient curcumin - would be to not just conduct a study showing curcumin decreased levels of inflammation biomarker C-reactive protein but also conduct a study showing decreased pain levels and overall improvements in quality of life, as a WOMAC score might reveal. That way, you decrease the odds of a study being disregarded as benefits due to a simple placebo effect because you have the underlying shift in biochemical markers to validate the patient "feeling."
"If you have just markers, it's not consumer-relevant," said Almada. "You need a connected study that links both."
Almada also pointed out another flaw that may elude even diligent science-based marketers. And that's having a robust comparative study against not merely placebo but against a generic version of your ingredient.
"Even Red Bull, which has 12 or 13 studies, has never compared it to a drink with equal amounts of caffeine," said Almada. "They say it's too generic. Wave the cocktail magic wand! It's interactive and synergistic! Well, show us the comparative study. What if it's no better than caffeine? How about the same study but with no caffeine? It's an intentional disservice."
Another example of an improper ingredient match-up is a study on the relative superiority of different sources of omega-3s (Kagan, 2013). Researchers concluded that blood levels of EPA were higher with algal oil than with krill oil, suggesting that an omega-3 from algae is superior to krill. But the algae provided 1.5 grams EPA and the krill oil provided 1.02 grams EPA, so the results relied on different amounts of the measured active ingredient. Nevertheless, you can be sure the algae omega-3 marketers made hay with that study proclaiming algae is the preferred omega-3 source. Consumers might end up with higher concentrations of EPA in their blood, but not for the reasons the marketing might imply. A higher dosage doesn't equal more bioavailability.
So how's a company supposed to evaluate existing evidence properly, and then communicate it in a way that will still resonate with consumers while still abiding by the claims laws ruled over by regulatory bodies?
"It requires someone who knows how to critically read and evaluate the literature and can understand and apply it within the regulatory framework of FDA and FTC," said Schulman. "How should companies design their own study? With help. Period."
The value of hiring out to a qualified CRO is two-fold. One, the study can be conducted faster. Two, that's all they do. In an age of specialists, if you've got a heart problem, why go to a general practitioner?
"The most advantageous reason, indeed why this sector has emerged," notes Almada, "is that they don't own the results. So if you put in $75,000 and it doesn't come out the way you want it, the results can be buried. If it's never published, only the company need know the results.
That could mean that even if you do things the exact right way with the exact right people on the exact right ingredient at the exact right dose, there's still a loophole if you need a way out. Cherry picking the findings is not exactly the scientific method incarnate. But it's often what passes for it in today's nutrition industry.