Mock choc study highlights peer review pitfalls
The incident shows how authenticating this kind of spurious research damages all parties – from general academia, to the nutrition science community, to the food and supplements industries and perhaps most crushingly, consumers.
Peer review needs to change.
Professor Johannes Bohannon from the Institute of Diet and Health showed how the whole cycle: Conceiving research ideas; conducting science; having it published in a peer-review journal and reported on in the broader media can be a total farce.
Professor Bohannon even left clues that his research was not all it appeared to be – such as hyperbolic conclusions and linking to music videos – but even that failed to halt publication in March of the study in the International Archives of Medicine (Vol 8) and subsequent reportage from mainstream newspapers around the world.
An example from the Daily Express tabloid newspaper in the UK: ‘Chocolate accelerates weight loss: Research claims it lowers cholesterol and aids sleep’
The study made news in more than 20 countries and half a dozen languages.
But it was a hoax.
Professor Johannes Bohannon is actually John Bohannon, a journalist, and the Institute of Diet and Health nothing more than a website he set up. Bohannon does possess a PhD but it is in bacterial molecular biology.
The study was real but poorly designed, underpowered (sample of 16) and the conclusions drawn from it highly manipulative of the data at hand.
"The only problem with the diet science beat is that it’s science. You have to know how to read a scientific paper—and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical."
Professor John Bohannon (aka Professor Johannes Bohannon)
The study conclusion stated: “Long-term weight loss, however, seems to occur easier and more successfully by adding chocolate. The effect of the chocolate, the so-called ‘weight loss turbo,’ seems to go hand in hand with personal well-being, which was significantly higher than in the control groups.”
It went unchallenged in both peer review and in the media.
As Bohannon said in a recent blog on the matter: “You can’t even trust the weight loss that our non-chocolate low-carb group experienced versus control. Who knows what the handful of people in the control group were eating? We didn’t even ask them.”
Bohannon noted some journals are upholding standards, even as others continue to put payment over peer review – it costs €600 to gain publication in the International Archives of Medicine.
Peer review: Open-source wounds
Bohannon is a correspondent for Science magazine. In 2013 he conducted an experiment where he sent seriously flawed research to more than 300 open-source journals. About half accepted the research. See the results here.
“Some journals are trying to phase out 'p' value significance testing altogether to nudge scientists into better habits,” wrote Bohannon. “And almost no one takes studies with fewer than 30 subjects seriously anymore. Editors of reputable journals reject them out of hand before sending them to peer reviewers. But there are plenty of journals that care more about money than reputation.”
Indeed Bohannon sent the paper ‘Chocolate with high cocoa content as a weight-loss accelerator’ to 20 open-source journals. Several accepted it within 24 hours before he chose to publish with the International Archives of Medicine. The publication followed rapidly and as Bohannon notes, “Not a single word was changed”.
Then the next phase of the cycle: Media manipulation.
“The key is to exploit journalists’ incredible laziness. If you lay out the information just right, you can shape the story that emerges in the media almost like you were writing those stories yourself. In fact, that’s literally what you’re doing, since many reporters just copied and pasted our text.”
Dodgy research, dodgy review, dodgy publication, dodgy reporting. So much dodgniness. When a dodgy music video that appears in the official press release doesn’t trigger alarm bells something is seriously amiss.
Is it any wonder consumers are so sceptical and groups like the National Confectioners' Association (NCA) engage in campaigns to convince them otherwise?
Nutrition science is better than this.
