Research Highlights

Back to News Archive
Scientific journals fail to enforce standards for reporting on clinical trials in non-regulated health care interventions
Research into health care interventions that are not directly regulated by government agencies - including surgery, rehabilitation, psychological treatments, nursing, and nutrition and diets - is most often published without public registration of any pre-defined primary outcomes. Without this important step, researchers can be selective about the data they report. In many cases, this leads to publication of misleading results about the effectiveness of interventions, but does not accurately reflect what really occurred in a trial. In by far the largest study of its type, researchers from the Lady Davis Institute (LDI) at the Jewish General Hospital examined 953 trials of non-regulated interventions published in 254 journals from the areas of surgery, rehabilitation, psychology, nursing, and nutrition. They did not identify any journals that consistently enforced pre-trial registration standards and reported that only 20% of the trials they reviewed had been registered before they were initiated. The study is published in JAMA Internal Medicine.

Most clinical trials of drugs, biologics and medical devices are legally required to register trial plans at open sites such as before beginning the trial and to adhere to these plans. Top medical journals also require pre-registration as a condition for publishing results. This ensures that trials results are not hidden from the public and that key trial outcomes are not changed after the fact to enhance the appearance of the effectiveness of a particular intervention.

“There is a long history, unfortunately, of investigators not reporting trial results at all or of reporting skewed versions of what actually happened in a trial,” explains Dr. Brett Thombs, senior investigator at the LDI and Professor of Psychiatry at McGill University, who led the study. “Cherry picking results can distort our understanding of whether or not an intervention benefits patients. Imagine a trial that measures ten outcomes and finds only one positive result out of the ten. If the researcher were compelled to reveal all ten findings, the intervention would be considered a failure. However, if that researcher publishes only the positive finding, it would be deemed a success and probably provided to patients.”

Dr. Thombs’ team found that only 11% of the 254 specialty journals whose contents they reviewed had a stated policy requiring pre-trial registration. However, only 34% of trials appearing in those journals were registered prospectively, compared to 18% in journals with no policy. Most remarkably, only 4% of the 953 trials they reviewed published primary outcomes that could be traced back and were consistent with a pre-trial registration.

“Unless objectives for a study are clearly stated in advance, there is no way to prevent bias from creeping in,” said Marleine Azar, an MSc student in epidemiology from McGill University who was first author on the paper and who is currently studying medicine at the University of Montreal. “Because governments do not enforce standards on trials of non-regulated interventions, we rely on journal editors to ensure that accepted standards are enforced, but that is not happening.”

The authors suggest that academic ethics boards or research funding agencies could be effective in making sure that pre-registration of trials occurs.

“Without transparency there is no way to know which trials accurately represent what would really occur if a treatment is applied,” insists Dr. Thombs. “As researchers, we owe it to the public to make sure that they have access to well-conducted, honestly reported science that allows them to make the best possible decisions about their healthcare. We are falling far short of this goal.”

Evaluation of Journal Registration Policies and Prospective Registration of Randomized Clinical Trials of Nonregulated Health Care Interventions
Marleine Azar, MSc, Kira E. Riehm, MSc, Nazanin Saadat, BSc, Tatiana Sanchez, BSc,
Matthew Chiovitti, MISt, Lin Qi, BSc, Danielle B. Rice, MSc, Brooke Levis, MSc, Claire
Fedoruk, BA, BSc, Alexander W. Levis, MSc, Lorie A. Kloda, PhD, Jonathan
Kimmelman, PhD, Andrea Benedetti, PhD, Brett D. Thombs, PhD
JAMA Internal Medicine

For media inquiries, and to arrange interviews with Dr. Thombs contact:
Tod Hoffman
Research Communications Officer
Lady Davis Institute
Tel.: 514-340-8222 x 28661

Support research at the Lady Davis Institute - Jewish General Hospital