Journal Editors Value Real-World Evidence—With Conditions

New research from the Department of Pharmaceutical Health Services Research at the University of Maryland School of Pharmacy and the National Pharmaceutical Council reveals that real-world evidence (RWE) is considered valuable by the editors of peer-reviewed journals—if it meets certain criteria for quality. The study results, which were published in the International Journal of Technology Assessment in Health Care, provide insights into how journal editors regard and assess RWE studies.

“Peer-reviewed publication is a critical step to the dissemination of research results,” says Jennifer S. Graff, PharmD, Vice President of Comparative Effectiveness Research at the National Pharmaceutical Council and a coauthor of the study. “Thus, journal editors serve as gatekeepers to the translation of evidence, including RWE, into clinical practice guidelines, reimbursement and payment policies, and clinical care.”

To obtain these insights, researchers conducted a three-part assessment of attitudes of 15 editors of peer-reviewed journals in general medicine, specialty medicine, and health policy/services research. Each editor participated in a telephone interview, an online survey, and/or an in-person roundtable discussion. Through these means, the researchers sought to answer these questions:

• What is the value of RWE and how does it compare to other types of studies, such as randomized controlled trials (RCTs)?
• What education and/or resources do journal editors provide to their peer reviewers or perceive as needed?

A specific definition of RWE was developed to ensure participants and researchers were using the term consistently: “RWE is information derived from studies using ‘real-world’ data about use, benefits, or risks of a treatment or other intervention.”

What is the perceived value of RWE in peer-reviewed journals? The researchers questioned journal editors about the relative value they assign to RWE manuscript submissions. Participants reported RWE, in general, to be valuable, offering these advantages:

• Ability to complement RCT evidence
• Ability to assess the impact of interventions in the real world
• Ability to understand treatment effects among more diverse, representative populations.

How does RWE compare to other types of studies seeking publication in peer-reviewed journals? Editors do not differentiate in the way they assess and review RWE manuscripts from RCT manuscripts. Rather, they assess the quality of all manuscripts in the same way, regardless of study design. However, some editors felt that RWE has disadvantages, including a lack of high-quality data and less established methodological standards for RWE compared to other study designs. They did acknowledge a prestige factor and comfort level with RCT designs that appear to provide advantages over RWE studies.

The characteristics of high-quality RWE studies include:

• Studies ask important (e.g., novel or relevant) research questions
• Studies fill a research gap
• Studies are generalizable
• Data sources are well aligned with the question.

By contrast, low-quality manuscripts were characterized by lack of an impactful research question, inappropriate comparators, poor-quality data sources, weak data analysis, presence of selection bias, and limited sample size.

“Importantly, none of the high-quality or low-quality characteristics were unique to RWE manuscripts; the same features were described for high-quality and low-quality intervention studies such as RCTs,” says Elisabeth M. Oehrlein, graduate student in the Department of Pharmaceutical Health Services Research at the University of Maryland School of Pharmacy and study coauthor.

The research revealed that editors are interested in improving the rigor and transparency of RWE studies, including:

• Use of online supplementary materials to facilitate research transparency
• Protocol and hypothesis-driven analysis plans, in order to differentiate results derived from pre-specified analyses from exploratory analyses with significant results
• Potential to advance the ability to replicate studies by publishing analytic code or making data available for readers and peer-reviewers.

What education and/or resources do journal editors provide to their peer reviewers? Editors reported difficulty in finding qualified reviewers. They do not frequently use training, tools, or checklists to assist peer reviewers, relying instead on existing peer reviewer knowledge. Of the small number of journals that do provide training, most employ online materials or web courses.

What do journal editors perceive as needed training or resources for their peer reviewers? Although participants did not support the use of checklists by peer reviewers, they did seek tools that might help them make decisions on manuscripts more quickly and provide constructive feedback to authors.

There are broad implications for these findings. “As research is increasingly designed to answer research or stakeholder questions, the study quality and ‘fit-for-purpose’ approach may prevail rather than historical study hierarchies in which RCTs are regarded as the ‘highest’ level of evidence,” Graff says.