Standards and Guidelines for Observational Studies: Quality Is in the Eye of the Beholder

Authors: Morton, SC; Costlow, MR; Graff, JS; Dubois, RW.
Publication: Journal of Clinical Epidemiology. Online ahead of print, November 2015.
Infographic: Defining High-Quality Observational Studies

Many organizations provide guidance on how to evaluate the quality of observational studies, but this research shows these standards and guidelines often lack agreement. This can lead to variations in funding priorities, research methods, publication and ultimately, the use of evidence in decision-making. The gaps identified in this research are increasingly relevant as insights from real-world clinical experience are being sought to improve patient care and reduce costs.

The researchers reviewed, compared and contrasted nine sets of standards and guidelines developed by public, private and professional societies in the U.S. and Europe. These nine sets outline how to conduct observational or real-world studies that leverage information from electronic health records, administrative claims, patient registries or data networks. Authors evaluated the presence and agreement of 23 methodological elements (such as the need to use a study protocol, how to link data from different sources, or handle missing information) and compared how actionable each element was.

The authors found that, out of the 23 methodological elements, 14 (61 percent) were addressed by seven or more standards and guidelines, reflecting general agreement that these elements are important. However, for all but two of these 14 elements (defining study objectives and research questions; including details on data sources), there was disagreement on how the element should be addressed or acted upon. The remaining 11 elements varied in whether the sets of standards and guidelines agreed that the element was important or was included at all. Just over half (57 percent) of the 23 methodological elements were considered actionable.

To create a consensus-based approach and engage stakeholders from different disciplines, the authors made four key recommendations:

• Gain Alignment and Consensus—While audiences and research focuses may differ among disciplines and types of stakeholders, good practices in conducting observational studies should not. 
• Identify the Level of Consensus—Minimum standards and guidelines may serve better than best practices – this approach would allow stakeholders to develop new standards and guidelines that would raise the bar for study conduct gradually over time. 
• Accomplish Consensus—Gaining consensus would likely need to occur via multiple and periodic meetings among stakeholders.
• Encourage or Enforce—While enforcement of minimum requirements would be needed, the broadest adoption of common standards would most likely be achieved with voluntary approaches.