Interested in sharing the research? You can order print copies of many research publications.

Order Reprints

Newsletter Sign-Up

Keep up to date with NPC news and events by signing up for our e-newsletter E.V.I.dently, and our CER Daily Newsfeed.

Subscribe POLICY

Making Informed Decisions: Assessing the Strengths and Weaknesses of Study Designs and Analytic Methods for Comparative Effectiveness Research

Comparative effectiveness research (CER) aims to close a gap in evidence by producing information that decision-makers can use to make informed treatment and coverage decisions. CER therefore must be designed to meet the real-world needs of decision-makers. This practical focus of CER introduces unique requirements for the design and implementation of studies. For example, tradeoffs of validity, relevance, feasibility, and timeliness must be considered in the context of the specific decision-makers and decisions. These unique considerations lead to questions concerning which study designs and methods are appropriate for CER questions.

Understanding which approach to conducting a CER study is best to use under which circumstances is a question of significant debate among researchers. Generally, CER approaches fall into two broad categories: experimental study designs and methods, and nonexperimental study designs and methods. In experimental designs, patients are randomized (assigned by chance, not by a physician’s decision) to a particular therapy based on the study protocol. In nonexperimental designs, patients and physicians make real-world treatment decisions, and patterns of care and outcomes are observed. Some argue that experimental study designs are needed to answer most CER questions, because randomization eliminates concerns regarding channeling bias (ie, the tendency of clinicians to prescribe specific treatments based on a patient’s prognosis), and achieves balance with regard to measured and unmeasured confounders (ie, extraneous variables that may play a role in the outcomes of interest). Others believe that nonexperimental studies, incorporating ways to address channeling bias and other confounding in the design and/or analysis, represent important alternatives to randomized studies. In practice, each research approach has advantages and disadvantages, and the research approach for a CER question should be selected based upon the specific features or characteristics of the study question.

The purpose of this document is to provide brief descriptions of both experimental and nonexperimental study designs and methods that may be used to address CER study questions. Each design or analytic topic is described, along with the strengths and limitations associated with the approach. Examples are provided to demonstrate the use of the described methods in the literature. While this document does not prescribe methodologies for specific CER research questions, it is part of a larger effort to develop a systematic approach to determining which methods are best able to address given CER questions. In its current form, this document provides information needed for researchers and consumers of the literature to understand the relative strengths and limitations of various CER design and analytic approaches, and how their use may affect study results and interpretation.

This document is organized into four sections: experimental study designs; experimental methods; nonexperimental study designs; and nonexperimental methods. It was published by the National Pharmaceutical Council, Outcome, and the Center for Medical Technology Policy.

Newsletter Volume

E.V.I.dently June 2016

MessageNPC Looks at Updated ASCO Framework, Conversation Around ValueThe American Society of Clinical Oncology (ASCO) released its updated value...
Blog Post

CER Tweets of the Week: PIPC Briefing on Value Frameworks, Evidence-Based Medicine in Health Journalism (June 20-24, 2016)

Last week, comparative effectiveness research (CER) stakeholders tweeted about the Partnership to Improve Patient Care’s (@PIPCpatients)...
Blog Post

CER Tweets of the Week: Access to Evidence-Based Resources, Value-Based Payment Models (June 13-17, 2016)

What have comparative effectiveness research (CER) stakeholders been tweeting about recently? Last week, we saw many conversations about a new...
Blog Post

NPC’s Dr. Graff on Developing Evidence That Is “Fit for Purpose”

What information do payers feel they need to guide their coverage decisions? Jennifer Graff, PharmD, vice president of comparative effectiveness...
Press Release

Data Can Offer Critical Insights, But Roadblocks to Data Access Persist

(Washington, D.C., April 1, 2016)—Data is key to improving health outcomes and creating efficiencies in our health care system, but a new study...
Press Release

Fit for Purpose? New NPC, AcademyHealth Framework Closes Gaps Between Payer Needs and Research Answers

(Washington, DC, September 21, 2015)—Despite multi-million dollar public and private investments to tap into big data and improve the evidence...
Press Release

NPC Comments on CMS’ Announcement to Allow Broader Access to Publicly Funded Databases

(Washington, DC, June 2, 2015)—The National Pharmaceutical Council (NPC) today commented on the Centers for Medicare and Medicaid Services’ (CMS)...
Press Release

State Medicaid Directors Share Views on Comparative Effectiveness Research And Its Impact on Policy in National Pharmaceutical Council Survey

(Washington, DC, April 16, 2015)—A new survey conducted on behalf of the National Pharmaceutical Council (NPC) shows that state Medicaid directors...
Commentary and Testimony Page

The Importance of Evidence, Value, Access and Innovation in 2016

Last year, in our 2015 Health Care Outlook published in Chain Drug Review, I highlighted the recent announcement by the U.S. Department of Health and...
Commentary and Testimony Page

Chain Drug Review – Commentary: “Health Care Outlook for 2015”

This commentary was originally published in the January 5, 2015 issue of Chain Drug Review.By: Dan Leonard, President, National Pharmaceutical...
Commentary and Testimony Page

NPC Comments on PCORI Proposal for Peer Review of Primary Research and Public Release of Research Findings

November 7, 2014Dr. Joe SelbyExecutive DirectorPatient-Centered Outcomes Research Institute1828 L St., NW, Suite 900Washington, DC 20036Dear Dr....
Commentary and Testimony Page

The State of CER and the Environment for Health Care Decision Making

This month, NPC released its third annual survey on stakeholders’ views on comparative effectiveness research (CER), “The State of Comparative...
YouTube Videos

2016 Comparative Effectiveness Research and the Environment for Health Care Decision-Making

This one-hour webinar highlights the results of NPC's annual survey of comparative effectiveness research (CER) stakeholders and includes a panel...
YouTube Videos

When Is Evidence Fit for Purpose? Closing Gaps Between Researchers, Payers

Despite multi-million dollar public and private investments to tap into big data and improve the evidence available for health care decision-making,...
YouTube Videos

Comparative Effectiveness Research and the Environment for Health Care Decision-Making

During a March 26, 2015 webinar, the National Pharmaceutical Council discussed the results of its annual stakeholder survey on comparative...
YouTube Videos

Real-World Impact of Comparative Effectiveness Research Findings on Clinical Practice

National Pharmaceutical Council Chief Science Officer Robert W. Dubois, MD, PhD, discusses a peer-reviewed study suggesting that changes are needed...
Blog Post

CER Tweets of the Week: PIPC Briefing on Value Frameworks, Evidence-Based Medicine in Health Journalism (June 20-24, 2016)

Last week, comparative effectiveness research (CER) stakeholders tweeted about the Partnership to Improve Patient Care’s (@PIPCpatients)...
Blog Post

CER Tweets of the Week: Access to Evidence-Based Resources, Value-Based Payment Models (June 13-17, 2016)

What have comparative effectiveness research (CER) stakeholders been tweeting about recently? Last week, we saw many conversations about a new...
Blog Post

NPC’s Dr. Graff on Developing Evidence That Is “Fit for Purpose”

What information do payers feel they need to guide their coverage decisions? Jennifer Graff, PharmD, vice president of comparative effectiveness...
Blog Post

Tackling Data Access at Health Datapalooza

By Jennifer Graff, PharmD, NPC VP for Comparative Effectiveness ResearchWe’re looking forward to participating in Health Datapalooza next week, which...

Got CER? Educating Pharmacists for Practice in the Future: New Tools for New Challenges

This study provides an early evaluation of the CER Collaborative's training program's impact on...

Data, Data Everywhere, But Access Remains a Big Issue for Researchers

This study captures the policy inconsistencies and hurdles that can hinder use of publicly funded...
cersurvey16

2016 Comparative Effectiveness Research and the Environment for Health Care Decision-Making

NPC's sixth annual survey of stakeholder views on comparative effectiveness research (CER) and the...
  •  
  • 1 of 14
  • >