Comments on “Request for Information on Development of an Inventory of Comparative Effectiveness Research”

Sherry A. Glied
Assistant Secretary for Planning and Evaluation
Department of Health and Human Services
200 Independence Avenue, SW
Washington, DC 20201
Online submission:
Attention: CER Inventory

Dear Dr. Glied,

The National Pharmaceutical Council (NPC) is pleased to provide comments on the “Request for Information on Development of an Inventory of Comparative Effectiveness Research” by the Office of the Assistant Secretary for Planning and Evaluation, Department of Health and Human Services (HHS) [FR Doc 2010-17244].

NPC is a policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. Founded in 1953 and supported by the nation’s major research-based pharmaceutical companies, NPC focuses on research development, information dissemination, education and promotion of the critical issues of evidence, innovation and the value of medicines for patients. NPC recognizes the important role comparative effectiveness research (CER) can provide in improving patient outcomes by developing and disseminating evidence-based information for patients, clinicians, and other decisions makers.

As envisioned, the CER Inventory will be designed to “facilitate access to CER for interested stakeholders; and assist in identifying research priorities and gaps for future research.” While an inventory of CER may serve many functions, it is critical given the perceived imprimatur of an HHS CER Inventory that information included in the inventory is generated using rigorous standards and methods recognized by expert bodies. The inventory also should allow adequate and sufficient information for diverse stakeholders to appropriately interpret results and to inform their treatment decisions. Thus, NPC’s comments are focused on ensuring the quality and rigor of CER studies included in an inventory of CER, the data elements for inclusion, coordination with the Patient-Centered Outcomes Research Institute (PCORI), and the integration of tools to aid diverse stakeholders in appropriately interpreting results to enable timely and actionable treatment decisions.

1) Sources of CER

a. Consistency of CER definitions

NPC encourages the types and sources of CER included in the inventory to remain consistent with the definition and framework outlined in the PCORI provisions in the Patient Protection and Affordable Care Act (PPACA). Consistent with PPACA language, the types and sources of CER nformation included in an inventory should encompass all health care services including rocedures, medical devices, diagnostic tools, pharmaceuticals, health care management and health care delivery systems. CER sources should be rigorous and transparent, utilize a full range of types and sources of evidence, and remain focused on comparative clinical effectiveness.

b. Limited to rigorous and transparent research conducted with a clear set of guidelines

As this inventory is envisioned to be hosted by HHS, the data included may be construed as government-endorsed evidence. This amplifies the need for studies to be rigorous, transparent, and conducted with a clear set of methodological guidelines. Within 18 months from the establishment of PCORI, the Methodology Committee will develop and identify standards for research that are scientifically based. Once available, the methodology standards for PCORI should be utilized as the minimum criteria for studies to be included in an inventory. In the interim, the framework identified by the GRACE Principles (Good ReseArch for Comparative Effectiveness, may assist in evaluating the quality of observational CER studies for inclusion.

c. Quality assurance and validation of available information

NPC recommends that appropriate measures be developed to ensure the accuracy and completeness of inventory information. Inclusion of all CER information without sufficient details on the conduct of studies or a vetting of the information to ensure completeness of data will hamper the users’ ability to discern the quality and utility of such information. Sources of information such as gray literature may not have been designed with similar intent nor include all relevant information needed for a CER inventory. Additionally, without appropriate oversight or knowledge, there could be variability in information leading to inadequate and inaccurate information available for end users.

2) Data Elements

a. Consideration of subpopulations

NPC encourages the appropriate consideration and reporting of potential differences in treatment effects among individual patients and subpopulations (i.e. age, gender, genetic variations, or presence of co-morbidities). This would encourage the measurement and reporting of subpopulations, when appropriate, which will best inform individual patient decisions.

b. Reporting of results

NPC suggests that an inventory include not only the research findings, but also the strengths, limitations, potential for generalizability, and external validity of the study. Furthermore, the inventory of CER should remain consistent with the PCORI provisions for releasing research findings and peer-review. Such elements may assist diverse users in judging the quality and applicability to their individual treatment decisions.

c. Consideration of new information

NPC strongly emphasizes the need for an inventory to have the opportunity to be amended with new information as new data emerges. It also will be important to include the collection timeframe of the original source data and the publication dates to allow users to understand how evidence may have evolved. This will help foster an environment where innovation in health care delivery and products is valued.

3) Sustainability

a. Coordination and efficiency with PCORI processes

The American Recovery and Reinvestment Act of 2009 created the Federal Coordinating Council for Comparative Effectiveness Research (FCCCER) to foster optimal coordination of CER conducted or supported by the federal government. The FCCCER was subsequently terminated in 2010 upon the establishment of PCORI. Given the suggestions highlighted above regarding the rigor and quality of studies to be included in an inventory of CER, the lack of existing clear methodologic standards, and the considerations in reporting research findings, the inventory can best ensure the reliability and integrity of information by limiting the inventory scope initially to publicly funded CER. Additionally, PCORI will be established in the near term and the PCORI methods, standards, and processes will be developed, thus enabling a more coordinated and efficient use of resources to develop the inventory once PCORI recommendations and standards are in place.

4) Additional Considerations

a. Develop and integrate appropriate interpretation tools in the CER Inventory

The CER Inventory is intended for individuals from diverse backgrounds with varying levels of technical expertise. As such, the CER evidence must be accessible in ways that are intuitive for all users. To date, there has been less focus on the tools for all stakeholders to use in interpreting CER. To minimize the likelihood for inaccurate or inappropriate interpretation of CER evidence by end users, appropriate translation and interpretation tools must be created and integrated into an inventory of CER.

As outlined in PPACA, the Agency for Healthcare Research and Quality (AHRQ) Office of Communications and Knowledge Transfer (OCKT) will create informational tools that aid the organization of findings for stakeholders. Such tools should be created and integrated prior to the public release of an inventory.

NPC and other organizations have already given thought to questions that could form the basis for the development of interpretation and translation tools. In 2009, NPC sponsored a white paper, “Demystifying Comparative Effectiveness Research: A Case Study Learning Guide” to assist end-users in understanding if CER findings are applicable to their treatment questions. It includes three considerations:

1. For whom are the findings applicable?
2. Are there any aspects of the study design the might affect the results?
3. Are the findings likely to change with new research?

NPC supports the use of CER to provide health care professionals and patients with timely and relevant information to make decisions that support quality patient health outcomes. To accomplish this goal, the information in a CER inventory must be credible and and include reliable information from studies conducted with appropriate and accepted methods. It also must be accompanied by tools that assist stakeholders in interpreting and translating evidence. For this reason, NPC recommends that HHS work closely with the PCORI Board, PCORI Methods Committee, and AHRQ OCKT. An inventory will be of limited value if it is perceived to be incomplete, inaccurate or in conflict with the new CER entity, PCORI and its recommendations. NPC appreciates the opportunity to comment on this important issue.

If you have questions or need additional information, please do not hesitate to contact me at 703-715-2750 or

Sincerely,Dan Leonard