The Certainty of Uncertainty in Health Technology Assessment 

When a new medical technology comes to market, regulators, public and private payers, employers and clinicians make decisions that affect the adoption of that new technology in patient care. Health technology assessments (HTAs), which analyze new technologies for their costs, potential impact on patient outcomes, and other societal factors, can inform those decisions. 

In a new Health Affairs Forefront article, the Tufts Medical Center's Center for the Evaluation of Value and Risk in Health (CEVR) offers an update from the pilot phase of an NPC-sponsored project exploring the potential flaws in how HTAs identify and report on uncertainty.  

“In health care, decisions can carry significant consequences for both individuals and our overall state of public health,” said Joshua Cohen, PhD, Deputy Director and Chief Science Officer of CEVR and lead author. “That’s why it’s critical to address gaps in the HTA process and pressure test whether HTAs can reliably help decision-makers understand the range of likely outcomes.”  

In Health Affairs Forefront, Dr. Cohen and his team identified two potentially problematic practices in HTA. In describing the “lamp post” problem, they noted that HTAs tend to more completely address and flesh out the range of possibilities for quantitative assumptions than they do for qualitative assumptions – probably because analysis of quantitative assumptions, like average adverse event rates, is more straight-forward than analysis of qualitative assumptions, like how to extrapolate beyond the observed data.   

A pilot study led by CEVR’s Dr. Natalia Olchanski that looked at an HTA of PARP (poly ADP-ribose polymerase) inhibitors to treat ovarian cancer conducted by the Institute for Clinical and Economic Review (ICER), found that in many cases, ICER’s HTA did not explore alternative assumptions that two expert groups convened by CEVR identified as most plausible.  CEVR found that for the PARP inhibitor assessment, use of assumptions selected by its experts produced notably different cost-effectiveness estimates than ICER reported in 9 of 12 cases. In 7 of these 9 cases, the revised cost-effectiveness estimates were more favorable than ICER’s. 

The second shortcoming that the team identified with HTAs was the “fit to print” problem – or the tendency for HTA organizations to over-simplify what they report about the range of results detailed in their technical reports. For example, ICER’s executive summary from its HTA for multiple myeloma therapies presented a narrower range of recommended prices than did its 209-page technical report, which presented a wider range of recommended prices that reflected a more complete set of areas of uncertainty.  

The CEVR authors recommend that HTA bodies more comprehensively characterize value implications attending the range of possibilities for qualitative assumptions, publish the assessment model source code to promote transparency and scientific dialogue, and comprehensively report areas of uncertainty in the HTA.  

“It’s fair for different experts to bring varying assumptions to the HTA process,” said Dr. Cohen. “But if we treat a singular set of assumptions as ‘correct’ or fail to help decision-makers appreciate the full range of possible outcomes, HTAs could contribute to poor decision-making and ultimately, put high-value treatments out of reach for patients who sorely need innovation in their care, or expend resources on therapies that do not deliver benefits that warrant their costs.”