Expert panel assessment of appropriateness of abdominal aortic aneurysm surgery: global judgement versus probability estimation.

ABSTRACT

OBJECTIVE: To compare an expert panel's global assessment of appropriateness of elective surgery for abdominal aortic aneurysms (AAA) with their assessment of the effect of surgery on the probability of 5-year mortality. METHODS: Nine expert panel members rated the appropriateness of 120 scenarios for elective AAA repair on a nine-point scale, and also estimated the 5-year probability of AAA-related death and of non-AAA related death among 30-day survivors of AAA surgery and among patients with unoperated AAA. These probabilities were used to determine differences in 5-year probability of mortality of surgery vs. no surgery for each scenario. Three categories of appropriateness were defined based on these differences inappropriate (< 0%), equivocal (0-5%), and appropriate (> 5%). RESULTS: The distribution of scenarios was inappropriate (39%), equivocal (12%), and appropriate (49%) based on probability estimates and inappropriate (43%), equivocal (22%), and appropriate (36%) based on global assessment. There was poor agreement between the two methods, with a Kappa coefficient = 0.28 (95% CI 0.23 to 0.32). Although a higher proportion of scenarios were rated as appropriate using probability estimation rather than global judgment, the level of agreement among members of the panel was similar, Kappa coefficient = 0.07 (95% CI -0.07 to 0.72). CONCLUSIONS: Experts disagree about the appropriate indications for elective surgery for AAA. Explicit estimates used in a decision analysis may provide a better assessment of appropriate indications than the global judgment of experts. Global assessment of the appropriateness of AAA surgery based on panel members' review of research evidence for increased survival appears to include implicitly their valuation of outcomes.