Lack of Alignment Between WHO Joint External Evaluation and State Party Self-Assessment Scores Undermines Utility as Evaluation Tools for the Department of Defense.

ABSTRACT

The International Health Regulations 2005 (IHR) set standards for countries to detect and respond to public health threats such as COVID-19. The US Department of Defense engages with partner nations to build IHR-related health security capacities. In this article, we compare 2 elements of the IHR Monitoring and Evaluation Framework to determine if they align in a useful way. The version of the State Party Self-Assessment Annual Reporting (SPAR) tool used for this study is a self-assessment of 13 capacities, while the Joint External Evaluation (JEE) requires collaboration with international subject matter experts to evaluate 19 capacities. The SPAR indicators are scored separately from 0% to 100%, whereas the JEE uses a rank-ordered scale from 1 to 5 for variable numbers of indicators in each capacity. Using 2018-2019 data from the World Health Organization, we quantitatively and qualitatively evaluated the alignment of the SPAR and JEE scoring systems, using paired t tests for related capacities and 3 approaches to matching the scales. Whether using a simple, evenly divided scale for the SPAR or downscaling the SPAR scores to match with lower JEE scores, the paired t tests indicate that the JEE and SPAR scoring systems are not aligned. Many of the capacities in the JEE and SPAR are defined differently, pointing to one of the reasons for the discordance. We discuss implications for revision of the JEE and SPAR assessment tools along with ways in which the scores might be used for planning global health engagement capacity-building activities.