Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research.

The need for pragmatic (i.e., practical) measures to evaluate implementation efforts has been repeatedly called for in the implementation science literature. This literature review focuses on understanding how pragmatism, as a measurement construct and quality of implementation measures, is defined, conceptualized and evaluated. We identified few articles (n = 9) that contribute to our understanding of how pragmatism is defined and evaluated. We found that the most frequently used terms to describe pragmatic qualities of implementation measures include "not burdensome", "brief", "reliable", "valid" and "sensitive to change". We identified one scale, the Psychometric and Pragmatic Evidence Rating Scale (PAPERS), developed to measure the pragmatic quality, as well as psychometric quality, of implementation measures. We identified several gaps and limitations of the current evidence-base and offer directions to further develop the concept and appraisal of pragmatism. Specifically, we recommend that future research focus on engaging and involving a wider diversity of implementation stakeholders in defining and conceptualizing pragmatism as well as subjecting existing pragmatic assessment measures to more rigorous and extensive evaluation.