Psychometric properties of two implementation measures: Normalization MeAsure Development questionnaire (NoMAD) and organizational readiness for implementing change (ORIC).

ABSTRACT

Background: Effective interventions need to be implemented successfully to achieve impact. Two theory-based measures exist for measuring the effectiveness of implementation strategies and monitor implementation progress. The Normalization MeAsure Development questionnaire (NoMAD) explores the four core concepts (Coherence, Cognitive Participation, Collective Action, Reflexive Monitoring) of the Normalization Process Theory. The Organizational Readiness for Implementing Change (ORIC) is based on the theory of Organizational Readiness for Change, measuring organization members' psychological and behavioral preparedness for implementing a change. We examined the measurement properties of the NoMAD and ORIC in a multi-national implementation effectiveness study. Method: Twelve mental health organizations in nine countries implemented Internet-based cognitive behavioral therapy (iCBT) for common mental disorders. Staff involved in iCBT service delivery (n = 318) participated in the study. Both measures were translated into eight languages using a standardized forward-backward translation procedure. Correlations between measures and subscales were estimated to examine convergent validity. The theoretical factor structures of the scales were tested using confirmatory factor analysis (CFA). Test-retest reliability was based on the correlation between scores at two time points 3 months apart. Internal consistency was assessed using Cronbach's alpha. Floor and ceiling effects were quantified using the proportion of zero and maximum scores. Results: NoMAD and ORIC measure related but distinct latent constructs. The CFA showed that the use of a total score for each measure is appropriate. The theoretical subscales of the NoMAD had adequate internal consistency. The total scale had high internal consistency. The total ORIC scale and subscales demonstrated high internal consistency. Test-retest reliability was suboptimal for both measures and floor and ceiling effects were absent. Conclusions: This study confirmed the psychometric properties of the NoMAD and ORIC in multi-national mental health care settings. While measuring on different but related aspects of implementation processes, the NoMAD and ORIC prove to be valid and reliable across different language settings.

ABSTRACT

Why was the study done? Effective interventions need to be implemented successfully to achieve impact. Reliable measurement instruments are needed to determine if an implementation was successful or not. Two theory-based instruments exist for measuring the effectiveness of implementation strategies and monitor progress. The NoMAD measures aspects of normalization related to sense-making, willingness to implement, the work people do, and reflection. The Organizational Readiness for Implementing Change (ORIC) measures organization members' preparedness for implementing a change. What did we do? This study examined whether the NoMAD and ORIC measure what they are supposed to measure. We translated the instruments from English to eight languages (Albanian, Danish, Dutch, French, German, Italian, and Spanish/Catalan) We applied various statistical methods to confirm the measurement properties, including correlations of scales, factor structures, test­retest reliability, consistency and floor and ceiling effects. 318 mental health professionals from nine countries participated in the study. What did we find? For both instruments, total scores can be used as well as the subscale scores. Internal consistency for ORIC was high and for NoMAD adequate. Test­retest reliability was demonstrated, and floor and ceiling effects were rare. What does this mean? NoMAD and ORIC are reliable instruments for measuring implementation processes and outcomes across mental health care settings in different countries and languages. They measure related but different aspects of implementation processes and outcomes. The measures are brief, and theory supported. However, more work is to be done on interpreting scores in relation to implementation success and regarding changes over time.