Comparison of Assertive Community Treatment Fidelity Assessment Methods: Reliability and Validity.

Assertive community treatment is known for improving consumer outcomes, but is difficult to implement. On-site fidelity measurement can help ensure model adherence, but is costly in large systems. This study compared reliability and validity of three methods of fidelity assessment (on-site, phone-administered, and expert-scored self-report) using a stratified random sample of 32 mental health intensive case management teams from the Department of Veterans Affairs. Overall, phone, and to a lesser extent, expert-scored self-report fidelity assessments compared favorably to on-site methods in inter-rater reliability and concurrent validity. If used appropriately, these alternative protocols hold promise in monitoring large-scale program fidelity with limited resources.

Measuring the recovery orientation of assertive community treatment.

BACKGROUND: Approaches to measuring recovery orientation are needed, particularly for programs that may struggle with implementing recovery-oriented treatment. OBJECTIVE: A mixed-methods comparative study was conducted to explore effective approaches to measuring recovery orientation of assertive community treatment (ACT) teams. DESIGN: Two ACT teams exhibiting high and low recovery orientation were compared using surveys, treatment plan ratings, diaries of treatment visits, and team leader-reported treatment control mechanisms. RESULTS: The recovery-oriented team differed on one survey measure (higher expectations for consumer recovery), treatment planning (greater consumer involvement and goal-directed content), and use of control mechanisms (less use of representative payee, agency-held lease, daily medication delivery, and family involvement). Staff and consumer diaries showed the most consistent differences (e.g., conveying hope and choice) and were the least susceptible to observer bias but had the lowest response rates. CONCLUSIONS: Several practices differentiate recovery orientation on ACT teams, and a mixed-methods assessment approach is feasible.

A comparison of phone-based and on-site assessment of fidelity for assertive community treatment in Indiana.

OBJECTIVE: This study investigated the reliability and validity of a phone-administered fidelity assessment instrument based on the Dartmouth Assertive Community Treatment Scale (DACTS). METHODS: An experienced rater paired with a research assistant without fidelity assessment experience or a consultant familiar with the treatment site conducted phone-based assessments of 23 teams providing assertive community treatment in Indiana. Using the DACTS, consultants conducted on-site evaluations of the programs. RESULTS: The pairs of phone raters revealed high levels of consistency [intraclass correlation coefficient (ICC)=.92] and consensus (mean absolute difference of .07). Phone and on-site assessment showed strong agreement (ICC=.87) and consensus (mean absolute difference of .07) and agreed within .1 scale point, or 2% of the scoring range, for 83% of sites and within .15 scale point for 91% of sites. Results were unaffected by the expertise level of the rater. CONCLUSIONS: Phone-based assessment could help agencies monitor faithful implementation of evidence-based practices.