Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study.

Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions-complete and 20-min segments-for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ2 = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention.

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist's delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist's selection of intervention strategies tailored to the family's context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

Proactive Parent Engagement in Public Schools: Using a Brief Strengths and Needs Assessment in a Multiple-Gating Risk Management Strategy.

This study examined the viability of a brief, parent-reported strengths and needs assessment as the first step in a multiple-gating approach to proactive positive behavior support for families. The Positive Family Support-Strengths and Needs Assessment (PFS-SaNA) was designed to collaboratively engage parents early in the school year in a home-school coordinated Positive Family Support (PFS) system. In this study, we evaluated the reliability and validity of the PFS-SaNA in the context of public middle schools. Findings suggest that the 14-item, unidimensional PFS-SaNA shows convergent validity with teacher ratings of risk. It can be easily and cost-effectively used by school personnel when parents register their children for school at the beginning of each school year.