RESUMEN
Background: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. Methods: In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. Results: Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%. Conclusions: These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. Trial registration: NCT03013309 ClinicalTrials.gov.
Research has shown that evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. However, if they are not implemented with fidelity, there is a potential that they will not produce the same effects. Gold-standard methods of implementation monitoring include observations of program sessions. This is expensive and difficult to implement in delivery settings with limited resources. Using data from a trial of the Family Check-Up 4 Health program in primary care settings that served Latino families, we investigated the potential to make use of a form of machine learning called natural language processing (NLP) to monitor program delivery. NLP-based ratings were significantly associated with independent observer ratings of fidelity and participant engagement outcomes. These results demonstrate the potential for NLP methods to monitor implementation in evidence-based parenting programs delivered at scale.
RESUMEN
OBJECTIVES: To describe the process of developing and implementing experiential learning through translational research teams that engage diverse undergraduate and graduate students. METHODS: After a college redesign, translational research teams were developed to foster multidisciplinary research and better integrate students with faculty research, community, and clinical activities. Three primary approaches were used to engage undergraduate and graduate students in the maternal and child health translational research team (MCH TrT). These included an undergraduate experiential learning course; participation in translational research team meetings and events; and mentorship activities including graduate student theses and supplementary projects. RESULTS: Since 2019, a total of 56 students have engaged with the MCH translational research team. The majority (64%) of students engaging in translational research were undergraduates. Racial and ethnic diversity was evident with 16% Latinx, 14% Black/African American, 12% Asian, 10% two or more races, and 4% Native American or Native Hawaiian. A large proportion (42%) of students indicated that they were first-generation college students, while 24% indicated they had a disability. Five themes emerged from student feedback about their involvement in the experiential learning course: the value of translational research, development of research skills, collaboration, practice development, and value for community partners. CONCLUSIONS FOR PRACTICE: Through an MCH translational research team, we have established a pathway to enhance diversity among the MCH workforce which will increase recruitment and retention of underrepresented groups, and ultimately improve MCH research and practice.
