Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Pilot Study of the MAP Curriculum for Psychotherapy Competencies in Child and Adolescent Psychiatry.

OBJECTIVE: This pilot study aimed to evaluate the relevance, feasibility, acceptability, and instructional efficacy of the Managing and Adapting Practice (MAP) curriculum for enhancing the teaching of psychotherapy to child and adolescent psychiatry (CAP) fellows. MAP is a system of resources and decision models that supports practitioners in selecting and implementing psychotherapeutic interventions for children and adolescents. The MAP curriculum includes modules to guide education about psychotherapeutic procedures (e.g., behavioral activation) common in evidence-based treatments for an array of childhood problems and to support development of competencies in assessment, treatment planning, and reflective practice. METHODS: Curriculum coding was used to examine the relevance of MAP's core components to the skills articulated in the Accreditation Council for Graduate Medical Education (ACGME) CAP milestones. Feasibility, acceptability, and learning outcomes were examined after delivery of the MAP curriculum to 12 CAP fellows at two sites, with instructional features tailored according to faculty preferences and training program structure. RESULTS: Coding suggested that the MAP curriculum was relevant to 95% of the 21 ACGME CAP training subcompetencies. Feasibility was indicated by the successful delivery of 100% of the planned MAP curriculum across the two sites. Acceptability was supported by positive feedback from the CAP fellows, and psychotherapy knowledge increased significantly. Finally, case review scores (mean±SD=2.21±0.15) showed positive posttraining application of MAP to two patients and exceeded scores achieved by other samples of mental health professionals. CONCLUSIONS: This pilot study demonstrated the potential for the MAP curriculum to support CAP education. MAP's versatility as a curriculum supports broader adoption, with continuing rigorous empirical evaluation.

Pediatric Consultation-Liaison: Patient Characteristics and Considerations for Training in Evidence-Based Practices.

Consultation-liaison services are an integral part of many pediatric hospital settings, yet characteristics of this patient population have not been extensively documented. The current study is a retrospective one-year chart review of the consultation-liaison service at a large pediatric hospital in the Southwestern United States. The purpose of this study is twofold: (1) to characterize this hospital's CL population and (2) to use these characteristics to identify preliminary evidence-based practices that should be considered for CL provider training. Identifying evidence-based practice elements that align with the characteristics of consultation-liaison patient populations may inform trainings for consultation-liaison staff. This would help to ensure that youth seen in hospital consultation-liaison services are getting the best available services, which is critical given the shortened time frame available to work with this patient population.

Do Treatment Plans Matter? Moving From Recommendations to Action.

We investigated whether a service-planning document outlining recommendations for what providers should address in treatment (i.e., targets) and the associated clinical techniques they should employ (i.e., practices) influenced the targets and practices that providers reported actually implementing during the subsequent treatment episode. Participants included 94 youths ages 4 to 17 (M = 13.57, SD = 3.59) who received community-based mental health services from the Hawai'i Child and Adolescent Mental Health Division. Data on targets and practices were compared across initial Mental Health Treatment Plans and Monthly Treatment and Progress Summaries. Data were analyzed using two-level, generalized mixed effects models with two-way cross-classification or linear mixed effects models. Providers were more likely to report the use of targets and practices in treatment if they were included within the treatment plan. In addition, the more closely targets addressed during treatment followed the recommended targets from the treatment plan, the more closely implemented practices followed the recommended practices listed in the treatment plan. Furthermore, as providers shifted their focus to different targets, a shift in their use of practices was also evident over time. Last, practices for which there is demonstrated efficacy for particular targets were more likely to be used. Service planning documents appear to help organize care; however, results also suggest possible limitations to the current system. These findings highlight potential areas for improvement in planning and care delivery.

Managing and Adapting Practice: Provider Perceptions of an Evidence-Informed Framework for Delivering Mental Health Services.

This study examined providers' reflections on delivering managing and adapting practice (MAP), an evidence-informed framework that guides decision-making from scientific and client data. Consensual qualitative research methods were used to analyze the reflections of 201 youth mental health providers. Results indicated that providers approached MAP according to their own preferences and particular cases. While most appeared to approach MAP from a practice management standpoint, when faced with challenging cases, providers used coordination and outcomes management resources. Regardless of approach, most providers came to appreciate the full framework through reflective practice. Their diverse approaches offer lessons for evidence-based practice implementation and sustainment.

Implementing Evidence Based Practices for Children's Mental Health: A Case Study in Implementing Modular Treatments in Community Mental Health.

There is strong enthusiasm for utilizing implementation science in the implementation of evidence-based programs in children's community mental health, but there remains work to be done to improve the process. Despite the proliferation of implementation frameworks, there is limited literature providing case examples of overcoming implementation barriers. This article examines whether the use of three implementations strategies, a structured training and coaching program, the use of professional development portfolios for coaching, and a progress monitoring data system, help to overcome barriers to implementation by facilitating four implementation drivers at a community mental health agency. Results suggest that implementation is a process of recognizing and adapting to both predictable and unpredictable barriers. Furthermore, the use of these implementation strategies is important in improving implementation outcomes.

Replicating Evidence-Based Practices with Flexibility for Perinatal Home Visiting by Paraprofessionals.

Introduction Strategies are needed to improve the efficacy of paraprofessional home visitors for pregnant women in the United States. This study evaluates the maternal and child outcomes when evidence-based practices (EBP) are replicated with flexibility, rather than fidelity to a manualized intervention. Methods Pregnant mothers (N = 203) in five clinics were recruited in the waiting rooms and randomized to standard clinic care as the control condition (n = 104) or standard care plus home visiting (n = 99). Home visitors (n = 9) were selected, trained in foundational skills common to EBP and four problem domains (weight control, breastfeeding, daily habits, and depression). Independent interviewers assessed targeted outcomes at birth (82%) and 6 months later (83%). RESULTS: Home visitors, called Mentor Mothers [MM], made an average of 14.9 home visits or telephone contacts (SD = 9; total contacts = 1491) addressing maternal daily habits, breastfeeding, and depression. Intervention and control mothers were similar in weight, Body Mass Index (BMI), depression and social support at baseline and 6 months later. The percentage of low birth weight babies was similar; intervention infants' growth (weight/height Z score) tended to be significantly better compared to the control condition. DISCUSSION: There are many explanations for the failure to find significant benefits: insufficient statistical power; the benefits of repeated assessments by warm, supportive peers to improve outcomes; or the failure of EBP and the need to maintain replication with fidelity. All study mothers had better outcomes than documented among comparable published samples of low-income, Latina and Korean-American mothers in Los Angeles, CA. ClinicalTrials.gov registration NCT01687634.

Adapting Current Strategies to Implement Evidence-Based Prevention Programs for Paraprofessional Home Visiting.

This paper describes a strategy for using evidence-based interventions (EBI) that does not require replication and fidelity. Eight parents, identified as positive role models, conducted home visits for 101 low-income Latina and Korean pregnant women. The home visitors, called mentor mothers (MM), were trained in 10 of the practice elements common to 80% of child-focused EBI and how to apply these skills to support mothers in obesity prevention, to increase the duration of breastfeeding, and to reduce depression. MM reported the content and skills utilized on each home visit on mobile phones. Each MM made an average of 153 home visits (SD = 173.3), with 28 of these visits being phone contacts. Body mass index (BMI) at 6 months was significantly associated with the frequency of MM focused on coping with depression (r = .24), but was not related to practice elements used by MM. The duration of breastfeeding was significantly related to the frequency with which MM focused on the topic of breastfeeding (r = .28) and parenting (r = .3), and MM use of attending (r = .24) and relaxation (r = .27). Depression was significantly correlated with the frequency of addressing depression (r = .27), but not to practice elements. MM did use different strategies in the first 150 visits compared to their last 150 visits, reflecting data-informed supervision. Evidence synthesized from EBI was used as a novel training method, with real-time monitoring and data-informed supervision providing evidence of iterative quality improvements in MM behaviors over time, as well as a way for linking implementation processes to outcomes.

At the Intersection of Health Information Technology and Decision Support: Measurement Feedback Systems... and Beyond.

We select and comment on concepts and examples from the target articles in this special issue on measurement feedback systems, placing them in the context of some of our own insights and ideas about measurement feedback systems, and where those systems lie at the intersection of technology and decision making. We contend that, connected to the many implementation challenges relevant to many new technologies, there are fundamental design challenges that await a more elaborate specification of the clinical information and decision models that underlie these systems. Candidate features of such models are discussed, which include referencing multiple evidence bases, facilitating observed and expected value comparisons, fostering collaboration, and allowing translation across multiple ontological systems. We call for a new metaphor for these technologies that goes beyond measurement feedback and encourages a deeper consideration of the increasingly complex clinical decision models needed to manage the uncertainty of delivering clinical care.

The common elements of engagement in children's mental health services: which elements for which outcomes?

Using the distillation component of the Distillation and Matching Model framework (Chorpita, Daleiden, & Weisz, 2005 ), we examined which engagement practices were associated with three domains of treatment engagement: attendance, adherence, and cognitive preparation (e.g., understanding of, readiness for treatment). Eighty-nine engagement interventions from 40 randomized controlled trials in children' s mental health services were coded according to their engagement practices and outcomes. Analyses examined whether the practices used in successful interventions differed according to engagement domain. Practice patterns differed somewhat depending on whether attendance, adherence, or cognitive preparation was the outcome of interest. For example, assessment of barriers to treatment frequently occurred in successful interventions targeting attendance, whereas homework assignment frequently occurred in successful interventions when adherence was the target outcome. Modeling and expectation setting were frequently used in successful interventions targeting cognitive preparation for treatment. Distillation provides a method for examining the practice patterns associated with different engagement outcomes. An example of the application of these findings to clinical practice includes using certain practices (e.g., assessment, psychoeducation about services, and accessibility promotion) with all youth and families to promote attendance, adherence, and cognitive preparation. Then, other practices (e.g., modeling, homework assignment) can be added on an as-needed basis to boost engagement or to address interference in a particular engagement domain. The use of a distillation framework promotes a common language around engagement and highlights practices that lend themselves well to training, thereby promoting the dissemination of engagement interventions.

Fit of evidence-based treatment components to youths served by wraparound process: a relevance mapping analysis.

This study investigated whether and which evidence-based treatment (EBT) components might generalize to youths served by the wraparound process. To examine these questions, the study used relevance mapping, an empirical methodology that compares youths in a given clinical population with participants in published randomized trials to determine who may be "coverable" by EBTs and which treatments may collectively be most applicable. In a large diverse clinical sample, youths receiving wraparound services (n = 828) were compared with youths receiving other services (n = 3,104) regarding (a) demographic and clinical profiles, (b) "coverability" by any EBTs, and (c) specific practices from those EBTs that most efficiently applied to each group. Participants in studies of EBTs matched the demographic and clinical characteristics of nearly as many youths receiving wraparound (58-59%) as those receiving non-wraparound services (61-64%). Moreover, the best-fitting solutions of relevant sets of practices were highly similar across groups. These results provide the first large-scale empirical characterization of fit between EBTs and youths receiving wraparound and suggest that these youths are well suited to benefit from clinical strategies commonly used in EBTs.

Doing more with what we know: introduction to the special issue.

This article introduces a special issue that provides an overarching conceptual model for advancing the impact of science on service delivery and includes some illustrations of these concepts in action from a variety of research teams in a diversity of contexts. Themes of the special issue include balancing considerations of efficacy, effectiveness, and extensiveness--for example, ensuring that treatments are not only reliable but also robust and widely relevant. It is argued that such a balance might be achieved through an increased focus on improving coordination of system resources (e.g., clients, providers, treatments, training programs) and through dynamic and developmental designs. The special issue is intended to raise the broader question of whether our current paradigms are satisfactory in moving us toward our shared vision, or whether we could do more with what we already know.

Structuring the collaboration of science and service in pursuit of a shared vision.

The enduring needs of our society highlight the importance of a shared vision to improve human functioning and yield better lives for families and communities. Science offers a powerful strategy for managing the inevitable uncertainty in pursuit of these goals. This article presents ideas and examples of methods that could preserve the strengths of the two major paradigms in children's mental health, evidence-based treatments and individualized care models, but that also have the potential to extend their applicability and impact. As exemplified in some of the articles throughout this issue, new models to connect science and service will likely emerge from novel consideration of better ways to structure and inform collaboration within mental health systems. We contend that the future models for effective systems will involve increased attention to (a) client and provider developmental pathways, (b) explicit frameworks for coordinating people and the knowledge and other resources they use, and (c) a balance of evidence-based planning and informed adaptation. We encourage the diverse community of scientists, providers, and administrators in our field to come together to enhance our collective wisdom through consideration of and reflection on these concepts and their illustrations.

Family voice with informed choice: coordinating wraparound with research-based treatment for children and adolescents.

The wraparound process is a type of individualized, team-based care coordination that has become central to many state and system efforts to reform children's mental health service delivery for youths with the most complex needs and their families. Although the emerging wraparound research base is generally positive regarding placements and costs, effect sizes are smaller for clinical and functional outcomes. This article presents a review of literature on care coordination and wraparound models, with a focus on theory and research that indicates the need to better connect wraparound-enrolled children and adolescents to evidence-based treatment (EBT). The article goes on to describe how recently developed applications of EBT that are based on quality improvement and flexible application of "common elements" of research-based care may provide a more individualized approach that better aligns with the philosophy and procedures of the wraparound process. Finally, this article presents preliminary studies that show the feasibility and potential effectiveness of coordinating wraparound with the Managing and Adapting Practice system, and discusses intervention development and research options that are currently under way.

MAPping Los Angeles County: taking an evidence-informed model of mental health care to scale.

We describe the scaling up of an evidence-informed model of care, Managing and Adapting Practice (MAP) in Los Angeles County, California. MAP complemented an array of evidence-based programs selected by the county as part of a large system reform effort designed to improve care for children and adolescents. In addition, we discuss the MAP model for training therapists and present data both on how the training model performed and on the outcomes of youths treated by therapists trained in MAP. We examined the success of two different training pathways for MAP therapists: (a) national training model and (b) MAP agency supervisor model (i.e., train the trainer). We also examined utilization of MAP and outcomes of clients served by MAP. Both the national training and MAP agency supervisor model were successful in producing MAP therapists in a timely fashion and with acceptable competency scores. Furthermore, a large number of clients were receiving MAP services. Finally, outcomes for youth treated with MAP were strong, with effect sizes ranging from .59 to .80 on the Youth Outcome Questionnaire. These data support the notion that scaling up a mental health services approach in a system can be achieved through a strong and broad partnership among relevant stakeholders, can involve a train-the-trainer model, and can result in strong outcomes for clients.

Tracking evidence based practice with youth: validity of the MATCH and standard manual consultation records.

This study sought to evaluate the agreement between therapist report and coder observation of therapy practices. The study sampled session data from a community-based, randomized trial of treatment for youth ages 7 to 13. We used therapist report of session content and coverage gathered using formal Consultation Records and developed complimentary records for coders to use when watching or listening to therapy tape. We established initial reliability between coders and then conducted a random, stratified, and comprehensive sample of sessions across youth (N = 121), therapists (N = 57), conditions (MATCH and Standard Manuals), and study sites (Honolulu and Boston) to code and compare with therapist record reports. Intraclass correlation coefficients (ICCs) representing coder versus therapist agreement on manual content delivered ranged from .42 to 1.0 across conditions and problem areas. Analyses revealed marked variability in agreement regarding whether behavioral rehearsals took place (ICCs from -.01 to 1.0) but strong agreement on client comprehension of therapy content and homework assignments. Overall, the findings indicate that therapists can be accurate reporters of the therapeutic practices they deliver, although they may need more support in reporting subtle but valuable aspects of implementation such as types of behavioral rehearsals. Developing means to support accurate reporting is important to developing future clinical feedback methodology applicable to the implementation of evidence-based treatments in the real world.

The role of implementation organizations in scaling evidence-based psychosocial interventions.

BACKGROUND: To bring evidence-based interventions (EBIs) to individuals with behavioral health needs, psychosocial interventions must be delivered at scale. Despite an increasing effort to implement effective treatments in communities, most individuals with mental health and behavioral problems do not receive EBIs. We posit that organizations that commercialize EBIs play an important role in disseminating EBIs, particularly in the USA. The behavioral health and implementation industry is growing, bringing the implementation field to an important inflection point: how to scale interventions to improve access while maintaining EBI effectiveness and minimizing inequities in access to psychosocial intervention. MAIN BODY: We offer a first-hand examination of five illustrative organizations specializing in EBI implementation: Beck Institute for Cognitive Behavioral Therapy; Incredible Years, Inc.; the PAXIS Institute; PracticeWise, LLC; and Triple P International. We use the Five Stages of Small Business Growth framework to organize themes. We discuss practical structures (e.g., corporate structures, intellectual property agreements, and business models) and considerations that arise when trying to scale EBIs including balancing fidelity and reach of the intervention. Business models consider who will pay for EBI implementation and allow organizations to scale EBIs. CONCLUSION: We propose research questions to guide scaling: understanding the level of fidelity needed to maintain efficacy, optimizing training outcomes, and researching business models to enable organizations to scale EBIs.

Assessing practical implementation of modular psychotherapy for youth in community-based settings using benchmarking.

Background: Achieving high quality outcomes in a community context requires the strategic coordination of many activities in a service system, involving families, clinicians, supervisors, and administrators. In modern implementation trials, the therapy itself is guided by a treatment manual; however, structured supports for other parts of the service system may remain less well-articulated (e.g., supervision, administrative policies for planning and review, information/feedback flow, resource availability). This implementation trial investigated how a psychosocial intervention performed when those non-therapy supports were not structured by a research team, but were instead provided as part of a scalable industrial implementation, testing whether outcomes achieved would meet benchmarks from published research trials. Method: In this single-arm observational benchmarking study, a total of 59 community clinicians were trained in the Modular Approach to Therapy for Children (MATCH) treatment program. These clinicians delivered MATCH treatment to 166 youth ages 6 to 17 naturally presenting for psychotherapy services. Clinicians received substantially fewer supports from the treatment developers or research team than in the original MATCH trials and instead relied on explicit process management tools to facilitate implementation. Prior RCTs of MATCH were used to benchmark the results of the current initiative. Client improvement was assessed using the Top Problems Assessment and Brief Problem Monitor. Results: Analysis of client symptom change indicated that youth experienced improvement equal to or better than the experimental condition in published research trials. Similarly, caregiver-reported outcomes were generally comparable to those in published trials. Conclusions: Although results must be interpreted cautiously, they support the feasibility of using process management tools to facilitate the successful implementation of MATCH outside the context of a formal research or funded implementation trial. Further, these results illustrate the value of benchmarking as a method to evaluation industrial implementation efforts.Plain Language Summary: Randomized effectiveness trials are inclusive of clinicians and cases that are routinely encountered in community-based settings, while continuing to rely on the research team for both clinical and administrative guidance. As a result, the field still struggles to understand what might be needed to support sustainable implementation and how interventions will perform when brought to scale in community settings without those clinical trial supports. Alternative approaches are needed to delineate and provide the clinical and operational support needed for implementation and to efficiently evaluate how evidence-based treatments perform. Benchmarking findings in the community against findings of more rigorous clinical trials is one such approach. This paper offers two main contributions to the literature. First, it provides an example of how benchmarking is used to evaluate how the Modular Approach to Therapy for Children (MATCH) treatment program performed outside the context of a research trial. Second, this study demonstrates that MATCH produced comparable symptom improvements to those seen in the original research trials and describes the implementation strategies associated with this success. In particular, although clinicians in this study had less rigorous expert clinical supervision as compared with the original trials, clinicians were provided with process management tools to support implementation. This study highlights the importance of evaluating the performance of intervention programs when brought to scale in community-based settings. This study also provides support for the use of process management tools to assist providers in effective implementation.

Implementing evidence-based practices for youth in an HMO: the roles of external ratings and market share.

A qualitative study of child clinicians in a non-profit HMO examined implementation of evidence-based practices (EBPs) for anxiety and oppositional defiant disorders using interviews and focus groups with 33 clinicians (97% of participants), and ethnography of emails and meetings. Analysis showed statistical measures of access and service-key elements of rating organizations' "report cards"- were central in creating "pressure" making transition to EBPs difficult. EBPs were secondary to access and service targets. "Research" and "statistics" were perceived as unrealistic, "literature" as lacking authority. Rating organizations should include outcome and fidelity metrics to align market share pressures with children's health.

Improvement in symptoms versus functioning: how do our best treatments measure up?

We examined the effects of redefining standards of evidence for treatments targeting childhood mental health problems by expanding outcomes beyond symptom reduction to include functioning. Over 750 treatment protocols from 435 randomized controlled trials were rated based on empirical evidence. Nearly two-thirds (63.9%) demonstrated at least a minimum level of evidence for reducing symptoms; however, only 18.8% of treatments demonstrated evidence for reducing functional impairment. Of those treatments with empirical support for symptom reduction, the majority did not demonstrate empirical support for improvement in functioning because measures of functioning were not included in the studies in which these treatments were tested. However, even when measures of functioning were included, it was much more difficult for treatments to achieve improvement. Among treatments that achieved improvement in functioning, the most notable were Collaborative Problem Solving for disruptive behavior and Cognitive Behavioral Therapy plus Medication for traumatic stress because they demonstrated no support for symptom reduction but good support for improvement in functioning. Results are discussed within the context of evaluating the standards of evidence for treatments and the opportunity to move towards a multidimensional framework whose utility has the potential to exceed the sum of its parts.