A Review of Public Health Training Needs Assessment Approaches: Opportunities to Move Forward.

CONTEXT: Assessing training needs of the public health workforce is crucial for creating professional development opportunities to improve knowledge, competence, and effectiveness of this workforce. DISSEMINATION: Regional Public Health Training Centers (RPHTCs) assess workforce training needs and deliver training based on identified needs. To determine training priorities, several needs assessment surveys have been administered by RPHTCs and national public health member organizations. EVALUATION: This study identified the types of training questions being asked to public health practitioners in the various assessment surveys implemented by RPHTCs and national membership organizations. Although the surveys measured similar overarching constructs, multiple approaches with limited consistency were used to measure training needs. DISCUSSION: Although successful in responding to the needs of their targeted constituents, the limited consistency among survey types makes generalization of findings difficult. Disseminating common metrics and aggregate survey findings would increase efficiency in determining workforce training needs and developing targeted training.

Creating quality improvement culture in public health agencies.

OBJECTIVES: We conducted case studies of 10 agencies that participated in early quality improvement efforts. METHODS: The agencies participated in a project conducted by the National Association of County and City Health Officials (2007-2008). Case study participants included health directors and quality improvement team leaders and members. We implemented multiple qualitative analysis processes, including cross-case analysis and logic modeling. We categorized agencies according to the extent to which they had developed a quality improvement culture. RESULTS: Agencies were conducting informal quality improvement projects (n = 4), conducting formal quality improvement projects (n = 3), or creating a quality improvement culture (n = 4). Agencies conducting formal quality improvement and creating a quality improvement culture had leadership support for quality improvement, participated in national quality improvement initiatives, had a greater number of staff trained in quality improvement and quality improvement teams that met regularly with decision-making authority. Agencies conducting informal quality improvement were likely to report that accreditation is the major driver for quality improvement work. Agencies creating a quality improvement culture were more likely to have a history of evidence-based decision-making and use quality improvement to address emerging issues. CONCLUSIONS: Our findings support previous research and add the roles of national public health accreditation and emerging issues as factors in agencies' ability to create and sustain a quality improvement culture.

Driving a public health culture of quality: how far down the highway have local health departments traveled?

CONTEXT: There has been an extensive investment in building public health organizational capacity to improve performance and prepare for accreditation. An evolving perspective has focused not only on the practice of quality improvement (QI) within the health department but also upon the extent the culture of QI is embraced within the agency. OBJECTIVE: No studies have examined the current national baseline of QI culture implementation, nor estimated the degree of QI sophistication local health departments (LHDs) have attained. We attempt to fill this void by aligning the findings from the QI module of the National Association of County & City Health Officials (NACCHO) 2010 Profile of LHDs against the constructs defined by the QI Maturity Tool and the NACCHO QI Roadmap (Roadmap to a Culture of Quality Improvement). DESIGN: Specific questions regarding QI activities from the 2010 Profile Study QI module were used to assign responding LHDs to stages within the Roadmap. We also used data from the QI Maturity Tool administered to all LHDs in the 16 participating Multi-State Learning Collaborative states in 2010 and 2011. On the basis of this matched set, we applied the summative domain scores algorithm, classified agencies into 1 of 5 groups, compared our findings with those of the NACCHO survey, and aligned our categories to those of the Roadmap. RESULTS: Nearly 80% of LHDs classified using the NACCHO Profile data were assigned to group 3 or 4 versus 48% using the QI Maturity Tool. Results from the cross-tabulations of the matched data set between the QI Maturity Tool classifications and the NACCHO Profile classifications revealed exact alignment 30% of the time. Forty-nine of 163 agencies were classified in the same grouping in both schemata. In addition, 84% of the agencies were classified within 1 neighboring category. CONCLUSIONS: The results revealed that half, if not most, LHDs fall within the middle categories of QI maturity and sophistication, regardless of which classification system was deployed.

Assessing quality improvement in local health departments: results from the Multi-State Learning Collaborative.

OBJECTIVE: This study examined changes in quality improvement: (1) over a 3-year period, (2) among mini-collaborative participants, and (3) among agencies that were classified in the lowest QI performance quartile. METHODS: : A QI Maturity Tool was administered to all local health departments in the Multi-State Learning Collaborative states. Factorial ANOVA was performed to determine differences in composite factor scores and interaction effects were explored. RESULTS: : The results revealed a significant increase in the percent of agencies that reported ever implementing a formal QI process from 2009 to 2011 and agencies that participated in a mini-collaborative were more likely than their non-mini-collaborative counterparts to report QI implementation. The findings also suggested significant changes in QI capacity and competency as well as alignment and spread over the course of the grant, among mini-collaborative participants, and among agencies that were originally identified in the lowest QI maturity quartile. No significant changes in organizational culture were revealed by year or among mini-collaborative participants. However, agencies in the lowest quartile saw significant changes in organizational culture during the project period. CONCLUSIONS: Significant changes in specific QI domains are possible to detect during a 3-year period. Our research gives us confidence that the QI Maturity Tool measures the right domains and is an important contribution to quantifying the adoption and spread of QI in public health. However, further refinement is needed to better standardize language and definitions of the component parts of a QI system.

Quality improvement learning collaboratives in public health: findings from a multisite case study.

OBJECTIVES: This study was designed to (1) test a theory on the key drivers of a successful mini-collaborative, (2) describe the application of quality improvement (QI) approaches and techniques among mini-collaborative participants, and (3) identify key attributes that affect the spread and sustainability of QI efforts within a local public health agency. METHODS: A multisite case study methodology was used to evaluate a subset of mini-collaboratives and their participating local health departments that were working in specific target areas. Data were collected during semistructured interviews and while observing mini-collaborative meetings. Documentation reviews were also conducted. We used standard techniques to code the data on the basis of themes and connections between themes. RESULTS: The findings provide early evidence that support our case study theory on the drivers of a successful mini-collaborative including advanced planning, the selection of faculty, timely training and technical assistance, the role of senior leaders, the application of evidence-based practices, the use of an improvement model, evaluation efforts, communication, the availability of resources, target selection, and prior experience with and application of QI. While the case studies provided limited evidence of sustainability and broad spread of QI within participating states, mini-collaboratives appear to have had a major impact and will likely influence the work of participating local health departments going forward. CONCLUSIONS: Our findings suggest that the mini-collaboratives served as a catalyst for engaging local health departments in the theory and practice of QI.

Effectiveness of public health quality improvement training approaches: application, application, application.

CONTEXT: Quality improvement (QI) has been identified as a key strategy to improve the performance of state and local public health agencies. Quality improvement training effectiveness has received little attention in the literature. OBJECTIVES: To evaluate the effectiveness of 3 QI training types: webinars, workshops, and demonstration site activities on improving participant knowledge, skill, and ability to conduct QI through a questionnaire conducted after training participation. DESIGN: We used a natural experimental design hypothesizing that demonstration site participants would have the greatest gains on outcomes of interest compared with webinar and workshop participants. Bivariate and multivariate models were used to examine outcome differences between questionnaire respondents who participated in various training types. PARTICIPANTS: Local health department employees who participated in the 3 training strategies. MAIN OUTCOME MEASURES: Measures included knowledge and skill gain, skill application, QI receptivity, and ability to successfully participate in a QI project. RESULTS: Two hundred eighty-four unique individuals who work in 143 health departments completed the questionnaire for a 59% response rate. The majority of these health departments serve midsize populations. Demonstration site respondents had significantly greater gains in knowledge and skills, skill application, and ability to successfully participate in a QI project. Webcast training participants had significantly higher QI receptivity in adjusted models. Respondents who participated in both webcast and demonstration site trainings had higher mean scores on all outcomes when compared with demonstration site single training participants, these differences were significant in unadjusted models. CONCLUSION: Our findings suggest that QI training for public health agency employees should include both didactic training on QI content and opportunities for QI application. Future research should examine if this approach can effectively increase successful participation in QI projects for staff in LHDs of all sizes.

Evaluating the Multi-State Learning Collaborative.

The third phase of the Multi-State Learning Collaborative (MLC) is a 3-year Robert Wood Johnson Foundation initiative designed to build quality improvement capacity within state and local health departments. This commentary provides a brief overview of the MLC, a methodological description of its evaluation, and preliminary findings. Major evaluation data sources include quarterly reports, surveys, case studies, and key informant interviews. Preliminary survey data and quarterly reports from year 1 provide some early evidence that states are progressing toward MLC goals. Approximately 27 percent of local health departments in the 16 participant states report that they have begun to prepare for national accreditation and approximately 39 percent report implementation of at least one quality improvement project within the past 12 months. Ongoing data collection efforts are underway to more fully address the evaluation questions.

Early Detection, Intervention and Prevention of Psychosis Program: Community Outreach and Early Identification at Six U.S. Sites.

OBJECTIVE: This study assessed the effects of a community outreach and education model implemented as part of the Early Detection, Intervention and Prevention of Psychosis Program (EDIPPP), a national multisite study in six U.S. regions. METHODS: EDIPPP's model was designed to generate rapid referrals of youths at clinical high risk of psychosis by creating a network of professionals and community members trained to identify signs of early psychosis. Qualitative and quantitative data were gathered through an evaluation of outreach efforts at five sites over a two-year period and through interviews with staff at all six sites. All outreach activities to groups (educational, medical, and mental health professionals; community groups; media; youth and parent groups; and multicultural communities) were counted for the six sites to determine correlations with total referrals and enrollments. RESULTS: During the study period (May 2007-May 2010), 848 formal presentations were made to 22,840 attendees and 145 informal presentations were made to 11,528 attendees at all six sites. These presentations led to 1,652 phone referrals. A total of 520 (31%) of these individuals were offered in-person orientation, and 392 (75%) of those were assessed for eligibility. A total of 337 individuals (86% of those assessed) met criteria for assignment to the EDIPPP study. CONCLUSIONS: EDIPPP's outreach and education model demonstrated the effectiveness of following a protocol-defined outreach strategy combined with flexibility to reach culturally diverse audiences or initially inaccessible systems. All EDIPPP sites yielded appropriate referrals of youths at risk of psychosis.

Writing community-centered evaluation reports.

Documenting the process, results, impact, and effectiveness of community-based health promotion programs is an important part of any evaluation. This article provides information on how to write community-centered evaluation reports for program stake-holders. Specific prerequisites and principles are provided. In addition, several tips for increasing the use of the results are highlighted.

Measuring quality improvement in public health: the development and psychometric testing of a QI Maturity Tool.

There is growing interest and investment in improving the quality of public health services and outcomes. Following the lead of other sectors, efforts are underway to introduce systematic quality improvement (QI) tools and approaches to state and local public health agencies. Little is known, however, about how to describe and reliably measure the level of QI maturity within a public health agency. The authors describe the development of a QI Maturity Tool using research from the fields of organizational design, psychology, health care, and complexity theory. The 37-item assessment tool is based on four quality domains derived from the literature: (a) organizational culture, (b) capacity and competency, (c) practice, and (d) alignment and spread. The tool was designed to identify features of an organization that may be enhancing or impeding QI; monitor the impact of efforts to create a more favorable environment for QI; and define potential cohorts of public health agencies for evaluation purposes. The article presents initial steps in testing and validating the QI Maturity Tool including: (a) developing a theoretical framework, (b) assuring face and content validity, (c) determining the tool's reliability based on estimates of internal consistency, (d) assessing the dimensionality, and (f) determining the construct validity of the instrument. The authors conclude that there is preliminary evidence that the QI Maturity Tool is a promising instrument. Further work is underway to explore whether self-reported survey results align with an agency's actions and the products of their QI efforts.

Linking accreditation and public health outcomes: a logic model approach.

Emerging public health standards, performance assessment tools, and accreditation models hold significant promise for defining and standardizing public health practice, yet the lack of empirical research on their relationship to outcomes represents a serious barrier to adoption. Given the growing interest and momentum related to public health agency assessment and accreditation efforts, there is increasing need for evidence that performance standards and associated accreditation programs are effective means for moving public health systems toward the ultimate goal of population and community health improvement. This article provides an overview of accreditation in health and other industries, and its relationship to outcomes. We examine lessons that might have meaningful public health translations, as well as influences in and on public health that pose challenges for research and evaluation in this area. Finally, we propose a logic model framework to help depict the ways in which we can begin to explore the impact accreditation has on various levels of outcomes. This logic model is intended to guide the development of measures and to serve as a tool to help convey the breadth and depth of research needed to link accreditation to health outcomes.

Lessons learned from the multistate learning collaborative.

Given the recent interest in public health accreditation programs and related efforts, there is a need to learn from the shared experiences of states that have developed, implemented, and evaluated their own efforts. The Multistate Learning Collaborative provided such an opportunity. Five states were selected to participate in this national peer group. The states represented in the Collaborative reflect different accreditation and assessment models, varying levels of maturity, and various designs based on the context and needs of a given state. However, despite these differences, common themes, critical elements, and shared challenges have emerged.

Evaluation as a critical factor of success in local public health accreditation programs.

This article presents the variety of approaches used to conduct evaluations of performance improvement or accreditation systems, while illustrating the complexity of conducting evaluations to inform local public health practice. We, in addition, hope to inform the Exploring Accreditation Program about relevant experiences involving accreditation and performance assessment processes, specifically evaluation, as it debates and discusses a national voluntary model. A background of each state is given. To further explore these issues, interviews were conducted with each state's evaluator to gain more in-depth information on the many different evaluation strategies and approaches used. On the basis of the interviews, the authors provide several overall themes, which suggest that evaluation is a critical tool and success factor for performance assessment or accreditation programs.

Use of national public health performance standards to assess Maine's diabetes system.

In the fall of 2003, Maine underwent a rigorous assessment of the diabetes public health system using a modified version of the State Public Health System Performance Instrument developed by the Centers for Disease Control and Prevention's National Public Health Performance Standards Program and other national partners. The assessment was intended to serve as the impetus for the development of a statewide improvement plan. This article details the assessment process and provides a case study highlighting the successful application of systems-based model standards for a categorical issue.

Train-the-trainer as an educational model in public health preparedness.

Train-the-trainer (TTT) is a widely acknowledged educational model across a number of disciplines, including public health preparedness. However, many questions arise about its proper use, its efficacy, and its optimal role in preparedness education. After careful consideration of the literature and past anecdotal experiences with this educational model, TTT was selected as the preferred and sustainable method for public health preparedness in Maine. Upon completion of Year 1 of the Maine TTT program, our evaluation results revealed that TTT was successful and well received by participants. It is evident that TTT is potentially a valuable tool in public health preparedness, but it should be noted that there is no clear prescription for implementing TTT. Institutions and communities adopting TTT as an educational model must be flexible and willing to revise the training plan as challenges are encountered. Future research is required to help clarify optimal ways of using TTT to promote public health preparedness in the 21st century.