A Skills Assessment Pathways-Based Program Assessment Approach in Multidisciplinary Graduate Health Informatics.

Multidisciplinary graduate education programs are hard to assess because of interdependent competencies. Students in these programs come with diverse disciplinary undergraduate degrees, and it is critical to identify knowledge gaps among these diverse learner groups to provide support to fill these gaps. Health Informatics (HI) is a multidisciplinary field in which health, technology, and social science knowledge are foundational to building HI competencies. In 2017, the American Medical Informatics Association identified ten functional domains in which HI competencies are divided. Using pre/post-semester knowledge assessment surveys of graduate students (n=60) between August 2021 to May 2022 in one of the largest graduate HI programs in the United States, we identified courses (n=9) across the curriculum that help build HI-specific competencies. Using statistical analysis, we identified three skills pathways by correlating knowledge gained with course learning objectives and used this to modify the curriculum over four semesters. These skills pathways are connected through one or two courses, where students can choose electives or, in some instances, course modules or assignments that link the skills pathways. Moreover, there is a statistically significant difference in how students gain these skills depending on their prior training, even though they take the same set of courses. Gender and other demographics did not show statistical differences in skills gained. Additionally, we found that research assistantships and internships/practicums provide additional skills not covered in our HI curriculum. Our program assessment methodology and resulting curricular changes might be relevant to HI and other multidisciplinary graduate training programs.

Assessing the quality of care and service trajectories in autism from families' perspective: Early intervention and interim services.

BACKGROUND: Following the Evaluation of the Autism Trajectory for Parents - Diagnostic Services (ETAP-1), the ETAP-2 instrument was created to assess the quality of the post-diagnostic phase of the care and service trajectory of families of children with autism. The instrument, based on an integrated care perspective, was developed with the input of multiple stakeholders (parents, service providers, researchers). AIMS: This study sought to evaluate the factor structure, reliability, and convergent and discriminant validity of ETAP-2. METHODS AND PROCEDURES: Parents (N = 197) of children recently diagnosed with autism (M = 5.1 years) were recruited from an assessment center and organizations providing early behavioral intervention and other supports for autism in the province of Québec, Canada. They completed the ETAP-2 questionnaire along with measures of satisfaction and family quality of life. OUTCOMES AND RESULTS: The instrument presented a five-construct structure generally consistent with previously identified dimensions of quality, except for three items previously associated with the continuity of the service trajectory. ETAP-2 had excellent internal consistency and demonstrated convergent and discriminant validity with other measures. CONCLUSIONS AND IMPLICATIONS: ETAP-2 is a brief parent-report measure with good psychometric properties. It can assist in gathering information on families' perception and experiences with early intervention and other post-diagnostic, interim services.

Implementation and assessment of an end-to-end Open Science & Data Collaborations program.

As research becomes more interdisciplinary, fast-paced, data-intensive, and collaborative, there is an increasing need to share data and other research products in accordance with Open Science principles. In response to this need, we created an Open Science & Data Collaborations (OSDC) program at the Carnegie Mellon University Libraries that provides Open Science tools, training, collaboration opportunities, and community-building events to support Open Research and Open Science adoption. This program presents a unique end-to-end model for Open Science programs because it extends open science support beyond open repositories and open access publishing to the entire research lifecycle. We developed a logic model and a preliminary assessment metrics framework to evaluate the impact of the program activities based on existing data collected through event and workshop registrations and platform usage. The combination of these evaluation instruments has provided initial insight into our service productivity and impact. It will further help to answer more in-depth questions regarding the program impact, launch targeted surveys, and identify priority service areas and interesting Open Science projects.

The associations between occupational health and safety management system programming level and prior injury and illness rates in the U.S. dairy industry.

U.S. dairy workers suffer occupational injuries and illnesses at rates higher than the national average. Occupational health and safety management systems (OHSMS) have been proposed as a way to reduce injuries and illnesses for businesses of all types and sizes. The Occupational Safety and Health Administration (OSHA) On-Site Consultation Service provides assistance establishing an OHSMS to U.S. businesses. As part of this service, the consultants determine the level of OHSMS programming using the Safety and Health Program Assessment Worksheet (Revised OSHA Form 33). A total of 167 dairy industry records were obtained from OSHA. Forty-five of those records had both injury rate and OHSMS data. Using these records, a Spearman Rank-Order correlation was used to determine the strength and significance of the associations between prior injury rates and OSHA measured OHSMS programming level for dairy operations. Additional analyses were conducted to examine potential relationships between workforce size, injury rates, and OHSMS programming levels. There was a negative correlation between OHSMS programming level and injury rates, both for the overall OHSMS and by OHSMS component. Management Leadership was the OHSMS component most strongly associated with lower injury and illness rates. OHSMS interventions, as part of a comprehensive risk management approach for the U.S. dairy industry, may be warranted to help reduce the unacceptable number of injury and illnesses in the U.S. dairy industry. Further research is needed to determine if similar relationships between OHSMS programming and injury rates occur in other industries.

The impact of social-emotional learning: A meta-analysis in China.

The cross-cultural adaptation of social-emotional learning (SEL) has cast doubts. Although there are significant differences between low- and high-context cultures, few analyses have been conducted on the effects of SEL intervention in high-context cultures. To explore the effectiveness of the SEL program in China, which is different from low-context cultural background, this study presents findings from a meta-analysis of 86 randomized SEL programs involving 8,736 students. Compared with the control group, SEL participants significantly improved social-emotional competence (SEC), including SEL skills, attitudes, positive social behavior, and emotional distress (reduction). However, there was no significant improvement in behavioral problems. Due to the lack of emotional education in China and the Hawthorne effect, compared with SEL programs in low-context countries, China's SEL programs have improved SEC more, up to three times that of low-context countries. The general area of the school, SEL framework, intervention object, and educational level of participants moderates SEL positive outcomes. Types of textbooks, SEL framework, participant features, and educational level of participants moderate SEL negative outcomes. These findings provide empirical evidence for the positive impact of SEL programs in China. To improve the SEC of Chinese students, policymakers should actively implement SEL programs in China.

Independent Agencies and Research Program Performance.

Combining performance data from the Bush Administration's Program Assessment Rating Tool (PART) initiative with measures of organizational independence, I examine whether insulated and plural leadership structures are consequential for the outcomes of the federal programs administered by them. Using regression modeling and controlling for program type, I find that embedding programs in independent agencies is positively and significantly related to ratings of program performance. The effects of independent commissions appear mediated in these models by their positive association with the PART scores given to certain program types, notably research programs. These results are problematic for any global attribution of greater effectiveness to executive agencies under single-headed control and closer presidential direction.

Assessing the effectiveness of performance standards for Salmonella contamination of chicken parts.

The United States Department of Agriculture's Food Safety and Inspection Service implemented Salmonella performance standards for establishments producing chicken parts in 2016. The standards were chosen based on the assumption that a 30 % reduction in the occurrence of Salmonella-contaminated chicken parts samples (i.e., legs, breasts or wings) would result following implementation of the performance standard program. The derivation of the performance standards was based on data collected prior to the implementation of the standards and in the intervening years, so overall changes in the Salmonella contamination of this product can be assessed. This study presents a historical review of changes in Salmonella contamination on chicken parts as these changes relate to the performance standard. The analysis demonstrates that the reduction in Salmonella contaminated chicken parts samples was more than 75 %, so the FSIS risk assessment significantly underestimated the actual reduction in Salmonella contamination. An analysis of chicken parts samples collected at retail demonstrates reductions of a similar magnitude. Changes in the characteristics of Salmonella contamination that are potentially relevant to the occurrence or severity of human illness, such as seasonal changes in contamination, the composition of serotypes and changes in antimicrobial resistance, are also assessed. Small but significant seasonal increases in contamination were observed, with the peaks occurring in late winter rather than the more traditional late summer peak. Rapid changes in both the five most common serotypes and antimicrobial resistance patterns were also observed.

Design and comparison of a hybrid to a traditional in-person point-of-care ultrasound course.

BACKGROUND: Traditional introductory point-of-care ultrasound (POCUS) courses are resource intensive, typically requiring 2-3 days at a remote site, consisting of lectures and hands-on components. Social distancing requirements resulting from the COVID-19 pandemic led us to create a novel hybrid course curriculum consisting of virtual and in-person components. METHODS: Faculty, chief residents, fellows and advanced practice providers (APPs) in the Department of Medicine were invited to participate in the hybrid curriculum. The course structure included 4 modules of recorded lectures, quizzes, online image interpretation sessions, online case discussions, and hands-on sessions at the bedside of course participant's patients. The components of the course were delivered over approximately 8 months. Those participants who completed a minimum of 3 modules over the year were invited for final assessments. Results from the hybrid curriculum cohort were compared to the year-end data from a prior traditional in-person cohort. RESULTS: Participant knowledge scores were not different between traditional (n = 19) and hybrid (n = 24) groups (81% and 84%, respectively, P = 0.9). There was no change in POCUS skills as measured by the hands-on test from both groups at end-of-course (76% and 76%, respectively, P = 0.93). Confidence ratings were similar across groups from 2.73 traditional to 3.0 hybrid (out of possible 4, P = 0.46). Participants rated the course highly, with an average overall rating of 4.6 out 5. CONCLUSIONS: A hybrid virtual and in-person POCUS course was highly rated and as successful as a traditional course in improving learner knowledge, hands-on skill and confidence at 8 months after course initiation. These results support expanding virtual elements of POCUS educational curricula.

A network analysis of dissemination and implementation research expertise across a university: Central actors and expertise clusters.

Background: Although dissemination and implementation (D&I) science is a growing field, many health researchers with relevant D&I expertise do not self-identify as D&I researchers. The goal of this work was to analyze the distribution, clustering, and recognition of D&I expertise in an academic institution. Methods: A snowball survey was administered to investigators at University of Rochester with experience and/or interest in D&I research. The respondents were asked to identify their level of D&I expertise and to nominate others who were experienced and/or active in D&I research. We used social network analysis to examine nomination networks. Results: Sixty-eight participants provided information about their D&I expertise. Thirty-eight percent of the survey respondents self-identified as D&I researchers, 24% as conducting D&I under different labels, and 38% were familiar with D&I concepts. D&I researchers were, on average, the most central actors in the network (nominated most by other survey participants) and had the highest within-group density, indicating wide recognition by colleagues and among themselves. Researchers who applied D&I under different labels had the highest within-group reciprocity (25%), and the highest between-group reciprocity (29%) with researchers familiar with D&I. Participants significantly tended to nominate peers within their departments and within their expertise categories. Conclusions: Identifying and engaging unrecognized clusters of expertise related to D&I research may provide opportunities for mutual learning and dialog and will be critical to bridging across departmental and topic area silos and building capacity for D&I in academic settings.

Improving Representation of Underrepresented Minority (URM) Students in Oncology Biomedical Research Workforce: Outcome Evaluation from the ReTOOL Program.

The Research Training Opportunities for Outstanding Leaders (ReTOOL) program was implemented in 2012 to increase the representation of racial and ethnic minorities in the biomedical workforce. Specifically, the ReTOOL program aims to foster the capacity for scientific research among underserved populations as well as address the cultural appropriateness of research projects. This paper describes the impact of the ReTOOL program in enhancing the research training of underrepresented minority (URM) students. Forty URM students who completed the ReTOOL program between 2012 and 2019 were invited to participate in the program evaluation. The response rate was 73% with 29 participants. Of the 29 participants, 26 trainees self-identified as Black or African-American. A structured survey developed for the program was employed for data collection, using a Likert Scale ranging from 1 to 5, with 5 being the best. The item ratings ranged from 4.45 to 4.80. Responses to open-ended questions show that ReTOOL has been instrumental in socializing and acculturating participants into the habits of scientific thinking. The combined use of quantitative and qualitative inquiry depicts that ReTOOL has been highly successful in fostering participant enrollment in advanced health-related or professional degree programs.

Sustainable Program Assessment Practices: A Review of the ABET and NCAAA Computer Information Systems Accreditation Process.

Program outcome assessment is a complex process that demands careful planning and resources in order to accurately assess higher-order thinking skills. A well-defined assessment approach provides detailed insights into program weaknesses and leads to continuous improvement. Whereas a poor assessment approach does not reflect the underlying weaknesses and may result in a useless effort. Furthermore, each accreditation body may have a different recommended outcome measurement approach. As a result, academic institutions may make adhoc choices just to satisfy accreditation requirements rather than designing a sustainable measurement approach. On the other hand, the magnitude of huge tasks for satisfying multiple accreditation bodies results in fatigue and mental stress for academic staff. ABET is a well-known international program accreditation body, and NCAAA is a local accreditation body for academic programs in the Kingdom of Saudi Arabia. In this paper, we have documented that how a sustainable outcome measurement mechanism can be designed to satisfy both ABET and NCAAA requirements. The core contribution of this paper is relevant specifically for academic programs in the Kingdom striving to meet both ABET and NCAAA requirements and is also relevant for all education programs to design an appropriate program assessment approach to ensure a sustainable process to foster better learning among students.

Social and Emotional Learning in the Ibero-American Context: A Systematic Review.

Social and emotional learning (SEL) has acquired great prominence in recent years, due to the skills it develops in students, influencing personal and social well-being. At the same time, society is moving toward a model in which understanding oneself and others is a fundamental aspect in order to function properly on a social level. Studies on SEL programmes have been carried out in various parts of the world, although recent reviews have focused exclusively on the Anglo-Saxon context. Therefore, the aim of this paper was to synthesize research on the efficacy and effectiveness of SEL programmes in Ibero-American contexts in early childhood, primary and secondary education. Systematic review was used as the method of enquiry, following the standards of The Campbell Collaboration. In total, 22 empirical studies of SEL programmes implemented in Ibero-America were collected. The results showed that the SEL variables with the highest incidence and significant results were self-awareness, social awareness, self-control, relationship skills, decision-making, school climate, well-being, and academic achievement. While no studies focused on sense of belonging or school safety. Finally, the establishment of programme components, duration, and integration, for each variable, scientifically evidences the keys that can ensure the success of future SEL programmes.

Formal evidence-based medicine instruction in Turkish undergraduate medical education: an initial evaluation.

BACKGROUND: Global and national undergraduate medical education accreditation organizations recommend the inclusion of Evidence-Based Medicine (EBM) instructions into the medical schools' curricula. Accordingly, some Turkish medical schools have individually developed and implemented EBM training programs, but there is no data of current programs' effectiveness and students' learning achievements due to the lack of a validated Turkish language EBM assessment tool. This study evaluates the effect of a newly introduced formal EBM instruction to the curriculum on students' knowledge and skills by using the recently published Turkish adaptation of the Fresno Test. METHODS: The study is an experimental investigation using pre- and post-test evaluations. A five-week EBM course was developed according to Kern's six-step curriculum development approach. A total of 78 students from the third (n = 30), fourth (n = 19) and fifth (n = 29) year of medical school voluntarily consented and were enrolled into the course. Overall, the Cerrahpasa Medical Faculty had a total of 555, 461, and 400 students enrolled in the third, fourth, and fifth year, respectively. The program has been evaluated based on students' learning achievements and survey responses. RESULTS: The students' mean pre-test Fresno Test score improved from 49.9 ± 18.2 to 118.9 ± 26.3 post-training. The Cohen's effect size was 3.04 (95% CI, 2.6-3.5). The overall students' satisfaction score was 8.66 ± 1.09 on a 1 to 10 scale. CONCLUSIONS: The program was effective in improving students' knowledge and skills on EBM. We propose to offer the program as an elective course during the third year of the medical school curriculum based on all data obtained during the program evaluation.

Design and evaluation of the I-SCAN faculty POCUS program.

BACKGROUND: Point-of-care ultrasound (POCUS) is becoming widely adopted with increasing accessibility of courses. Little is known about the optimal design of the introductory course or longitudinal training programs targeting hospitalists that are critical to success. METHODS: Hospitalists at four academic sites participated in a two-day introductory course and a longitudinal phase comprising clinical POCUS practice, clip uploading with online feedback, hands-on teaching, and monthly ultrasound conferences. Assessments were performed immediately before and after the two-day course and after 1 year. RESULTS: Knowledge increased from baseline to post two-day course (median score 58 and 85%, respectively, p < 0.001) and decreased slightly at 1 year (median score 81%, p = 0.012). After the two-day introductory course, the median score for hands-on image acquisition skills, the principal metric of participant success, was 75%. After 1 year, scores were similar (median score 74%). Confidence increased from baseline to post two-day course (1.5 to 3.1 on a 4 point Likert scale from Not at all confident (1) to Very confident (4), p < 0.001), and remained unchanged after 1 year (2.73). Course elements correlating with a passing score on the final hands-on test included number of clip uploads (r = 0.85, p,0.001), attendance at hands-on sessions (r = 0.7, p = 0.001), and attendance at monthly conferences (r = 0.50, p = 0.03). CONCLUSIONS: The I-ScaN POCUS training program increased hospitalist knowledge, skill and confidence with maintained skill and confidence after 1 year. Uploading clips and attending hands-on teaching sessions were most correlative with participant success.

[How health care professionals perceive chronic disease: Changes in therapeutic patient education (TPE) assessment from 2011 to 2017 in a French university hospital]. / Perception du malade chronique chez les professionnels de santé : une évolution marquée par les indicateurs d'évaluation de programmes d'éducation thérapeutique du patient entre 2011 et 2017 au CHU de Clermont-Ferrand, France.

BACKGROUND: Since 2010, in France, Therapeutic Patient Education (TPE) programs have applied to the Regional Health Agency (RHA) for authorization. Every four years, these programs are mandatorily re-evaluated, and the assessment allows for change in the program management criteria. In our hospital, we studied the evolution of the Therapeutic Patient Education (TPE) appraisal benchmarks, otherwise known as "indicators", in the 17 programs having been authorized and renewed at least once by the RHA. METHOD: The TPE program appraisal benchmarks are classified in terms of structure, process and outcomes; program activity itself as well as pedagogic, psychosocial and bioclinical indicators are taken into consideration. We wished to determine the extent to which these indicators were addressed, applied and renewed or created during renewal of the TPE programs. Statistical tests were carried out in order to compare changes in the number of benchmarks in each category before and after the renewal process. RESULTS: During the first authorization, there existed 533 appraisal benchmarks, while they numbered 550 for the second. As for "before-and-after" changes, they consisted in a reduced number of outcome indicators (43.7% to 35.1%), whereas process indicators increased (36.8% to 43.1%) (P=0.0141). In comparison to the category pertaining to pedagogic, psychosocial and bioclinical indicators, the most widely registered indicator category (55.5%) and the most frequently collected indicator category involved the program activity itself (54.7%) (P<0.0001), which increased pronouncedly during renewal periods (67.6%) (P=0.0002). Conversely, the pedagogic and psychosocial indicators were little if at all collected. As regards the latter, there was nevertheless a considerable increase in indicators related to skills and changes favoring health-promoting behaviours. Strictly bioclinical indicators have been largely supplanted by those having to do with the disease evolution, its impact and risk management. CONCLUSION: The major role assigned to process and structure indicators reflects the fact that they are predominantly structured by RHA requests. Even if this initial study necessitates further research, it highlights a change in the design of educational and psychosocial assessments among caregivers, a change likely to reflect their interest in how patients go about managing their illnesses, (more or less healthy) lifestyles and daily lives.

Saving more than R$ 85 billion of the Brazilian Social Security System: the case of the PRBI / Economía de más de R$ 85 mil millones en el Régimen General de Previsión Social de Brasil: el caso del PRBI / Economizando mais de R$ 85 bilhões ao Regime Geral de Previdência Social do Brasil: o caso do PRBI

Abstract Since 2016, the number of recipients of incapacity allowance in Brazil has been continuously falling. This article presents the program of incapacity benefits assessment (PRBI) to help understand the dynamics around incapacity allowance and similar benefits. The study shows that the PRBI can save more than R$ 85 billion of the budget allocated to social security in the country.

Resumen El número de beneficiarios de subsidios por incapacidad laboral ha disminuido drásticamente desde 2016. Este artículo muestra que el Programa para la Evaluación de Subsidios por Incapacidad Laboral (PRBI) es clave para entender esta dinámica y es responsable de una economía de más de R$ 85 mil millones para el Régimen General de Previsión Social de Brasil.

Resumo O número de benefícios de auxílio-doença vem caindo drasticamente desde 2016. Este artigo mostra que o Programa de Revisão dos Benefícios por Incapacidade (PRBI) é fundamental para entender essa dinâmica, e estima que o Programa seja responsável por uma economia de mais de R$ 85 bilhões ao Regime Geral de Previdência Social.

Implementing bubble continuous positive airway pressure in a lower middle-income country: a Nigerian experience.

Bubble CPAP (bCPAP) is used for respiratory distress (RD) in neonates. The leading causes of neonatal mortality can lead to severe RD. Many neonatal deaths are preventable using evidence-based interventions like bCPAP as part of a comprehensive approach. The study aimed to assess the implementation of a multi-center, comprehensive hospital-based bCPAP program in a low-middle-income country using a low-cost bCPAP device. Seven established hospitals in three Nigerian States were selected using purposive sampling. A respiratory support program was developed and implemented using the Pumani® bCPAP. Neonates <28 days old with severe RD, birth weight >1000g and breathing spontaneously, were eligible. The program lasted 22 months. Focus group discussions and in-depth interviews of healthcare workers and hospital administrators were used in program assessment. Content analysis of qualitative data completed. The staff reported that the bCPAP device was easy to use and effective. All staff reported comfort in eligible patient identification, effective set up and bCPAP administration. All study sites experienced varying degrees of electric power interruption and oxygen availability and affordability. Staff training, staffing disruptions, data collection challenges and use of improvised bCPAP contributed to low enrollment. Advocacy, direct program support, and innovation using locally available resources improved enrollment. Professional organization collaboration, competency-based training and peer mentoring contributed to program success. Thorough pre-program assessment, with comprehensive understanding of all aspects of the existing system within the local context which are likely to impact the introduction of a new program is important to implementation success.

Development of a questionnaire to assess the quality of service trajectories in autism spectrum disorder from families' perspective.

BACKGROUND: ETAP-1 was created to evaluate the quality of services trajectory from families' perspective. The items of ETAP-1 were developed from previous studies on integrated care, existing quality assessments, and consultations with families and experts in evaluation and in autism spectrum disorder (ASD). METHOD: The questionnaire was completed by 200 parents of children aged 5 and under who were recently diagnosed with ASD or intellectual disability. Of these, 183 received diagnostic evaluation through a clinic specialized in ASD; the other 17 underwent diagnostic evaluation in hospital settings. RESULTS: Factor analysis supported the a priori dimensions of quality and distinctions between experiences before and during diagnostic evaluation. The instrument had high internal consistency, convergent and discriminant validity with other measures and was sensitive to differences in service delivery models. DISCUSSION: ETAP-1 is useful in organizing information on families' experiences throughout their services trajectories and according to a dynamic perspective.

Sexual Abuse Disclosure Mediates the Effect of an Abuse Prevention Program on Substantiation.

Although abuse prevention programs have proliferated, little research has explored the direct effects of such programs on actual child sexual abuse disclosure rates, and no research has explored the effects of such programs on child sexual abuse substantiation. Employing a quasi-experimental design, the present research reflects an exploration of the effects of exposure to the Think First and Stay Safe™ abuse prevention program on abuse disclosure rates of 319 children who underwent a child forensic interview within 2015-2018 in a Midwestern child advocacy center. Supporting our mediational hypotheses, children exposed (vs. not exposed) to the Think First and Stay Safe™ program were significantly more likely to disclose abuse during the forensic interview, which in turn predicted significantly increased abuse substantiation likelihood.

Faculty development program evaluation: a need to embrace complexity.

Faculty development is essential for renewing and assisting faculty to maintain teaching effectiveness and adapt to innovations in Health Professions educational institutions. The evaluation of faculty development programs appears to be a significant step in maintaining its relevance and efficiency. Yet, little has been published on the specific case of faculty development program evaluation in spite of the availability of general program evaluation models. These models do not measure or capture the information educators want to know about outcomes and impacts of faculty development. We posit that two reasons account for this. The first is the evolving nature of faculty development programs as they adapt to current reforms and innovations. The second involves the limitations imposed by program evaluation models that fail to take into account the multiple and unpredictable outcomes and impacts of faculty development. It is generally accepted that the outcomes and impacts are situated at various levels, ranging from the individual to the institutional and cultural levels. This calls for evaluation models that better capture the complexity of the impacts of faculty development, in particular the reciprocal relationships between program components and outcomes. We suggest conceptual avenues, based on Structuration Theory, that could lead to identifying the multilevel impacts of faculty development.