Challenges and potential improvements in the Accreditation Standards of the Korean Institute of Medical Education and Evaluation 2019 (ASK2019) derived through meta-evaluation: a cross-sectional study

PURPOSE: This study aimed to identify challenges and potential improvements in Korea's medical education accreditation process according to the Accreditation Standards of the Korean Institute of Medical Education and Evaluation 2019 (ASK2019). Meta-evaluation was conducted to survey the experiences and perceptions of stakeholders, including self-assessment committee members, site visit committee members, administrative staff, and medical school professors. METHODS: A cross-sectional study was conducted using surveys sent to 40 medical schools. The 332 participants included self-assessment committee members, site visit team members, administrative staff, and medical school professors. The t-test, one-way analysis of variance and the chi-square test were used to analyze and compare opinions on medical education accreditation between the categories of participants. RESULTS: Site visit committee members placed greater importance on the necessity of accreditation than faculty members. A shared positive view on accreditation's role in improving educational quality was seen among self-evaluation committee members and professors. Administrative staff highly regarded the Korean Institute of Medical Education and Evaluation's reliability and objectivity, unlike the self-evaluation committee members. Site visit committee members positively perceived the clarity of accreditation standards, differing from self-assessment committee members. Administrative staff were most optimistic about implementing standards. However, the accreditation process encountered challenges, especially in duplicating content and preparing self-evaluation reports. Finally, perceptions regarding the accuracy of final site visit reports varied significantly between the self-evaluation committee members and the site visit committee members. CONCLUSION: This study revealed diverse views on medical education accreditation, highlighting the need for improved communication, expectation alignment, and stakeholder collaboration to refine the accreditation process and quality.

Evaluating governance in a clinical and translational research organization.

Institutional Development Awards for Clinical and Translational Research (IDeA-CTR) networks, funded by NIH/NIGMS, aim to advance CTR infrastructure to address historically unmet state and regional health needs. Success depends on the response to actionable feedback to IDeA-CTR leadership from network partners and governance groups through annual surveys, interviews, and governance body recommendations. The Great Plains IDeA-CTR applied internal formative meta-evaluation to evaluate dispositions of 172 governance recommendations from 2017 to 2021. Results provided insights to improve the classification and quality of recommendations, credibility of evaluation processes, responsiveness to recommendations, and communications and governance in a complex CTR network comprising multiple coalitions.

Clearing the confusion about post-accreditation monitoring, meta-evaluation and meta-accreditation.

We have recently published the experience of the accreditation body of undergraduate medical education in Iran on developing and validating standards based on the WFME framework (Gandomkar et al., BMC Med Educ 23:379, 2023). Agabagheri et al. extended our work and proposed a blueprint for post-accreditation monitoring based on their experience in developing an official guide in their Matters Arising (Aghabagheri et al., BMC Med Educ). The authors have used post-accreditation monitoring as a process of monitoring and controlling accreditation activities, procedures often referred to as meta-evaluation or meta-accreditation (depending on the objectives of evaluation) in the literature. On the contrary, post-accreditation monitoring alludes to the process of continuous quality improvement of educational programs after accreditation. We would like to make clarifications between post-accreditation monitoring, meta-evaluation and meta-accreditation which have been used interchangeably in their paper. Considering the emerging interests in scholarship and non-scholarship activities and reports in undergraduate medical education accreditation, this clarification provides a better understanding of the roles of these crucial concepts in the accreditation process.

Meta-evaluation of studies on community physical activity programs in Brazil.

BACKGROUND: The use of the meta-evaluation, especially in health promotion programs, is still scarce. AIM: To verify if the evaluation studies conducted about the Academia da Cidade Programme and the Academia da Saúde Programme comply with the principles and standards of the meta-evaluation. METHODS: This meta-evaluation study was conducted using articles from scientific databases. The parameters of utility, propriety, and accuracy, as defined by the Joint Committee on Standards for Educational Evaluation, and specificity, were applied. The classification of the studies was based on the Stufflebeam's checklist. RESULTS: Of the 18 selected articles, the principle of utility was fully met in 50%, given that 27.8% were classified as very good, 5.6% as good, and 16.7% as fair. The principle of propriety mostly received the result of very good (44.4%), followed by excellent (38.9%), good (11.15), and weak (5.6%). The principle of accuracy was classified as excellent in 44.4% of the studies, considering that in 33.3% it was considered very good; 11.1% good, and 11.1% fair. Specificity was classified as very good in 94.4% of the studies. CONCLUSION: The physical activity community programs evaluations were carried out in accordance to the principles and standards of meta-evaluation tenets. Most studies were classified as excellent regarding the principle of utility and accuracy, or very good as for the principle of propriety and specificity.

Using the PMAQ-AB Mobile App and Management System to Evaluate the Quality of Primary Health Care in Brazil: Qualitative Case Study.

BACKGROUND: The application of cell phones, similar portable devices (ie, tablets), apps, the internet, and GPS in evaluation have established new ways of collecting, storing, retrieving, transmitting, and processing data or information. However, evidence is incipient as to which technological resources remain at the center of assessment practice and the factors that promote their use by the assessment community. OBJECTIVE: This study aimed to analyze the relationship between the use of the National Program for Improving Primary Healthcare Access and Quality's (PMAQ-AB; Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica) mobile app and management system and the external evaluation quality of Brazil's PMAQ-AB. METHODS: We conducted a qualitative case study during the external evaluation of Brazil's PMAQ-AB. Data collection consisted of interviews, focus groups, and document analysis. A total of 7 members from the Department of Primary Care of the Ministry of Health and 47 researchers from various higher education and research institutions across the country participated in the study. Data were categorized using the ATLAS.ti software program, according to the quality standards of the Joint Committee on Standards for Educational Evaluation, following the content analysis approach by Bardin. RESULTS: The results related to feasibility, thematic scope, field activity management, standardized data collection, data consistency, and transparency. They demonstrated improvements and opportunities for advancements in evaluation mediated by the use of information technology (IT), favored the emergence of new practices and remodeling of existing ones, and took into account the multiple components required by the complex assessment of access and quality in primary health care. Difficulties in technology operation, inoperative systems, and lack of investment in equipment and human resources posed challenges to increasing the effectiveness of IT in evaluation. CONCLUSIONS: The use of technology-based tools-the app and the management system-during the external evaluation offered evaluators a greater opportunity for stakeholder engagement. This also allowed the insertion of different organizational, operational, and methodological components that are capable of triggering influences and confluences. In addition, this allowed connections in collaborative and synergistic networks to increase the quality and allow the development of a more consistent and efficient evaluation process with greater possibility of incorporating the results into public health policies.

Avaliação da atenção primária à saúde e a incorporação da tecnologia da informação para a melhoria da qualidade da avaliação / Evaluation of primary health care and the incorporation of information technology to improve the quality of the evaluation

Apesar da ascendência de avaliações na Atenção Primária à Saúde, com possibilidades para o desenvolvimento de meta-avaliações, bem como, o incremento de investimentos e acesso às Tecnologias da Informação, com aumento de sua aplicabilidade em processos avaliativos, principalmente, para a coleta de dados e disseminação dos resultados, até o momento, são incipientes estudos meta-avaliativos que evidenciem as consequências das relações que se estabelecem entre as tecnologias e os avaliadores na qualidade da avaliação. Utilizando a Scoping Review, o Estudo de Caso e a estrutura conceitual da sociomaterialidade, esta tese procurou contribuir para este campo de pesquisa e para as práticas avaliativas, avaliando a incorporação da Tecnologia da Informação na avaliação da Atenção Primária à Saúde no Brasil considerando os padrões internacionais de qualidade. A Scoping Review teve o objetivo de mapear os usos da meta-avaliação dos estudos avaliativos desenvolvidos na Atenção Primária à Saúde. Foram incluídos 23 estudos, publicados entre 1999 e 2018, predominantemente realizados no Brasil. Tiveram mais ênfase as meta-avaliações somativas, com foco em programas de saúde e na gestão da Atenção Primária, que utilizaram como padrões avaliativos os recomendados pelo Joint Committeeon Standards for Educational Evaluation. O Estudo de Caso teve abordagem qualitativa. Participaram 07 técnicos da coordenação do Ministério da Saúde e 47 pesquisadores de Instituições de Ensino e Pesquisa, com uma amostra total de54 participantes. A coleta de dados ocorreu entre julho de 2018 a julho de 2019, utilizaram-se entrevistas, grupos focais e análise documental. Os dados foram categorizados por padrões de qualidade com uso do software Atlas ti, seguido de análise de conteúdo de Bardin. Para o estudo de caso sobre coleta de dados, os resultados evidenciaram que a tecnologia viabilizou a realização da avaliação em todo território nacional; contribuiu para a operacionalização de uma avaliação com grande escopo temático; forneceu recursos para a gestão da atividade de campo; minimizou erros, agilizou e homogeneizou a coleta; garantiu a integralidade das informações necessárias com o mínimo de inconsistência, permitindo o alcance de resultados uteis e válidos para certificação das equipes, políticas de saúde e pesquisas. Além disto, fatores como dificuldade de manuseio, inoperância do sistema e falta de investimentos em equipamentos e recursos humanos podem representar possíveis restrições para a eficiência da aplicação da tecnologia na avaliação. Quanto ao estudo sobre disseminação dos resultados, constaram-se avanços na visualização dos dados e na implementação de iniciativas de dados abertos, possibilitando sua disponibilidade para profissionais, formuladores de políticas e universidades. Desafios relacionados ao tempo de entrega dos relatórios, restrição de acesso, referência à falta de interesse e habilidades digitais, influenciaram no alcance dos objetivos esperados pelo proponente da avaliação. Considera-se que a incorporação da Tecnologia da Informação tem potencial ao alcance de padrões de qualidade, oportunizando a implementação de avaliações mais eficazes, com a produção de melhores resultados, atuando construtivamente no processo de melhorias de desempenho das equipes de saúde e da gestão, induzindo a redefinição dos seus processos prioritários de trabalho, com vistas ao aprimoramento e fortalecimento da Atenção Primária à Saúde (AU).

Despite the ascendancy of evaluations in Primary Health Care, with possibilities for the development of meta-evaluations, as well as the increase in investments and access to Information Technologies, with an increase in its applicability in evaluation processes, mainly for the collection of data and dissemination of results, so far, are incipient meta-evaluative studies that show the consequences of the relationships established between technologies and evaluators in the quality of the assessment. Using the Scoping Review, the Case Study, and the conceptual structure of sociomateriality, this thesis sought to contribute to this field of research and evaluative practices, evaluating the incorporation of Information Technology in the evaluation of Primary Health Care in Brazil considering international quality standards. The scoping review was developed following the Joanna Briggs Institute manual to map the uses of the meta-evaluation of evaluative studies carried out in Primary Health Care. 23 studies, published between 1999 and 2018, predominantly carried out in Brazil, were included. There was more emphasis on summative meta-evaluations, with a focus on health programs and on the management of Primary Care, which used as evaluative standards those recommended by the Joint Committee on Standards for Educational Evaluation. The Case Study had a qualitative approach. 07 technicians from the coordination of the Ministry of Health and 47 researchers from Education and Research Institutions participated, with a total sample of 54 participants. Data collection took place between July 2018 and July 2019, using interviews, focus groups, and document analysis. Data were categorized by quality standards using the Atlas ti software followed by Bardin's content analysis. For the case study on data collection, the results showed that the technology made it possible to assess the national territory; contributed to the operationalization of an assessment with a large thematic scope; provided resources for field activity management; minimized errors, streamlined and homogenized the collection; ensured the completeness of the necessary information with a minimum of inconsistency, allowing the achievement of useful and valid results for the certification of teams, health policies and research. In addition, factors such as difficulty in handling, inoperability of the system, and lack of investment in equipment and human resources can represent possible restrictions for the efficiency of the application of technology in the assessment. As for the study on the dissemination of results, advances were found in data visualization and the implementation of open data initiatives, enabling its availability to professionals, policymakers and universities. Challenges related to reporting delivery time, access restriction, reference to lack of interest, and digital skills influenced the achievement of the objectives expected by the evaluation proponent. It is considered that the incorporation of Information Technology has the potential to reach quality standards, providing opportunities for the implementation of more effective assessments, with the production of better results, acting constructively in the process of improving the performance of health and management teams, inducing the redefinition of its priority work processes, to improve and strengthen Primary Health Care (AU).

Human brain responses to gustatory and food stimuli: A meta-evaluation of neuroimaging meta-analyses.

Multiple neuroimaging meta-analyses have been published concerning gustation, food and taste. A meta-evaluation of these meta-analyses was conducted to qualitatively evaluate the presented evidence. A systematic search was done using multiple databases, in which no restriction was placed on participants and nature of interventions (stimuli vs control). Twenty-three meta-analyses were identified and analyzed. All of them have met 4-9 criteria, out of 11, from the modified checklist constructed by Müller et al. (2018), which implied moderate to high quality of evidence. One of the concerns we found was that no meta-analysis surveyed had been explicitly pre-registered. Also, only three meta-analyses (13.0%) provided clear explanation of how they accounted for sample overlap. Only six meta-analyses (26.1%) explicitly described how they double checked the data. Only two of the 20 meta-analyses (10.0%) using GingerALE software used both the debugged version (v2.3.6) as well as the recommended cluster-level inference with familywise error rate correction. Overall, meta-analyses are increasingly adopting more stringent statistical thresholds, but unfortunately not larger number of studies contained in the analyses.

Synthesis of articles in Evaluation and Program Planning: 2010-2016.

Published articles from the Evaluation and Program Planning journal were examined over a six year period from 2010 to 2016. We investigated the focus of the journal, evaluation type (formative vs summative), number of articles published, place of authorship, number of authors, research domain of articles, research topics, and data collection method used. Results indicated that (a) public health, evaluation, and adolescent/child research domains were most prevalent; (b) most authors were from North-America; (c) most articles had three or more authors; and (d) document review was most prevalent data collection method. We suggest that more articles with a multicultural background be published, and more articles be solicited from other countries to fulfull the international mission of the journal.

Meta-Evaluation: Experiences in an Accelerated Graduate Nurse Education Program.

Most schools of nursing are engaged in some form of program evaluation and recognize the potential benefits in using program evaluation outcomes to influence continuous improvement in program quality. A number of factors exist that may negatively influence program evaluation quality and adversely affect the ability to make sound decisions based on program evaluation outcomes. The potential limitations that threaten program evaluation quality underscore the importance of evaluating the evaluation process itself, also known as meta-evaluation. However, there is an absence of discussion in the nursing literature of the importance of program meta-evaluation. This article seeks to address this gap in the nursing literature and illuminate the need for more schools of nursing to engage in the meta-evaluation process. By introducing 1 model of program meta-evaluation and describing our own endeavors in the program meta-evaluation process, we hope to inspire other schools of nursing to consider using a systematic and formalized process to evaluate their own program evaluation processes to ensure that data obtained from program evaluation are of optimal quality to influence sound, data-driven decisions to promote continued quality and excellence in nursing education programs.

Parâmetros e paradigmas em meta-avaliação: uma revisão exploratória e reflexiva / Parameters and paradigms for meta-evaluation: an exploratory and reflective review

A avaliação da avaliação não é um simples jogo de palavras ou uma indagação sobre quem avalia a avaliação. Muito mais do que isso, trata-se de avaliar os diversos componentes de um processo avaliativo e os próprios avaliadores. No presente artigo, pretendemos abordar algumas possibilidades da meta-avaliação no campo da avaliação de programas e serviços: fomentar a interação entre a produção teórica e prática no campo; auxiliar no balizamento de métodos e estratégias em uma área marcada por interesses políticos e pela profusão de possibilidades metodológicas; e fomento do rigor ético e científico das práticas avaliativas. Concluiremos esta resenha exploratória da literatura discutindo algumas questões históricas e políticas envolvendo a avaliação e a meta-avaliação de programas e serviços, apresentando críticas às pretensões universalistas e igualitárias presentes em alguns posicionamentos.

The evaluation of an evaluation is not a mere play on words or simply an investigation into the person who evaluates the evaluation. It is far more than that. It is about how to evaluate the diverse components of the evaluation process and the evaluators per se. In this paper, we discuss some promising possibilities for meta-evaluation in the field of evaluation of programs and services. These include fostering interaction between theoretical and practical production in fieldwork and supporting the definition of methods and strategies in a sector imbued with political interests and a profusion of methodological possibilities, and promoting ethical and scientific rigor in the evaluation practices. We conclude this exploratory and reflective review by raising some historical and political questions related to evaluation and meta-evaluation of programs and services and by criticizing the universalist and egalitarian pretensions contained in some evaluation approaches.