How does the structure of data impact cell-cell similarity? Evaluating how structural properties influence the performance of proximity metrics in single cell RNA-seq data.

Accurately identifying cell-populations is paramount to the quality of downstream analyses and overall interpretations of single-cell RNA-seq (scRNA-seq) datasets but remains a challenge. The quality of single-cell clustering depends on the proximity metric used to generate cell-to-cell distances. Accordingly, proximity metrics have been benchmarked for scRNA-seq clustering, typically with results averaged across datasets to identify a highest performing metric. However, the 'best-performing' metric varies between studies, with the performance differing significantly between datasets. This suggests that the unique structural properties of an scRNA-seq dataset, specific to the biological system under study, have a substantial impact on proximity metric performance. Previous benchmarking studies have omitted to factor the structural properties into their evaluations. To address this gap, we developed a framework for the in-depth evaluation of the performance of 17 proximity metrics with respect to core structural properties of scRNA-seq data, including sparsity, dimensionality, cell-population distribution and rarity. We find that clustering performance can be improved substantially by the selection of an appropriate proximity metric and neighbourhood size for the structural properties of a dataset, in addition to performing suitable pre-processing and dimensionality reduction. Furthermore, popular metrics such as Euclidean and Manhattan distance performed poorly in comparison to several lessor applied metrics, suggesting that the default metric for many scRNA-seq methods should be re-evaluated. Our findings highlight the critical nature of tailoring scRNA-seq analyses pipelines to the dataset under study and provide practical guidance for researchers looking to optimize cell-similarity search for the structural properties of their own data.

Automatic generation of pseudoknotted RNAs taxonomy.

BACKGROUND: The ability to compare RNA secondary structures is important in understanding their biological function and for grouping similar organisms into families by looking at evolutionarily conserved sequences such as 16S rRNA. Most comparison methods and benchmarks in the literature focus on pseudoknot-free structures due to the difficulty of mapping pseudoknots in classical tree representations. Some approaches exist that permit to cluster pseudoknotted RNAs but there is not a general framework for evaluating their performance. RESULTS: We introduce an evaluation framework based on a similarity/dissimilarity measure obtained by a comparison method and agglomerative clustering. Their combination automatically partition a set of molecules into groups. To illustrate the framework we define and make available a benchmark of pseudoknotted (16S and 23S) and pseudoknot-free (5S) rRNA secondary structures belonging to Archaea, Bacteria and Eukaryota. We also consider five different comparison methods from the literature that are able to manage pseudoknots. For each method we clusterize the molecules in the benchmark to obtain the taxa at the rank phylum according to the European Nucleotide Archive curated taxonomy. We compute appropriate metrics for each method and we compare their suitability to reconstruct the taxa.

A method for comparing multiple imputation techniques: A case study on the U.S. national COVID cohort collaborative.

Healthcare datasets obtained from Electronic Health Records have proven to be extremely useful for assessing associations between patients' predictors and outcomes of interest. However, these datasets often suffer from missing values in a high proportion of cases, whose removal may introduce severe bias. Several multiple imputation algorithms have been proposed to attempt to recover the missing information under an assumed missingness mechanism. Each algorithm presents strengths and weaknesses, and there is currently no consensus on which multiple imputation algorithm works best in a given scenario. Furthermore, the selection of each algorithm's parameters and data-related modeling choices are also both crucial and challenging. In this paper we propose a novel framework to numerically evaluate strategies for handling missing data in the context of statistical analysis, with a particular focus on multiple imputation techniques. We demonstrate the feasibility of our approach on a large cohort of type-2 diabetes patients provided by the National COVID Cohort Collaborative (N3C) Enclave, where we explored the influence of various patient characteristics on outcomes related to COVID-19. Our analysis included classic multiple imputation techniques as well as simple complete-case Inverse Probability Weighted models. Extensive experiments show that our approach can effectively highlight the most promising and performant missing-data handling strategy for our case study. Moreover, our methodology allowed a better understanding of the behavior of the different models and of how it changed as we modified their parameters. Our method is general and can be applied to different research fields and on datasets containing heterogeneous types.

Assessing safe and personalised maternity and neonatal care through a pandemic: a case study of outcomes and experiences in two trusts in England using the ASPIRE COVID-19 framework.

BACKGROUND: The COVID-19 pandemic has resulted in profound and far-reaching impacts on maternal and newborn care and outcomes. As part of the ASPIRE COVID-19 project, we describe processes and outcome measures relating to safe and personalised maternity care in England which we map against a pre-developed ASPIRE framework to establish the potential impact of the COVID-19 pandemic for two UK trusts. METHODS: We undertook a mixed-methods system-wide case study using quantitative routinely collected data and qualitative data from two Trusts and their service users from 2019 to 2021 (start and completion dates varied by available data). We mapped findings to our prior ASPIRE conceptual framework that explains pathways for the impact of COVID-19 on safe and personalised care. RESULTS: The ASPIRE framework enabled us to develop a comprehensive, systems-level understanding of the impact of the pandemic on service delivery, user experience and staff wellbeing, and place it within the context of pre-existing challenges. Maternity services experienced some impacts on core service coverage, though not on Trust level clinical health outcomes (with the possible exception of readmissions in one Trust). Both users and staff found some pandemic-driven changes challenging such as remote or reduced antenatal and community postnatal contacts, and restrictions on companionship. Other key changes included an increased need for mental health support, changes in the availability and uptake of home birth services and changes in induction procedures. Many emergency adaptations persisted at the end of data collection. Differences between the trusts indicate complex change pathways. Staff reported some removal of bureaucracy, which allowed greater flexibility. During the first wave of COVID-19 staffing numbers increased, resolving some pre-pandemic shortages: however, by October 2021 they declined markedly. Trying to maintain the quality and availability of services had marked negative consequences for personnel. Timely routine clinical and staffing data were not always available and personalised care and user and staff experiences were poorly captured. CONCLUSIONS: The COVID-19 crisis magnified pre-pandemic problems and in particular, poor staffing levels. Maintaining services took a significant toll on staff wellbeing. There is some evidence that these pressures are continuing. There was marked variation in Trust responses. Lack of accessible and timely data at Trust and national levels hampered rapid insights. The ASPIRE COVID-19 framework could be useful for modelling the impact of future crises on routine care.

Global Perceptions on Social Accountability and Outcomes: A Survey of Medical Schools.

Phenomenon: Social accountability has become a universal component in medical education. However, medical schools have little guidance for operationalizing and applying this concept in practice. This study explored institutional practices and administrative perceptions of social accountability in medical education. Approach: An online survey was distributed to a purposeful sample of English-speaking undergraduate medical school deans and program directors/leads from 245 institutions in 14 countries. The survey comprised of 38-items related to program mission statements, admission processes, curricular content, and educational outcomes. Survey items were developed using previous literature and categorized using a context-input-process-products (CIPP) evaluation model. Exploratory Factor Analysis (EFA) was used to assess the inter-relationship among survey items. Reliability and internal consistency of items were evaluated using McDonald's Omega. Findings: Results from 81 medical schools in 14 countries collected between February and June 2020 are presented. Institutional commonalities of social accountability were observed. However, our findings suggest programs focus predominately on educational inputs and processes, and not necessarily on outcomes. Findings from our EFA demonstrated excellent internal consistency and reliability. Four-factors were extracted: (1) selection and recruitment; (2) institutional mandates; (3) institutional activities; and (4) community awareness, accounting for 71% of the variance. McDonald's Omega reliability estimates for subscales ranged from 0.80-0.87. Insights: This study identified common practices of social accountability. While many medical schools expressed an institutional commitment to social accountability, their effects on the community remain unknown and not evaluated. Overall, this paper offers programs and educators a psychometrically supported tool to aid in the operationalization and reliability of evaluating social accountability.

Analysis and intervention of heatwave related economic loss: Comprehensive insights from supply, demand, and public expenditure into the relationship between the influencing factors.

Increasing extreme temperatures are producing a serious impact on the economies of cities. However, the importance of social factors is typically neglected by the existing research. In this work, we first establish a supply-demand-public expenditure (SDP) framework for assessing and forecasting heat-related economic loss. Compared with the previous framework, SDP possesses a more comprehensive index system and functions that apply to all types of cities. We selected different economic development and geographical locations (Nanjing, Suzhou, and Yancheng) as case studies to verify the wide applicability of the SDP framework. A qualitative analysis and quantitative prediction of heatwaves and socioeconomic factors on losses were conducted for different cities. The results showed that different loss types displayed obvious regional heterogeneity among the cities. The labor value loss was the most significant type, and health loss was the most vulnerable type. In addition, public expenditure played a neglected critical regulatory role. Apart from these, the current level of public expenditure for heat prevention and control remains insufficient. Based on an assessment of the effects of interventions, policymakers need to make more efforts to increase the proportion of heat-related public spending and ensure stable socio-economic development by utilizing pathways with positive intervention potentials.

Same, Same but Different: Comparing Program Fidelity at Early Versus Late Adopters of a Successful Long-Term School-Based Physical Activity Intervention.

Schools have been identified as a promising setting for promoting physical activity (PA). Yet, to realize changes at the population level, successful school-based PA programs need to go to scale. The Svendborgproject is an effective school-based program promoting additional physical education (PE) lessons. The aim of this study is to determine program fidelity across different school groups, representing early and late adopters of the Svendborgproject, and how these are adapting the intervention. Three different school groups were identified, covering the original intervention schools and two groups of late adopters consisting of four former control schools, and five normal schools without any previous connection to the program. A PE teacher questionnaire (n = 122) was used to determine school fidelity. The results show that, while the original intervention schools have implemented the program with the highest fidelity, all schools have implemented the program with medium to high fidelity. It is suggested that having front-runner schools achieving early success with the program both strengthens political project support and provides strategies to back late adopters' implementation of the program. Furthermore, results from the current study suggest that continual promotion of the program by school heads is less important if support is established at the structural and organizational macro level. Finally, we highlight the importance of scaling up organizational capacity when scaling up program reach to assure a workable balance between fidelity and improving the fit to specific contexts.

Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework.

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

National influenza surveillance systems in five European countries: a qualitative comparative framework based on WHO guidance.

BACKGROUND: Influenza surveillance systems vary widely between countries and there is no framework to evaluate national surveillance systems in terms of data generation and dissemination. This study aimed to develop and test a comparative framework for European influenza surveillance. METHODS: Surveillance systems were evaluated qualitatively in five European countries (France, Germany, Italy, Spain, and the United Kingdom) by a panel of influenza experts and researchers from each country. Seven surveillance sub-systems were defined: non-medically attended community surveillance, virological surveillance, community surveillance, outbreak surveillance, primary care surveillance, hospital surveillance, mortality surveillance). These covered a total of 19 comparable outcomes of increasing severity, ranging from non-medically attended cases to deaths, which were evaluated using 5 comparison criteria based on WHO guidance (granularity, timing, representativeness, sampling strategy, communication) to produce a framework to compare the five countries. RESULTS: France and the United Kingdom showed the widest range of surveillance sub-systems, particularly for hospital surveillance, followed by Germany, Spain, and Italy. In all countries, virological, primary care and hospital surveillance were well developed, but non-medically attended events, influenza cases in the community, outbreaks in closed settings and mortality estimates were not consistently reported or published. The framework also allowed the comparison of variations in data granularity, timing, representativeness, sampling strategy, and communication between countries. For data granularity, breakdown per risk condition were available in France and Spain, but not in the United Kingdom, Germany and Italy. For data communication, there were disparities in the timeliness and accessibility of surveillance data. CONCLUSIONS: This new framework can be used to compare influenza surveillance systems qualitatively between countries to allow the identification of structural differences as well as to evaluate adherence to WHO guidance. The framework may be adapted for other infectious respiratory diseases.

A multi-step approach to developing a health system evaluation framework for community-based health care.

BACKGROUND: Community-based health care (CBHC) is a shift towards healthcare integration and community services closer to home. Variation in system approaches harkens the need for a conceptual framework to evaluate outcomes and impacts. We set out to develop a CBHC-specific evaluation framework in the context of a provincial ministry of health planning process in Canada. METHODS: A multi-step approach was used to develop the CBHC evaluation framework. Modified Delphi informed conceptualization and prioritization of indicators. Formative research identified evaluation framework elements (triple aim, global measures, and impact), health system levels (tiers), and potential CBHC indicators (n = 461). Two Delphi rounds were held. Round 1, panelists independently ranked indicators on CBHC relevance and health system tiering. Results were analyzed by coding agreement/disagreement frequency and central tendency measures. Round 2, a consensus meeting was used to discuss disagreement, identify Tier 1 indicators and concepts, and define indicators not relevant to CBHC (Tier 4). Post-Delphi, indicators and concepts were refined, Tier 1 concepts mapped to the evaluation framework, and indicator narratives developed. Three stakeholder consultations (scientific, government, and public/patient communities) were held for endorsement and recommendation. RESULTS: Round 1 Delphi results showed agreement for 300 and disagreement for 161 indicators. Round 2 consensus resulted in 103 top tier indicators (Tier 1 = 19, Tier 2 = 84), 358 bottom Tier 3 and 4 indicators, non-CBHC measure definitions, and eight Tier 1 indicator concepts-Mortality/Suicide; Quality of Life, and Patient Reported Outcome Measures; Global Patient Reported Experience Measures; Cost of Care, Access to Integrated Primary Care; Avoidable Emergency Department Use; Avoidable Hospitalization; and E-health Penetration. Post Delphi results refined Tier 3 (n = 289) and 4 (n = 69) indicators, and identified 18 Tier 2 and 3 concepts. When mapped to the evaluation framework, Tier 1 concepts showed full coverage across the elements. 'Indicator narratives' depicted systemness and integration for evaluating CBHC. Stakeholder consultations affirmed endorsement of the approach and evaluation framework; refined concepts; and provided key considerations to further operationalize and contextualize indicators, and evaluate CBHC as a health system approach. CONCLUSIONS: This research produced a novel evaluation framework to conceptualize and evaluate CBHC initiatives. The evaluation framework revealed the importance of a health system approach for evaluating CBHC.

Designing evaluation framework for the empirical assessment of COVID-19 mobile apps in Pakistan.

The significant proliferation in the mobile health applications (Apps) amidst Coronaviruses disease 2019 (COVID-19) resulted in decision making problems for healthcare professionals, decision makers and mobile users in Pakistan. This decision making process is also hampered by mobile app trade-offs, multiple features support, evolving healthcare needs and varying vendors. In this regard, evaluation model for mobile apps is presented which completes in three different phases. In first phase, features-based criteria is designed by leveraging Delphi method, and twenty (20) mobile apps are selected from app stores. In second stage, empirical evaluation is performed by using hybrid multi criteria decision approaches like CRiteria Importance Through Inter-criteria Correlation (CRITIC) method has been used for assigning weights to criteria features; and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method has been used for assessment of mobile app alternatives. In last step, decision making is performed to select the best mobile app for COVID-19 situations. The results suggest that proposed model can be adopted as a guideline by mobile app subscribers, patients and healthcare professionals.

Evaluation Framework for Successful Artificial Intelligence-Enabled Clinical Decision Support Systems: Mixed Methods Study.

BACKGROUND: Clinical decision support systems are designed to utilize medical data, knowledge, and analysis engines and to generate patient-specific assessments or recommendations to health professionals in order to assist decision making. Artificial intelligence-enabled clinical decision support systems aid the decision-making process through an intelligent component. Well-defined evaluation methods are essential to ensure the seamless integration and contribution of these systems to clinical practice. OBJECTIVE: The purpose of this study was to develop and validate a measurement instrument and test the interrelationships of evaluation variables for an artificial intelligence-enabled clinical decision support system evaluation framework. METHODS: An artificial intelligence-enabled clinical decision support system evaluation framework consisting of 6 variables was developed. A Delphi process was conducted to develop the measurement instrument items. Cognitive interviews and pretesting were performed to refine the questions. Web-based survey response data were analyzed to remove irrelevant questions from the measurement instrument, to test dimensional structure, and to assess reliability and validity. The interrelationships of relevant variables were tested and verified using path analysis, and a 28-item measurement instrument was developed. Measurement instrument survey responses were collected from 156 respondents. RESULTS: The Cronbach α of the measurement instrument was 0.963, and its content validity was 0.943. Values of average variance extracted ranged from 0.582 to 0.756, and values of the heterotrait-monotrait ratio ranged from 0.376 to 0.896. The final model had a good fit (χ262=36.984; P=.08; comparative fit index 0.991; goodness-of-fit index 0.957; root mean square error of approximation 0.052; standardized root mean square residual 0.028). Variables in the final model accounted for 89% of the variance in the user acceptance dimension. CONCLUSIONS: User acceptance is the central dimension of artificial intelligence-enabled clinical decision support system success. Acceptance was directly influenced by perceived ease of use, information quality, service quality, and perceived benefit. Acceptance was also indirectly influenced by system quality and information quality through perceived ease of use. User acceptance and perceived benefit were interrelated.

Evaluation of Task Scheduling Algorithms in Heterogeneous Computing Environments.

This work establishes a set of methodologies to evaluate the performance of any task scheduling policy in heterogeneous computing contexts. We formally state a scheduling model for hybrid edge-cloud computing ecosystems and conduct simulation-based experiments on large workloads. In addition to the conventional cloud datacenters, we consider edge datacenters comprising smartphone and Raspberry Pi edge devices, which are battery powered. We define realistic capacities of the computational resources. Once a schedule is found, the various task demands can or cannot be fulfilled by the resource capacities. We build a scheduling and evaluation framework and measure typical scheduling metrics such as mean waiting time, mean turnaround time, makespan, throughput on the Round-Robin, Shortest Job First, Min-Min and Max-Min scheduling schemes. Our analysis and results show that the state-of-the-art independent task scheduling algorithms suffer from performance degradation in terms of significant task failures and nonoptimal resource utilization of datacenters in heterogeneous edge-cloud mediums in comparison to cloud-only mediums. In particular, for large sets of tasks, due to low battery or limited memory, more than 25% of tasks fail to execute for each scheduling scheme.

Resource Management Techniques for Cloud/Fog and Edge Computing: An Evaluation Framework and Classification.

Processing IoT applications directly in the cloud may not be the most efficient solution for each IoT scenario, especially for time-sensitive applications. A promising alternative is to use fog and edge computing, which address the issue of managing the large data bandwidth needed by end devices. These paradigms impose to process the large amounts of generated data close to the data sources rather than in the cloud. One of the considerations of cloud-based IoT environments is resource management, which typically revolves around resource allocation, workload balance, resource provisioning, task scheduling, and QoS to achieve performance improvements. In this paper, we review resource management techniques that can be applied for cloud, fog, and edge computing. The goal of this review is to provide an evaluation framework of metrics for resource management algorithms aiming at the cloud/fog and edge environments. To this end, we first address research challenges on resource management techniques in that domain. Consequently, we classify current research contributions to support in conducting an evaluation framework. One of the main contributions is an overview and analysis of research papers addressing resource management techniques. Concluding, this review highlights opportunities of using resource management techniques within the cloud/fog/edge paradigm. This practice is still at early development and barriers need to be overcome.

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA.

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA's educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a "wicked problem" in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

Community-based performance indicators for monitoring and evaluating livestock interventions.

The research aimed at identifying livestock performance indicators used by farmers in Malipati community, Zimbabwe, and use them in developing a monitoring and evaluation framework for livestock interventions. Mixed methods research was used in the study. A questionnaire was administered to identify performance indicators of preference by farmers. Focus group discussions were done to rank performance indicators. Data analysis was done using SPSS version 25, and data were analysed using the ranking matrix. Scientific validity of performance indicators was determined through literature review. The study concluded that performance indicators of importance in poultry, cattle, goats/sheep, and donkeys were egg production, milk yield, kidding/lambing interval, and animal power, respectively. All performance indicators identified by farmers in Malipati are scientifically valid and were used in the development of the monitoring and evaluation framework.

A systematic review of the use and reporting of evaluation frameworks within evaluations of physical activity interventions.

BACKGROUND: Evaluation of physical activity interventions is vital to inform, and justify, evidence-based policy and practice to support population-wide changes in physical activity. Several evaluation frameworks and guidance documents have been developed to facilitate the evaluation and reporting of evaluation studies in public health. However, there is a lack of evidence about whether frameworks are being used to guide evaluation. There continues to be claims of poor and inconsistent reporting in evaluation studies. The aim of this review was to assess the use of evaluation frameworks and the quality of reporting of how they were applied within evaluation studies of physical activity interventions. OBJECTIVES: 1. To identify whether evaluation frameworks are reported to have been used within evaluation studies of physical activity interventions, and which frameworks have been used. 2. To appraise the quality of reporting with regards to how evaluation frameworks have been used. METHOD: We developed a checklist of indicators to enable a critical appraisal of the use and reporting of different evaluation frameworks in evaluation studies. We conducted a systematic search and review of evaluation studies published between 2015 and the date of the search to appraise the use and reporting of evaluation frameworks. A narrative synthesis is provided. RESULTS: The review identified 292 evaluation studies of physical activity interventions, only 69 (23%) of these mentioned using an evaluation framework, and only 16 different frameworks were referred to. There was variation in the quality of reporting of framework use. 51 (74%) studies were identified as being explicitly based on the stated framework, however only 26 (38%) provided detailed descriptions consistently across all the checklist indicators. Details of adaptations and limitations in how frameworks were applied were less frequently reported. The review also highlighted variability in the reporting of intervention components. More consistent and precise reporting of framework and intervention components is needed. CONCLUSION: Evaluation frameworks can facilitate a more systematic evaluation report and we argue their limited use suggests missed opportunities to apply frameworks to guide evaluation and reporting in evaluation studies. Variability in the quality of reporting of framework use limits the comparability and transferability of evidence. Where a framework has been used, the checklist of indicators can be employed to facilitate the reporting of an evaluation study and to review the quality of an evaluation report.

Simulation exercises and after action reviews - analysis of outputs during 2016-2019 to strengthen global health emergency preparedness and response.

BACKGROUND: Under the International Health Regulations (2005) [IHR (2005)] Monitoring and Evaluation Framework, after action reviews (AAR) and simulation exercises (SimEx) are two critical components which measure the functionality of a country's health emergency preparedness and response under a "real-life" event or simulated situation. The objective of this study was to describe the AAR and SimEx supported by the World Health Organization (WHO) globally in 2016-2019. METHODS: In 2016-2019, WHO supported 63 AAR and 117 SimEx, of which 42 (66.7%) AAR reports and 56 (47.9%) SimEx reports were available. We extracted key information from these reports and created two central databases for AAR and SimEx, respectively. We conducted descriptive analysis and linked the findings according to the 13 IHR (2005) core capacities. RESULTS: Among the 42 AAR and 56 SimEx available reports, AAR and SimEx were most commonly conducted in the WHO African Region (AAR: n = 32, 76.2%; SimEx: n = 32, 52.5%). The most common public health events reviewed or tested in AAR and SimEx, respectively, were epidemics and pandemics (AAR: n = 38, 90.5%; SimEx: n = 46, 82.1%). For AAR, 10 (76.9%) of the 13 IHR core capacities were reviewed at least once, with no AAR conducted for food safety, chemical events, and radiation emergencies, among the reports available. For SimEx, all 13 (100.0%) IHR capacities were tested at least once. For AAR, the most commonly reviewed IHR core capacities were health services provision (n = 41, 97.6%), risk communication (n = 39, 92.9%), national health emergency framework (n = 39, 92.9%), surveillance (n = 37, 88.1%) and laboratory (n = 35, 83.3%). For SimEx, the most commonly tested IHR core capacity were national health emergency framework (n = 56, 91.1%), followed by risk communication (n = 48, 85.7%), IHR coordination and national IHR focal point functions (n = 45, 80.4%), surveillance (n = 31, 55.4%), and health service provision (n = 29, 51.8%). For AAR, the median timeframe between the end of the event and AAR was 125 days (range = 25-399 days). CONCLUSIONS: WHO has recently published guidance for the planning, execution, and follow-up of AAR and SimEx. Through the guidance and the simplified reporting format provided, we hope to see more countries conduct AAR and SimEx and standardization in their methodology, practice, reporting and follow-up.

A pragmatic guide to geoparsing evaluation: Toponyms, Named Entity Recognition and pragmatics.

Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. (Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. (Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. (Part 3) Evaluation data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.