Validation of algorithms in studies based on routinely collected health data: general principles.

Clinicians, researchers, regulators, and other decision-makers increasingly rely on evidence from real-world data (RWD), including data routinely accumulating in health and administrative databases. RWD studies often rely on algorithms to operationalize variable definitions. An algorithm is a combination of codes or concepts used to identify persons with a specific health condition or characteristic. Establishing the validity of algorithms is a prerequisite for generating valid study findings that can ultimately inform evidence-based health care. In this paper, we aim to systematize terminology, methods, and practical considerations relevant to the conduct of validation studies of RWD-based algorithms. We discuss measures of algorithm accuracy, gold/reference standards, study size, prioritization of accuracy measures, algorithm portability, and implications for interpretation. Information bias is common in epidemiologic studies, underscoring the importance of transparency in decisions regarding choice and prioritizing measures of algorithm validity. The validity of an algorithm should be judged in the context of a data source, and one size does not fit all. Prioritizing validity measures within a given data source depends on the role of a given variable in the analysis (eligibility criterion, exposure, outcome, or covariate). Validation work should be part of routine maintenance of RWD sources. This article is part of a Special Collection on Pharmacoepidemiology.

EuMAR stakeholder engagement: an analysis of medically assisted reproduction (MAR) data collection practices in EU countries†.

STUDY QUESTION: What are the current national medically assisted reproduction (MAR) data collection systems across EU Member States, and how can these countries contribute to a unique, cycle-by-cycle registry for the European Monitoring of Medically Assisted Reproduction (EuMAR) project? SUMMARY ANSWER: The study identified significant variation in MAR data collection practices across Member States, with differences in data types, collection methods, and reporting requirements; the EuMAR project emerges as an opportunity to enhance data standardization and improve MAR data collection in the EU. WHAT IS KNOWN ALREADY: There is a need for new approaches in MAR data collection that include long-term and cross border follow-up. The EuMAR project intends to establish a unified, cycle-by-cycle registry of data on MAR treatments in EU countries, from which accurate cumulative outcomes can be calculated. STUDY DESIGN, SIZE, DURATION: This cross-sectional study involved a survey and interviews with stakeholders from 26 EU Member States conducted in 2023 over a period of seven months. PARTICIPANTS/MATERIALS, SETTING, METHODS: Representatives from national competent authorities and professional associations involved in MAR data collection in EU countries were invited to complete the survey and interviewed to assess current data flows, information requirements, and their interest in the EuMAR project. MAIN RESULTS AND THE ROLE OF CHANCE: Half of the participating countries reported having a national MAR registry with cycle-by-cycle data (n = 13), while 31% reported having a national registry with aggregated data (n = 8) and 19% reported having no national registry (n = 5). Of the countries with a national cycle-by-cycle registry, eight countries collect identifiable data, five countries collect pseudonymized data, and one country collects fully anonymized data. Informed consent is required in 10 countries. The main advantages that participants expected from a European registry like EuMAR were the possibility of obtaining national statistics in the absence of a national registry and improving the calculation of cumulative outcomes. LIMITATIONS, REASONS FOR CAUTION: The results of the study are based on self-reported data, which may be subject to bias, however, the validity of the collected information was verified with different means, including follow-up calls for clarifications and sharing final transcript reports. The feasibility of the proposed data flow models will be tested in a pilot study. WIDER IMPLICATIONS OF THE FINDINGS: Despite the heterogeneity of data collection practices across EU countries, the results show that stakeholders have high expectations of the benefits that the EuMAR registry can bring, namely the improvement of data consistency, cross-border comparability, and cumulative live birth rates, leading to better information for patients, health care providers and policy makers. STUDY FUNDING/COMPETING INTEREST(S): The EuMAR project was co-founded by ESHRE and the European Commission (101079865-EuMAR-EU4H-2021-PJ2). No competing interests were declared. TRIAL REGISTRATION NUMBER: N/A.

Approaches to protocol standardization and data harmonization in the ECHO-wide cohort study.

The United States (U.S.) National Institutes of Health-funded Environmental influences on Child Health Outcomes (ECHO)-wide Cohort was established to conduct high impact, transdisciplinary science to improve child health and development. The cohort is a collaborative research design in which both extant and new data are contributed by over 57,000 children across 69 cohorts. In this review article, we focus on two key challenging issues in the ECHO-wide Cohort: data collection standardization and data harmonization. Data standardization using a Common Data Model and derived analytical variables based on a team science approach should facilitate timely analyses and reduce errors due to data misuse. However, given the complexity of collaborative research designs, such as the ECHO-wide Cohort, dedicated time is needed for harmonization and derivation of analytic variables. These activities need to be done methodically and with transparency to enhance research reproducibility. IMPACT: Many collaborative research studies require data harmonization either prior to analyses or in the analyses of compiled data. The Environmental influences on Child Health Outcomes (ECHO) Cohort pools extant data with new data collection from over 57,000 children in 69 cohorts to conduct high-impact, transdisciplinary science to improve child health and development, and to provide a national database and biorepository for use by the scientific community at-large. We describe the tools, systems, and approaches we employed to facilitate harmonized data for impactful analyses of child health outcomes.

How do quantitative studies involving people with dementia report experiences of standardised data collection? A narrative synthesis of NIHR published studies.

BACKGROUND: People with dementia are routinely included as research participants in trials and other quantitative studies in which they are invited to respond to standardised measures. This paper reviews the reporting of standardised data collection from people with dementia in reports published in the National Institute for Health and Care Research (NIHR) Journals Library. The aim was to understand how the administration of standardised, self-report measures with people with dementia is reported in NIHR monographs and what could be learnt from this about the feasibility and acceptability of data collection approaches for future studies. METHODS: This was a systematic review with narrative synthesis. Broad search terms (Dementia OR Alzheimer*) were used to search the NIHR Journals Library website in December 2021. All studies that used (or intended to use) standardised measures to collect research data directly from people with dementia were eligible for inclusion. Information was extracted (where reported) on the process of data collection, dementia severity, levels of missing data and the experiences and reflections of those involved. RESULTS: Searches returned 42 records, from which 17 reports were assessed as eligible for inclusion, containing 22 studies. Response rates from participants with dementia in these studies varied considerably and appeared to be related to dementia severity and place of residence. Little information was reported on the process of data collection or the reasons for missing data, and most studies did not report the experiences of participants or those administering the measures. However, there was an indication from two studies that standardised data collection could provoke emotional distress in some participants with dementia. CONCLUSIONS: Through this review we identified both variation in levels of missing data and gaps in reporting which make it difficult to ascertain the reasons for this variation. We also identified potential risks to the well-being of participants with dementia which may be associated with the content of standardised measures and the context of data collection. Open reporting of and reflection upon data collection processes and the experiences of people involved is essential to ensure both the success of future data collection and the wellbeing of study participants. TRIAL REGISTRATION: Registered with Research on Research https://ror-hub.org/study/2905/ .

Recommendations for data collection in cohort studies of preterm born individuals - The RECAP Preterm Core Dataset.

BACKGROUND: Preterm birth (before 37 completed weeks of gestation) is associated with an increased risk of adverse health and developmental outcomes relative to birth at term. Existing guidelines for data collection in cohort studies of individuals born preterm are either limited in scope, have not been developed using formal consensus methodology, or did not involve a range of stakeholders in their development. Recommendations meeting these criteria would facilitate data pooling and harmonisation across studies. OBJECTIVES: To develop a Core Dataset for use in longitudinal cohort studies of individuals born preterm. METHODS: This work was carried out as part of the RECAP Preterm project. A systematic review of variables included in existing core outcome sets was combined with a scoping exercise conducted with experts on preterm birth. The results were used to generate a draft core dataset. A modified Delphi process was implemented using two stages with three rounds each. Three stakeholder groups participated: RECAP Preterm project partners; external experts in the field; people with lived experience of preterm birth. The Delphi used a 9-point Likert scale. Higher values indicated greater importance for inclusion. Participants also suggested additional variables they considered important for inclusion which were voted on in later rounds. RESULTS: An initial list of 140 data items was generated. Ninety-six participants across 22 countries participated in the Delphi, of which 29% were individuals with lived experience of preterm birth. Consensus was reached on 160 data items covering Antenatal and Birth Information, Neonatal Care, Mortality, Administrative Information, Organisational Level Information, Socio-economic and Demographic information, Physical Health, Education and Learning, Neurodevelopmental Outcomes, Social, Lifestyle and Leisure, Healthcare Utilisation and Quality of Life. CONCLUSIONS: This core dataset includes 160 data items covering antenatal care through outcomes in adulthood. Its use will guide data collection in new studies and facilitate pooling and harmonisation of existing data internationally.

Describing diversity of real world data sources in pharmacoepidemiologic studies: The DIVERSE scoping review.

PURPOSE: Real-world evidence (RWE) is increasingly used for medical regulatory decisions, yet concerns persist regarding its reproducibility and hence validity. This study addresses reproducibility challenges associated with diversity across real-world data sources (RWDS) repurposed for secondary use in pharmacoepidemiologic studies. Our aims were to identify, describe and characterize practices, recommendations and tools for collecting and reporting diversity across RWDSs, and explore how leveraging diversity could improve the quality of evidence. METHODS: In a preliminary phase, keywords for a literature search and selection tool were designed using a set of documents considered to be key by the coauthors. Next, a systematic search was conducted up to December 2021. The resulting documents were screened based on titles and abstracts, then based on full texts using the selection tool. Selected documents were reviewed to extract information on topics related to collecting and reporting RWDS diversity. A content analysis of the topics identified explicit and latent themes. RESULTS: Across the 91 selected documents, 12 topics were identified: 9 dimensions used to describe RWDS (organization accessing the data source, data originator, prompt, inclusion of population, content, data dictionary, time span, healthcare system and culture, and data quality), tools to summarize such dimensions, challenges, and opportunities arising from diversity. Thirty-six themes were identified within the dimensions. Opportunities arising from data diversity included multiple imputation and standardization. CONCLUSIONS: The dimensions identified across a large number of publications lay the foundation for formal guidance on reporting diversity of data sources to facilitate interpretation and enhance replicability and validity of RWE.

Metadata for Data dIscoverability aNd Study rEplicability in obseRVAtional Studies (MINERVA): Development and Pilot of a Metadata List and Catalogue in Europe.

PURPOSE: Metadata for data dIscoverability aNd study rEplicability in obseRVAtional studies (MINERVA), a European Medicines Agency-funded project (EUPAS39322), defined a set of metadata to describe real-world data sources (RWDSs) and piloted metadata collection in a prototype catalogue to assist investigators from data source discoverability through study conduct. METHODS: A list of metadata was created from a review of existing metadata catalogues and recommendations, structured interviews, a stakeholder survey, and a technical workshop. The prototype was designed to comply with the FAIR principles (findable, accessible, interoperable, reusable), using MOLGENIS software. Metadata collection was piloted by 15 data access partners (DAPs) from across Europe. RESULTS: A total of 442 metadata variables were defined in six domains: institutions (organizations connected to a data source); data banks (data collections sustained by an organization); data sources (collections of linkable data banks covering a common underlying population); studies; networks (of institutions); and common data models (CDMs). A total of 26 institutions were recorded in the prototype. Each DAP populated the metadata of one data source and its selected data banks. The number of data banks varied by data source; the most common data banks were hospital administrative records and pharmacy dispensation records (10 data sources each). Quantitative metadata were successfully extracted from three data sources conforming to different CDMs and entered into the prototype. CONCLUSIONS: A metadata list was finalized, a prototype was successfully populated, and a good practice guide was developed. Setting up and maintaining a metadata catalogue on RWDSs will require substantial effort to support discoverability of data sources and reproducibility of studies in Europe.

Replication of a local record keeping method for collecting road crash data in low resource settings: lessons from Bangladesh and Nepal.

BACKGROUND: Police road crash and injury data in low-income and middle-income countries are known to under-report crashes, fatalities and injuries, especially for vulnerable road users. Local record keepers, who are members of the public, can be engaged to provide an additional source of crash and injury data. METHODS: This paper compares the application of a local record keeper method to capture road crash and injury data in Bangladesh and Nepal, assesses the quality of the data collected and evaluates the replicability and value of the methodology using a framework developed to evaluate the impact of being a local record keeper. OUTCOME: Application in research studies in both Bangladesh and Nepal found the local record keeper methodology provided high-quality and complete data compared with local police records. The methodology was flexible enough to adapt to project and context differences. The evaluation framework enabled the identification of the challenges and unexpected benefits realised in each study. This led to the development of an 11-step process for conducting road crash data collection using local record keepers, which is presented to facilitate replication in other settings. CONCLUSION: Data collected by local record keepers are a flexible and replicable method to understand the strengths and limitations of existing police data, adding to the evidence base and informing local and national decision-making. The method may create additional benefits for data collectors and communities, help design and assess road safety interventions and support advocacy for improved routine police data.

Collecting routine and timely cancer stage at diagnosis by implementing a cancer staging tiered framework: the Western Australian Cancer Registry experience.

BACKGROUND: Current processes collecting cancer stage data in population-based cancer registries (PBCRs) lack standardisation, resulting in difficulty utilising diverse data sources and incomplete, low-quality data. Implementing a cancer staging tiered framework aims to improve stage collection and facilitate inter-PBCR benchmarking. OBJECTIVE: Demonstrate the application of a cancer staging tiered framework in the Western Australian Cancer Staging Project to establish a standardised method for collecting cancer stage at diagnosis data in PBCRs. METHODS: The tiered framework, developed in collaboration with a Project Advisory Group and applied to breast, colorectal, and melanoma cancers, provides business rules - procedures for stage collection. Tier 1 represents the highest staging level, involving complete American Joint Committee on Cancer (AJCC) tumour-node-metastasis (TNM) data collection and other critical staging information. Tier 2 (registry-derived stage) relies on supplementary data, including hospital admission data, to make assumptions based on data availability. Tier 3 (pathology stage) solely uses pathology reports. FINDINGS: The tiered framework promotes flexible utilisation of staging data, recognising various levels of data completeness. Tier 1 is suitable for all purposes, including clinical and epidemiological applications. Tiers 2 and 3 are recommended for epidemiological analysis alone. Lower tiers provide valuable insights into disease patterns, risk factors, and overall disease burden for public health planning and policy decisions. Capture of staging at each tier depends on data availability, with potential shifts to higher tiers as new data sources are acquired. CONCLUSIONS: The tiered framework offers a dynamic approach for PBCRs to record stage at diagnosis, promoting consistency in population-level staging data and enabling practical use for benchmarking across jurisdictions, public health planning, policy development, epidemiological analyses, and assessing cancer outcomes. Evolution with staging classifications and data variable changes will futureproof the tiered framework. Its adaptability fosters continuous refinement of data collection processes and encourages improvements in data quality.

[Analysis Of Data Quality/Completeness In Covid-19 Cases: Why A Digital Integrated Data Collection Is Also Necessary For Pandemic Control]. / Analyse zur Datenqualität/-vollständigkeit bei COVID-19-Fällen: Warum eine digitale integrierte Datenerfassung auch zur Pandemiebekämpfung notwendig ist.

BACKGROUND: Epidemiological data on the corona pandemic collected in the public health sector in Germany have been less useful in estimating vaccine effectiveness and clinical outcomes compared to other countries. METHODS: In this retrospective observational study, we examined the completeness of selected own data collected during the pandemic. Information on the important parameters of hospitalization, vaccination status and risk factors for severe course and death over different periods were considered and evaluated descriptively. The data are discussed in the extended context of required digital strategies in Germany. RESULTS: From January 1, 2022 to June 30, 2022, we found 126,920 administrative procedures related to COVID-19. With regard to the data on hospitalization, in 19,749 cases, it was stated "No", in 1,990 cases "Yes" and in 105,181 cases (83+%) "Not collected" or "Not ascertainable". Concerning vaccinations, only a small proportion of procedures contained information on the type of vaccine (11.1+%), number of vaccinations (4.4+%) and date of the last vaccination (2.1+%). The completeness of data on chronic conditions/risk factors in COVID-19-related deaths decreased over four consecutive periods between 2020 and 2022 as case numbers increased. CONCLUSION: Future strategies taking into account meaningfulness and completeness of data must comprise modern technical solutions with digital data collection on infections without putting the principle of data protection at risk.

Errata: Rare Compound Heterozygous Missense Mutation of the SCN5A Gene with Childhood-Onset Sick Sinus Syndrome in Two Chinese Sisters: A Case Report.

An error appeared in the article entitled "Rare Compound Heterozygous Missense Mutation of the SCN5A Gene with Childhood-Onset Sick Sinus Syndrome in Two Chinese Sisters: A Case Report" by Yanyun Wang, Siyu Long, Chenxi Wei, and Xiaoqin Wang (Vol. 64 No.2, 299-305, 2023). The name of the first affiliation on page 299 was wrong. It should be "Laboratory of Molecular Translational Medicine, Center for Translational Medicine, West China Second University Hospital, Sichuan University, Chengdu, China" and not "Laboratory of Molecular Translational Medicine, Center for Translational Medicine, Sichuan University, Chengdu, China".

Development and Utility of an Interactive Online Dashboard for Monitoring Data Collection and Data Processing During a School-Based Health Survey.

We built an interactive online dashboard using Google Looker Studio to monitor data collection and data processing activities during the Adolescent Health Survey (AHS) 2022, a large-scale nationwide survey conducted among school-going adolescents in Malaysia. Through user testing and training, refinements were made to the initial dashboard, resulting in a more streamlined and concise dashboard design. The dashboard comprised 2 pages that provided key metrics on the progress of data collection and data processing, respectively. The introduction of the dashboard enhanced the quality and ease of weekly progress reporting during meetings of the survey's central coordinating team, while its drill-down and filtering functionalities helped us detect arising issues early and supported collaborative problem-solving. Research teams coordinating comparable school-based health surveys are invited to duplicate the dashboard using Looker Studio's built-in "Make a copy" function and customize it further based on their country- or survey-specific requirements.

Maximising the Quality of Stroke Care: Reporting of Data Collection Methods and Resourcing in National Stroke Registries: A Systematic Review.

Stroke registries are tools for improving care and advancing research. We aim to describe the methodology and resourcing of existing national stroke registries. We conducted a systematic search of the published, peer-reviewed literature and grey literature examining descriptions of data collection methods and resourcing of national stroke registries published from 2012 to 2023. The systematic review was registered in PROSPERO (CRD42023393841). 101 records relating to 21 registries in 19 countries were identified. They universally employed web-based platforms for data collection. The principal profession of data collectors was nursing. All included the acute phase of care, 28% (6) registered the pre-hospital (ambulance) phase and 14% (3) included rehabilitation. 80% (17) collected outcome data. The registries varied in their approach to outcome data collection; in 9 registries it was collected by hospitals, in 2 it was collected by the registry, and 7 had linkage to national administrative databases allowing follow-up of a limited number of end points. Coverage of the total number of strokes varies from 6 to 95%. Despite widespread use of Electronic Health Records (EHRs) the ability to automatically populate variables remained limited. Governance and management structures are diverse, making it challenging to compare their resourcing. Data collection for clinical registries requires time and necessary skills and imposes a significant administrative burden on the professionals entering data. We highlight the role of clinical registries as powerful instruments for quality improvement. Future work should involve creating a central repository of stroke registries to enable the development of new registries and facilitate international collaboration.

Clinical Genetics Lacks Standard Definitions and Protocols for the Collection and Use of Diversity Measures.

Genetics researchers and clinical professionals rely on diversity measures such as race, ethnicity, and ancestry (REA) to stratify study participants and patients for a variety of applications in research and precision medicine. However, there are no comprehensive, widely accepted standards or guidelines for collecting and using such data in clinical genetics practice. Two NIH-funded research consortia, the Clinical Genome Resource (ClinGen) and Clinical Sequencing Evidence-generating Research (CSER), have partnered to address this issue and report how REA are currently collected, conceptualized, and used. Surveying clinical genetics professionals and researchers (n = 448), we found heterogeneity in the way REA are perceived, defined, and measured, with variation in the perceived importance of REA in both clinical and research settings. The majority of respondents (>55%) felt that REA are at least somewhat important for clinical variant interpretation, ordering genetic tests, and communicating results to patients. However, there was no consensus on the relevance of REA, including how each of these measures should be used in different scenarios and what information they can convey in the context of human genetics. A lack of common definitions and applications of REA across the precision medicine pipeline may contribute to inconsistencies in data collection, missing or inaccurate classifications, and misleading or inconclusive results. Thus, our findings support the need for standardization and harmonization of REA data collection and use in clinical genetics and precision health research.

Ensuring the quality and specificity of preregistrations.

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.