Guidance of development, validation, and evaluation of algorithms for populating health status in observational studies of routinely collected data (DEVELOP-RCD).

BACKGROUND: In recent years, there has been a growing trend in the utilization of observational studies that make use of routinely collected healthcare data (RCD). These studies rely on algorithms to identify specific health conditions (e.g. diabetes or sepsis) for statistical analyses. However, there has been substantial variation in the algorithm development and validation, leading to frequently suboptimal performance and posing a significant threat to the validity of study findings. Unfortunately, these issues are often overlooked. METHODS: We systematically developed guidance for the development, validation, and evaluation of algorithms designed to identify health status (DEVELOP-RCD). Our initial efforts involved conducting both a narrative review and a systematic review of published studies on the concepts and methodological issues related to algorithm development, validation, and evaluation. Subsequently, we conducted an empirical study on an algorithm for identifying sepsis. Based on these findings, we formulated specific workflow and recommendations for algorithm development, validation, and evaluation within the guidance. Finally, the guidance underwent independent review by a panel of 20 external experts who then convened a consensus meeting to finalize it. RESULTS: A standardized workflow for algorithm development, validation, and evaluation was established. Guided by specific health status considerations, the workflow comprises four integrated steps: assessing an existing algorithm's suitability for the target health status; developing a new algorithm using recommended methods; validating the algorithm using prescribed performance measures; and evaluating the impact of the algorithm on study results. Additionally, 13 good practice recommendations were formulated with detailed explanations. Furthermore, a practical study on sepsis identification was included to demonstrate the application of this guidance. CONCLUSIONS: The establishment of guidance is intended to aid researchers and clinicians in the appropriate and accurate development and application of algorithms for identifying health status from RCD. This guidance has the potential to enhance the credibility of findings from observational studies involving RCD.

Metadata for Data dIscoverability aNd Study rEplicability in obseRVAtional Studies (MINERVA): Development and Pilot of a Metadata List and Catalogue in Europe.

PURPOSE: Metadata for data dIscoverability aNd study rEplicability in obseRVAtional studies (MINERVA), a European Medicines Agency-funded project (EUPAS39322), defined a set of metadata to describe real-world data sources (RWDSs) and piloted metadata collection in a prototype catalogue to assist investigators from data source discoverability through study conduct. METHODS: A list of metadata was created from a review of existing metadata catalogues and recommendations, structured interviews, a stakeholder survey, and a technical workshop. The prototype was designed to comply with the FAIR principles (findable, accessible, interoperable, reusable), using MOLGENIS software. Metadata collection was piloted by 15 data access partners (DAPs) from across Europe. RESULTS: A total of 442 metadata variables were defined in six domains: institutions (organizations connected to a data source); data banks (data collections sustained by an organization); data sources (collections of linkable data banks covering a common underlying population); studies; networks (of institutions); and common data models (CDMs). A total of 26 institutions were recorded in the prototype. Each DAP populated the metadata of one data source and its selected data banks. The number of data banks varied by data source; the most common data banks were hospital administrative records and pharmacy dispensation records (10 data sources each). Quantitative metadata were successfully extracted from three data sources conforming to different CDMs and entered into the prototype. CONCLUSIONS: A metadata list was finalized, a prototype was successfully populated, and a good practice guide was developed. Setting up and maintaining a metadata catalogue on RWDSs will require substantial effort to support discoverability of data sources and reproducibility of studies in Europe.

A Framework to Support the Standardized Management of Real-World Data.

Relying on our experience on the development of data registration and management systems for clinical and biological data coming from patients with hematological malignancies, as well as on the design of strategies for data collection and analysis to support multi-center, clinical association studies, we designed a framework for the standardized collection and transformation of clinically relevant real-world data into evidence, to meet the challenges of gathering biomedical data collected during daily clinical practice in order to promote basic and clinical research.

Evaluation of software technical quality for collecting data from patients under palliative care.

OBJECTIVES: to evaluate software technical quality for collecting data from patients under palliative care. METHODS: this is methodological technology evaluation research, according to the technical standard International Organization for Standardization/International Electrotechnical Commission 25040-2011, developed from August 2021 to August 2023. Eight nurses and eight information technology professionals participated as judges, who evaluated six quality characteristics and 23 subcharacteristics. Items that reached a percentage of agreement greater than 70% were considered suitable. RESULTS: the characteristics evaluated by nurses/information technology professionals received the following percentages of agreement, respectively: functional suitability (94%-84%); reliability (100-70%); usability (89.9-66.8%); performance efficiency (95.8%-86.1%); compatibility (95.8-79.6%); and safety (96%-83.4%). CONCLUSIONS: the software was considered suitable in quality evaluation to offer support to nurses in collecting patient data under palliative care, with the potential to operationalize the first Nursing Process stage.

Collecting routine and timely cancer stage at diagnosis by implementing a cancer staging tiered framework: the Western Australian Cancer Registry experience.

BACKGROUND: Current processes collecting cancer stage data in population-based cancer registries (PBCRs) lack standardisation, resulting in difficulty utilising diverse data sources and incomplete, low-quality data. Implementing a cancer staging tiered framework aims to improve stage collection and facilitate inter-PBCR benchmarking. OBJECTIVE: Demonstrate the application of a cancer staging tiered framework in the Western Australian Cancer Staging Project to establish a standardised method for collecting cancer stage at diagnosis data in PBCRs. METHODS: The tiered framework, developed in collaboration with a Project Advisory Group and applied to breast, colorectal, and melanoma cancers, provides business rules - procedures for stage collection. Tier 1 represents the highest staging level, involving complete American Joint Committee on Cancer (AJCC) tumour-node-metastasis (TNM) data collection and other critical staging information. Tier 2 (registry-derived stage) relies on supplementary data, including hospital admission data, to make assumptions based on data availability. Tier 3 (pathology stage) solely uses pathology reports. FINDINGS: The tiered framework promotes flexible utilisation of staging data, recognising various levels of data completeness. Tier 1 is suitable for all purposes, including clinical and epidemiological applications. Tiers 2 and 3 are recommended for epidemiological analysis alone. Lower tiers provide valuable insights into disease patterns, risk factors, and overall disease burden for public health planning and policy decisions. Capture of staging at each tier depends on data availability, with potential shifts to higher tiers as new data sources are acquired. CONCLUSIONS: The tiered framework offers a dynamic approach for PBCRs to record stage at diagnosis, promoting consistency in population-level staging data and enabling practical use for benchmarking across jurisdictions, public health planning, policy development, epidemiological analyses, and assessing cancer outcomes. Evolution with staging classifications and data variable changes will futureproof the tiered framework. Its adaptability fosters continuous refinement of data collection processes and encourages improvements in data quality.

Development and Utility of an Interactive Online Dashboard for Monitoring Data Collection and Data Processing During a School-Based Health Survey.

We built an interactive online dashboard using Google Looker Studio to monitor data collection and data processing activities during the Adolescent Health Survey (AHS) 2022, a large-scale nationwide survey conducted among school-going adolescents in Malaysia. Through user testing and training, refinements were made to the initial dashboard, resulting in a more streamlined and concise dashboard design. The dashboard comprised 2 pages that provided key metrics on the progress of data collection and data processing, respectively. The introduction of the dashboard enhanced the quality and ease of weekly progress reporting during meetings of the survey's central coordinating team, while its drill-down and filtering functionalities helped us detect arising issues early and supported collaborative problem-solving. Research teams coordinating comparable school-based health surveys are invited to duplicate the dashboard using Looker Studio's built-in "Make a copy" function and customize it further based on their country- or survey-specific requirements.

Improving quality of care by standardising patient data collection in electronic medical records in an oncology department in Spain.

BACKGROUND: Evaluation of quality of care in oncology is key in ensuring patients receive adequate treatment. American Society of Clinical Oncology's (ASCO) Quality Oncology Practice Initiative (QOPI) Certification Program (QCP) is an international initiative that evaluates quality of care in outpatient oncology practices. METHODS: We retrospectively reviewed free-text electronic medical records from patients with breast cancer (BR), colorectal cancer (CRC) or non-small cell lung cancer (NSCLC). In a baseline measurement, high scores were obtained for the nine disease-specific measures of QCP Track (2021 version had 26 measures); thus, they were not further analysed. We evaluated two sets of measures: the remaining 17 QCP Track measures, as well as these plus other 17 measures selected by us (combined measures). Review of data from 58 patients (26 BR; 18 CRC; 14 NSCLC) seen in June 2021 revealed low overall quality scores (OQS)-below ASCO's 75% threshold-for QCP Track measures (46%) and combined measures (58%). We developed a plan to improve OQS and monitored the impact of the intervention by abstracting data at subsequent time points. RESULTS: We evaluated potential causes for the low OQS and developed a plan to improve it over time by educating oncologists at our hospital on the importance of improving collection of measures and highlighting the goal of applying for QOPI certification. We conducted seven plan-do-study-act cycles and evaluated the scores at seven subsequent data abstraction time points from November 2021 to December 2022, reviewing 404 patients (199 BR; 114 CRC; 91 NSCLC). All measures were improved. Four months after the intervention, OQS surpassed the quality threshold and was maintained for 10 months until the end of the study (range, 78-87% for QCP Track measures; 78-86% for combined measures). CONCLUSIONS: We developed an easy-to-implement intervention that achieved a fast improvement in OQS, enabling our Medical Oncology Department to aim for QOPI certification.

Protocol for the Evaluation of Data-Related Processes and Challenges at Level 1-4 Trauma Centers in Arkansas: A Mixed-Methods Case Study.

Background: In the following manuscript, we describe the detailed protocol for a mixed-methods, observational case study conducted to identify and evaluate existing data-related processes and challenges currently faced by trauma centers in a rural state. The data will be utilized to assess the impact of these challenges on registry data collection. Methods: The study relies on a series of interviews and observations to collect data from trauma registry staff at level 1-4 trauma centers across the state of Arkansas. A think-aloud protocol will be used to facilitate observations to gather keystroke-level modeling data and insight into site processes and workflows for collecting and submitting data to the Arkansas Trauma Registry. Informal, semi-structured interviews will follow the observation period to assess the participant's perspective on current processes, potential barriers to data collection or submission to the registry, and recommendations for improvement. Each session will be recorded, and de-identified transcripts and session notes will be used for analysis. Keystroke level modeling data derived from observations will be extracted and analyzed quantitatively to determine time spent performing end-to-end registry-related activities. Qualitative data from interviews will be reviewed and coded by 2 independent reviewers following a thematic analysis methodology. Each set of codes will then be adjudicated by the reviewers using a consensus-driven approach to extrapolate the final set of themes. Discussion: We will utilize a mixed methods approach to understand existing processes and barriers to data collection for the Arkansas Trauma Registry. Anticipated results will provide a baseline measure of the data collection and submission processes at various trauma centers across the state. We aim to assess strengths and limitations of existing processes and identify existing barriers to interoperability. These results will provide first-hand knowledge on existing practices for the trauma registry use case and will provide quantifiable data that can be utilized in future research to measure outcomes of future process improvement efforts. The potential implications of this study can form the basis for identifying potential solutions for streamlining data collection, exchange, and utilization of trauma registry data for clinical practice, public health, and clinical and translational research.

Best Practices and Recommendations for Research Using Virtual Real-Time Data Collection: Protocol for Virtual Data Collection Studies.

BACKGROUND: The COVID-19 pandemic and the subsequent need for social distancing required the immediate pivoting of research modalities. Research that had previously been conducted in person had to pivot to remote data collection. Researchers had to develop data collection protocols that could be conducted remotely with limited or no evidence to guide the process. Therefore, the use of web-based platforms to conduct real-time research visits surged despite the lack of evidence backing these novel approaches. OBJECTIVE: This paper aims to review the remote or virtual research protocols that have been used in the past 10 years, gather existing best practices, and propose recommendations for continuing to use virtual real-time methods when appropriate. METHODS: Articles (n=22) published from 2013 to June 2023 were reviewed and analyzed to understand how researchers conducted virtual research that implemented real-time protocols. "Real-time" was defined as data collection with a participant through a live medium where a participant and research staff could talk to each other back and forth in the moment. We excluded studies for the following reasons: (1) studies that collected participant or patient measures for the sole purpose of engaging in a clinical encounter; (2) studies that solely conducted qualitative interview data collection; (3) studies that conducted virtual data collection such as surveys or self-report measures that had no interaction with research staff; (4) studies that described research interventions but did not involve the collection of data through a web-based platform; (5) studies that were reviews or not original research; (6) studies that described research protocols and did not include actual data collection; and (7) studies that did not collect data in real time, focused on telehealth or telemedicine, and were exclusively intended for medical and not research purposes. RESULTS: Findings from studies conducted both before and during the COVID-19 pandemic suggest that many types of data can be collected virtually in real time. Results and best practice recommendations from the current protocol review will be used in the design and implementation of a substudy to provide more evidence for virtual real-time data collection over the next year. CONCLUSIONS: Our findings suggest that virtual real-time visits are doable across a range of participant populations and can answer a range of research questions. Recommended best practices for virtual real-time data collection include (1) providing adequate equipment for real-time data collection, (2) creating protocols and materials for research staff to facilitate or guide participants through data collection, (3) piloting data collection, (4) iteratively accepting feedback, and (5) providing instructions in multiple forms. The implementation of these best practices and recommendations for future research are further discussed in the paper. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/53790.

Standardizing data collection in adjuvant colon cancer trials: A consensus project from the IDEA and ACCENT international consortia and national experts.

BACKGROUND: Despite contributions provided by the recent clinical trials, several issues and challenges still remain unsolved in adjuvant colon cancer (CC). Hence, further studies should be planned to better refine risk assessment as well as to establish the optimal treatment strategy in the adjuvant setting. However, it is necessary to request adequate, contemporary and relevant variables and report them homogeneously in order to bring maximal information when analyzing their prognostic value. MATERIAL AND METHODS: The project was devised to gain a consensus from experts engaged in the planning, accrual and analyses of stage II and III CC clinical trials, to identify mandatory and recommended baseline variables in order to i) harmonize future data collection worldwide in clinical trials dedicated to adjuvant treatment of CC; ii) propose guidance for Case Report Forms to be used for clinical trials in this setting. A total of 72 questions related to variables that should be reported and how to report them in adjuvant clinical trials were approved and then voted to reach a final consensus from panelists. RESULTS: Data items on patient-related factors, histopathological features, molecular profile, circulating biomarkers and blood analyses were analyzed and discussed by the whole expert panel. For each item, we report data supporting the acquired consensus and the relevant issues that were discussed. Nineteen items were deemed to be mandatory for resected stage III patients and 24 for resected stage II disease. In addition, 9 and 4 items were judged as recommended for stage III and II, respectively. CONCLUSION: In our opinion, these 28 variables should be used and uniformly reported in more comprehensive CRFs as research groups design future clinical trials in the field of adjuvant colon cancer.

Describing diversity of real world data sources in pharmacoepidemiologic studies: The DIVERSE scoping review.

PURPOSE: Real-world evidence (RWE) is increasingly used for medical regulatory decisions, yet concerns persist regarding its reproducibility and hence validity. This study addresses reproducibility challenges associated with diversity across real-world data sources (RWDS) repurposed for secondary use in pharmacoepidemiologic studies. Our aims were to identify, describe and characterize practices, recommendations and tools for collecting and reporting diversity across RWDSs, and explore how leveraging diversity could improve the quality of evidence. METHODS: In a preliminary phase, keywords for a literature search and selection tool were designed using a set of documents considered to be key by the coauthors. Next, a systematic search was conducted up to December 2021. The resulting documents were screened based on titles and abstracts, then based on full texts using the selection tool. Selected documents were reviewed to extract information on topics related to collecting and reporting RWDS diversity. A content analysis of the topics identified explicit and latent themes. RESULTS: Across the 91 selected documents, 12 topics were identified: 9 dimensions used to describe RWDS (organization accessing the data source, data originator, prompt, inclusion of population, content, data dictionary, time span, healthcare system and culture, and data quality), tools to summarize such dimensions, challenges, and opportunities arising from diversity. Thirty-six themes were identified within the dimensions. Opportunities arising from data diversity included multiple imputation and standardization. CONCLUSIONS: The dimensions identified across a large number of publications lay the foundation for formal guidance on reporting diversity of data sources to facilitate interpretation and enhance replicability and validity of RWE.

Developing evaluation measures for health departments sharing and linking HIV and STI surveillance data.

Enhancing data sharing, quality, and use across siloed HIV and STI programs is critical for national and global initiatives to reduce new HIV infections and improve the health of people with HIV. As part of the Enhancing Linkage of STI and HIV Surveillance Data in the Ryan White HIV/AIDS Program initiative, four health departments (HDs) in the U.S. received technical assistance to better share and link their HIV and STI surveillance data. The process used to develop evaluation measures assessing implementation and outcomes of linking HIV and STI data systems involved six steps: 1) measure selection and development, 2) review and refinement, 3) testing, 4) implementation and data collection, 5) data quality review and feedback, and 6) dissemination. Findings from pilot testing warranted slight adaptations, including starting with a core set of measures and progressively scaling up. Early findings showed improvements in data quality over time. Lessons learned included identifying and engaging key stakeholders early; developing resources to assist HDs; and considering measure development as iterative processes requiring periodic review and reassessment to ensure continued utility. These findings can guide programs and evaluations, especially those linking data across multiple systems, in developing measures to track implementation and outcomes over time.

[Analysis Of Data Quality/Completeness In Covid-19 Cases: Why A Digital Integrated Data Collection Is Also Necessary For Pandemic Control]. / Analyse zur Datenqualität/-vollständigkeit bei COVID-19-Fällen: Warum eine digitale integrierte Datenerfassung auch zur Pandemiebekämpfung notwendig ist.

BACKGROUND: Epidemiological data on the corona pandemic collected in the public health sector in Germany have been less useful in estimating vaccine effectiveness and clinical outcomes compared to other countries. METHODS: In this retrospective observational study, we examined the completeness of selected own data collected during the pandemic. Information on the important parameters of hospitalization, vaccination status and risk factors for severe course and death over different periods were considered and evaluated descriptively. The data are discussed in the extended context of required digital strategies in Germany. RESULTS: From January 1, 2022 to June 30, 2022, we found 126,920 administrative procedures related to COVID-19. With regard to the data on hospitalization, in 19,749 cases, it was stated "No", in 1,990 cases "Yes" and in 105,181 cases (83+%) "Not collected" or "Not ascertainable". Concerning vaccinations, only a small proportion of procedures contained information on the type of vaccine (11.1+%), number of vaccinations (4.4+%) and date of the last vaccination (2.1+%). The completeness of data on chronic conditions/risk factors in COVID-19-related deaths decreased over four consecutive periods between 2020 and 2022 as case numbers increased. CONCLUSION: Future strategies taking into account meaningfulness and completeness of data must comprise modern technical solutions with digital data collection on infections without putting the principle of data protection at risk.