Validation of a machine learning approach to estimate Clinical Disease Activity Index Scores for rheumatoid arthritis.

OBJECTIVE: Disease activity measures, such as the Clinical Disease Activity Index (CDAI), are important tools for informing treatment decisions and monitoring patient outcomes in rheumatoid arthritis (RA). Yet, documentation of CDAI scores in electronic medical records and other real-world data sources is inconsistent, making it challenging to use these data for research. The purpose of this study was to validate a machine learning model to estimate CDAI scores for patients with RA using clinical notes. METHODS: A machine learning model was developed to estimate CDAI score values using clinical notes from a specific rheumatology visit. Data from the OM1 RA Registry were used to create a training cohort of 56 177 encounters and a separate validation cohort of 18 726 encounters, 11 985 of which passed a model-derived confidence filter; all included encounters had both a clinician-recorded CDAI score and a clinical note. Model performance was assessed using the area under the receiver operating characteristic curve (AUC), positive predictive value (PPV) and negative predictive value (NPV), calculated using a binarised version of the outcome. The Spearman's R and Pearson's R values were also calculated. RESULTS: The model had a PPV of 0.80, NPV of 0.84 and AUC of 0.88 when evaluating performance using the binarised version of the outcome. The model had a Spearman's R value of 0.72 and a Pearson's R value of 0.69 when evaluating performance using the continuous CDAI numeric scores. CONCLUSION: A machine learning model estimates CDAI scores from clinical notes with good performance. Application of the model to real-world data sets may allow estimated CDAI scores to be used for research purposes.

Harmonized Outcome Measures for Use in Non-Small Cell Lung Cancer Patient Registries and Clinical Practice.

BACKGROUND: Lung cancer is the leading cause of cancer-related death in the United States and globally, and many questions exist about treatment options. Harmonizing data across registries and other data collection efforts would yield a robust data infrastructure to help address many research questions. The purpose of this project was to develop a minimum set of patient and clinician relevant harmonized outcome measures that can be collected in non-small cell lung cancer (NSCLC) patient registries and clinical practice. METHODS: Seventeen lung cancer registries and related efforts were identified and invited to submit outcome measures. Representatives from medical specialty societies, government agencies, health systems, health information technology groups, patient advocacy organizations, and industry formed a stakeholder panel to categorize the measures and harmonize definitions using the Agency for Healthcare Research and Quality's supported Outcome Measures Framework (OMF). RESULTS: The panel reviewed 66 outcome measures and identified a minimum set of 8 broadly relevant measures in the OMF categories of patient survival, clinical response, events of interest, and resource utilization. The panel harmonized definitions for the 8 measures through in-person and virtual meetings. The panel did not reach consensus on 1 specific validated instrument for capturing patient-reported outcomes. The minimum set of harmonized outcome measures is broadly relevant to clinicians and patients and feasible to capture across NSCLC disease stages and treatment pathways. A pilot test of these measures would be useful to document the burden and value of the measures for research and in clinical practice. CONCLUSIONS: By collecting the harmonized measures consistently, registries and other data collection systems could contribute to the development research infrastructure and learning health systems to support new research and improve patient outcomes.

Validation of a machine learning approach to estimate Systemic Lupus Erythematosus Disease Activity Index score categories and application in a real-world dataset.

OBJECTIVE: Use of the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) in routine clinical practice is inconsistent, and availability of clinician-recorded SLEDAI scores in real-world datasets is limited. This study aimed to validate a machine learning model to estimate SLEDAI score categories using clinical notes and to apply the model to a large, real-world dataset to generate estimated score categories for use in future research studies. METHODS: A machine learning model was developed to estimate an individual patient's SLEDAI score category (no activity, mild activity, moderate activity or high/very high activity) for a specific encounter date using clinical notes. A training cohort of 3504 encounters and a separate validation cohort of 1576 encounters were created from the OM1 SLE Registry. Model performance was assessed using the area under the receiver operating characteristic curve (AUC), calculated using a binarised version of the outcome that sets the positive class to be those records with clinician-recorded SLEDAI scores >5 and the negative class to be records with scores ≤5. Model performance was evaluated by categorising the scores into the four disease activity categories and by calculating the Spearman's R value and Pearson's R value. RESULTS: The AUC for the two categories was 0.93 for the development cohort and 0.91 for the validation cohort. The model had a Spearman's R value of 0.7 and a Pearson's R value of 0.7 when calculated using the four disease activity categories. CONCLUSION: The model performs well when estimating SLEDAI score categories using unstructured clinical notes.

Harmonized outcome measures for use in degenerative lumbar spondylolisthesis patient registries and clinical practice.

OBJECTIVE: The development of new treatment approaches for degenerative lumbar spondylolisthesis (DLS) has introduced many questions about comparative effectiveness and long-term outcomes. Patient registries collect robust, longitudinal data that could be combined or aggregated to form a national and potentially international research data infrastructure to address these and other research questions. However, linking data across registries is challenging because registries typically define and capture different outcome measures. Variation in outcome measures occurs in clinical practice and other types of research studies as well, limiting the utility of existing data sources for addressing new research questions. The purpose of this project was to develop a minimum set of patient- and clinician-relevant standardized outcome measures that are feasible for collection in DLS registries and clinical practice. METHODS: Nineteen DLS registries, observational studies, and quality improvement efforts were invited to participate and submit outcome measures. A stakeholder panel was organized that included representatives from medical specialty societies, health systems, government agencies, payers, industries, health information technology organizations, and patient advocacy groups. The panel categorized the measures using the Agency for Healthcare Research and Quality's Outcome Measures Framework (OMF), identified a minimum set of outcome measures, and developed standardized definitions through a consensus-based process. RESULTS: The panel identified and harmonized 57 outcome measures into a minimum set of 10 core outcome measure areas and 6 supplemental outcome measure areas. The measures are organized into the OMF categories of survival, clinical response, events of interest, patient-reported outcomes, and resource utilization. CONCLUSIONS: This effort identified a minimum set of standardized measures that are relevant to patients and clinicians and appropriate for use in DLS registries, other research efforts, and clinical practice. Collection of these measures across registries and clinical practice is an important step for building research data infrastructure, creating learning healthcare systems, and improving patient management and outcomes in DLS.

Harmonized Outcome Measures for Use in Depression Patient Registries and Clinical Practice.

Major depressive disorder is a common mental health condition that affects an estimated 16.2 million adults and 3.1 million adolescents in the United States. Yet, a lack of uniformity remains in measurements and monitoring for depression both in clinical practice and in research settings. This project aimed to develop a minimum set of standardized outcome measures relevant to both patients and clinicians that can be collected in depression registries and clinical practice. Twenty-nine depression registries and related data collection efforts were identified and invited to submit outcome measures. Additional measures were identified through literature searches and reviews of quality measures. A multistakeholder panel representing clinicians; payers; government agencies; industry; and medical specialty, health care quality, and patient advocacy organizations categorized the 27 identified measures using the Agency for Healthcare Research and Quality's supported Outcome Measures Framework. The panel identified 10 broadly relevant measures and harmonized definitions for these measures through in-person and virtual meetings. The harmonized measures represent a minimum set of outcomes that are relevant to clinicians and patients and appropriate for use in depression research and clinical practice. Routine and consistent collection of these measures in registries and other systems would support creation of a national research infrastructure to efficiently address new questions, improve patient management and outcomes, and facilitate care coordination.

Assessing Real-World Data Quality: The Application of Patient Registry Quality Criteria to Real-World Data and Real-World Evidence.

The use of real-world data and real-world evidence to inform health care decisions is increasing. Yet, the variable quality of these data and the lack of widely-accepted criteria by which to assess quality create uncertainty about how and when to use these data and the associated evidence in decision making. Patient registries are an important source of real-world data and real-world evidence. The good practices and evaluation criteria developed for patient registries are highly relevant to real-world data and real-world evidence and offer a foundation for a unified set of quality criteria that can be applied across sources of real-world data and real-world evidence intended for use in medical product evaluation.

Harmonized outcome measures for use in asthma patient registries and clinical practice.

BACKGROUND: Asthma, a common chronic airway disorder, affects an estimated 25 million persons in the United States and 330 million persons worldwide. Although many asthma patient registries exist, the ability to link and compare data across registries is hindered by a lack of harmonization in the outcome measures collected by each registry. OBJECTIVES: The purpose of this project was to develop a minimum set of patient- and provider-relevant standardized outcome measures that could be collected in asthma patient registries and clinical practice. METHODS: Asthma registries were identified through multiple sources and invited to join the workgroup and submit outcome measures. Additional measures were identified through literature searches and reviews of quality measures and consensus statements. Outcome measures were categorized by using the Agency for Healthcare Research and Quality's supported Outcome Measures Framework. A minimum set of broadly relevant measures was identified. Measure definitions were harmonized through in-person and virtual meetings. RESULTS: Forty-six outcome measures, including those identified from 13 registries, were curated and harmonized into a minimum set of 21 measures in the Outcome Measures Framework categories of survival, clinical response, events of interest, patient-reported outcomes, resource utilization, and experience of care. The harmonized definitions build on existing consensus statements and are appropriate for adult and pediatric patients. CONCLUSIONS: The harmonized measures represent a minimum set of outcomes that are relevant in asthma research and clinical practice. Routine and consistent collection of these measures in registries and other systems would support creation of a national research infrastructure to efficiently address new questions and improve patient management and outcomes.

Harmonized outcome measures for use in atrial fibrillation patient registries and clinical practice: Endorsed by the Heart Rhythm Society Board of Trustees.

BACKGROUND: Atrial fibrillation (AF) affects an estimated 33 million people worldwide, leading to increased mortality and an increased risk of heart failure and stroke. Many AF patient registries exist, but the ability to link and compare data across registries is hindered by differences in the outcome measures collected by each registry and a lack of harmonization. OBJECTIVES: The purpose of this project was to develop a minimum set of standardized outcome measures that could be collected in AF patient registries and clinical practice. METHODS: AF patient registries were identified through multiple sources and invited to join the workgroup and submit outcome measures. Additional measures were identified through literature searches and reviews of consensus statements. Outcome measures were categorized using the Agency for Healthcare Research and Quality's supported Outcome Measures Framework (OMF). A minimum set of broadly relevant measures was identified. Measure definitions were harmonized through in-person and virtual meetings. RESULTS: One hundred twelve outcome measures, including those from thirteen registries, were curated according to the OMF and then harmonized into a minimum set of measures in the OMF categories of survival (3 measures), clinical response (3 measures), events of interest (9 measures), patient-reported outcomes (2 measures), and resource utilization (3 measures). The harmonized definitions build on existing consensus statements. CONCLUSIONS: The harmonized measures represent a minimum set of outcomes that are relevant in AF research and clinical practice. Routine and consistent collection of these measures in registries and in other systems would support creation of a research infrastructure to efficiently address new questions and improve patient outcomes.

A framework for creating standardized outcome measures for patient registries.

AIM: Our objectives were to create a conceptual framework for development of standard outcome measures and to design and pilot test a tool for displaying outcome measures. MATERIALS & METHODS: Information on outcome measures used in registries was gathered through stakeholder discussions, which informed the development of the outcome measurement framework and the related tool. RESULTS: The outcome measurement framework is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health areas. The related tool facilitates collecting, displaying and searching for information on outcome measures. CONCLUSION: The model developed through this process offers a framework that can be used to define outcome measures in a standard way across medical conditions.

Incorporating stakeholder perspectives in developing a translation table framework for comparative effectiveness research.

This project used a stakeholder-driven process to understand the factors that drive the selection of study designs for comparative effectiveness research (CER). The project assembled a diverse stakeholder committee to explore the basis of a translation framework and gathered input through surveys, interviews and an in-person meeting. Stakeholders recommended different study designs for the CER topic areas and identified different outcomes as the most important outcomes to study in each area. During the discussions, stakeholders described a variety of factors that influenced their study design recommendations. The stakeholder activities resulted in the identification of several key themes, including the need to have a highly specific detailed research question before discussing appropriate designs and the need to use multiple studies, potentially of different designs, to address the CER topic areas. The insights and themes from this project may inform efforts to develop a translation table.