Deploying a national clinical text processing infrastructure.

OBJECTIVES: Clinical text processing offers a promising avenue for improving multiple aspects of healthcare, though operational deployment remains a substantial challenge. This case report details the implementation of a national clinical text processing infrastructure within the Department of Veterans Affairs (VA). METHODS: Two foundational use cases, cancer case management and suicide and overdose prevention, illustrate how text processing can be practically implemented at scale for diverse clinical applications using shared services. RESULTS: Insights from these use cases underline both commonalities and differences, providing a replicable model for future text processing applications. CONCLUSIONS: This project enables more efficient initiation, testing, and future deployment of text processing models, streamlining the integration of these use cases into healthcare operations. This project implementation is in a large integrated health delivery system in the United States, but we expect the lessons learned to be relevant to any health system, including smaller local and regional health systems in the United States.

An examination of the criterion-related validity of varying methods of indexing clinically significant change in posttraumatic stress disorder treatment.

OBJECTIVE: Clinicians, patients, and researchers need benchmarks to index individual-level clinically significant change (CSC) to guide decision making and inferences about treatment efficacy. Yet, there is no consensus best practice for determining CSC for posttraumatic stress disorder (PTSD) treatments. We examined criterion-related validity of the most common approach-Jacobson and Truax's (J&T; 1991) procedures for indexing CSC. We generated and compared four methods of calculating the J&T indices of CSC (two sets of sample-specific inputs, putatively norm-referenced benchmarks, and a combination of sample-specific and norm-referenced criteria) with respect to their association with a criterion index of quality of life (QoL). METHOD: Participants were 91 women Veterans enrolled in a randomized clinical trial for PTSD who completed self-report measures on PTSD symptoms and various domains of QoL and functioning, pre- and posttreatment. For each of the four methods used to calculate CSC, the QoL composite was regressed onto the CSC categories. RESULTS: All methods explained large variance in change in QoL. Across all methods, participants categorized as unchanged had smaller changes in QoL, compared with those who improved or had probable recovery. The norm-referenced benchmarks accounted for the relatively largest amount of variance in QoL, but categorized the fewest patients as having made CSC. CONCLUSIONS: The J&T methodology for indexing CSC in PTSD symptoms has criterion-related validity, and a norm-referenced benchmark appears to be the most potent. However, the norm-referenced parameters may be overly specific, potentially leading to an underestimate of improvement. Research is needed to test the generalizability of these results. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Leveraging available metadata in VA PTSD clinics and generating benchmarks for clinically significant change.

OBJECTIVE: Measurement-based care is designed to track symptom levels during treatment and leverage clinically significant change benchmarks to improve quality and outcomes. Though the Veterans Health Administration promotes monitoring progress within posttraumatic stress disorder (PTSD) clinical teams, actionability of data is diminished by a lack of population-based benchmarks for clinically significant change. We reported the state of repeated measurement within PTSD clinical teams, generated benchmarks, and examined outcomes based on these benchmarks. METHOD: PTSD Checklist for the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition data were culled from the Corporate Data Warehouse from the pre-COVID-19 year for Veterans who received at least eight sessions in 14 weeks (episode of care [EOC] cohort) and those who received sporadic care (modal cohort). We used the Jacobson and Truax (1991) approach to generate clinically significant change benchmarks at clinic, regional, and national levels and calculated the frequency of cases that deteriorated, were unchanged, improved, or probably recovered, using our generated benchmarks and benchmarks from a recent study, for both cohorts. RESULTS: Both the number of repeated measurements and the cases who had multisession care in the Corporate Data Warehouse were very low. Clinically significant change benchmarks were similar across locality levels. The modal cohort had worse outcomes than the EOC cohort. CONCLUSIONS: National benchmarks for clinically significant change could improve the actionability of assessment data for measurement-based care. Benchmarks created using data from Veterans who received multisession care had better outcomes than those receiving sporadic care. Measurement-based care in PTSD clinical teams is hampered by low rates of repeated assessments of outcome. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Integration of Face-to-Face Screening With Real-time Machine Learning to Predict Risk of Suicide Among Adults.

Importance: Understanding the differences and potential synergies between traditional clinician assessment and automated machine learning might enable more accurate and useful suicide risk detection. Objective: To evaluate the respective and combined abilities of a real-time machine learning model and the Columbia Suicide Severity Rating Scale (C-SSRS) to predict suicide attempt (SA) and suicidal ideation (SI). Design, Setting, and Participants: This cohort study included encounters with adult patients (aged ≥18 years) at a major academic medical center. The C-SSRS was administered during routine care, and a Vanderbilt Suicide Attempt and Ideation Likelihood (VSAIL) prediction was generated in the electronic health record. Encounters took place in the inpatient, ambulatory surgical, and emergency department settings. Data were collected from June 2019 to September 2020. Main Outcomes and Measures: Primary outcomes were the incidence of SA and SI, encoded as International Classification of Diseases codes, occurring within various time periods after an index visit. We evaluated the retrospective validity of the C-SSRS, VSAIL, and ensemble models combining both. Discrimination metrics included area under the receiver operating curve (AUROC), area under the precision-recall curve (AUPR), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Results: The cohort included 120 398 unique index visits for 83 394 patients (mean [SD] age, 51.2 [20.6] years; 38 107 [46%] men; 45 273 [54%] women; 13 644 [16%] Black; 63 869 [77%] White). Within 30 days of an index visit, the combined models had higher AUROC (SA: 0.874-0.887; SI: 0.869-0.879) than both the VSAIL (SA: 0.729; SI: 0.773) and C-SSRS (SA: 0.823; SI: 0.777) models. In the highest risk-decile, ensemble methods had PPV of 1.3% to 1.4% for SA and 8.3% to 8.7% for SI and sensitivity of 77.6% to 79.5% for SA and 67.4% to 70.1% for SI, outperforming VSAIL (PPV for SA: 0.4%; PPV for SI: 3.9%; sensitivity for SA: 28.8%; sensitivity for SI: 35.1%) and C-SSRS (PPV for SA: 0.5%; PPV for SI: 3.5%; sensitivity for SA: 76.6%; sensitivity for SI: 68.8%). Conclusions and Relevance: In this study, suicide risk prediction was optimal when leveraging both in-person screening (for acute measures of risk in patient-reported suicidality) and historical EHR data (for underlying clinical factors that can quantify a patient's passive risk level). To improve suicide risk classification, prediction systems could combine pretrained machine learning with structured clinician assessment without needing to retrain the original model.

Correction: Evaluating the extent of reusability of CYP2C19 genotype data among patients genotyped for antiplatelet therapy selection.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Evaluating the extent of reusability of CYP2C19 genotype data among patients genotyped for antiplatelet therapy selection.

PURPOSE: Genotype-guided antiplatelet therapy is increasingly being incorporated into clinical care. The purpose of this study is to determine the extent to which patients initially genotyped for CYP2C19 to guide antiplatelet therapy were prescribed additional medications affected by CYP2C19. METHODS: We assembled a cohort of patients from eight sites performingCYP2C19 genotyping to inform antiplatelet therapy. Medication orders were evaluated from time of genotyping through one year. The primary endpoint was the proportion of patients prescribed two or more CYP2C19 substrates. Secondary endpoints were the proportion of patients with a drug-genotype interaction and time to receiving a CYP2C19 substrate. RESULTS: Nine thousand one hundred ninety-one genotyped patients (17% nonwhite) with a mean age of 68 ± 3 years were evaluated; 4701 (51%) of patients received two or more CYP2C19 substrates and 3835 (42%) of patients had a drug-genotype interaction. The average time between genotyping and CYP2C19 substrate other than antiplatelet therapy was 25 ± 10 days. CONCLUSIONS: More than half of patients genotyped in the setting of CYP2C19-guided antiplatelet therapy received another medication impacted by CYP2C19 in the following year. Given that genotype is stable for a patient's lifetime, this finding has implications for cost effectiveness, patient care, and treatment outcomes beyond the indication for which it was originally performed.

Using patient-reported outcome measures for program evaluation: Design and findings on intention-to-treat outcomes from the Veterans Outcome Assessment survey.

The Veterans Outcomes Assessment (VOA) program surveys Veteran Health Administration (VHA) patients when they begin mental health treatment and at follow-up at three months to obtain patient-reported outcomes measures (PROM). It complements VA's evolving program in measurement-based care by providing additional data that can be useful for program evaluation including assessments of patients who have not been seen for ongoing mental health care. In principle, it provides data on intention-to-treat outcomes for program evaluation to complement the outcomes for patients who are receiving ongoing treatment that can be derived from measurement-based care. VOA findings confirm differences in outcomes between patients who have continued to be seen for treatment and those who have not. Patients in general mental health clinics with no encounters between the baseline and follow-up assessments who reported discontinuing care because they did not want or need treatment improved more, and those who discontinued due to problems improved less than those who remained in treatment. Experience with VOA has identified a number of issues that must be addressed before it is possible to use intention-to-treat outcomes for program evaluation.

Exploring the frontier of electronic health record surveillance: the case of postoperative complications.

BACKGROUND: The aim of this study was to build electronic algorithms using a combination of structured data and natural language processing (NLP) of text notes for potential safety surveillance of 9 postoperative complications. METHODS: Postoperative complications from 6 medical centers in the Southeastern United States were obtained from the Veterans Affairs Surgical Quality Improvement Program (VASQIP) registry. Development and test datasets were constructed using stratification by facility and date of procedure for patients with and without complications. Algorithms were developed from VASQIP outcome definitions using NLP-coded concepts, regular expressions, and structured data. The VASQIP nurse reviewer served as the reference standard for evaluating sensitivity and specificity. The algorithms were designed in the development and evaluated in the test dataset. RESULTS: Sensitivity and specificity in the test set were 85% and 92% for acute renal failure, 80% and 93% for sepsis, 56% and 94% for deep vein thrombosis, 80% and 97% for pulmonary embolism, 88% and 89% for acute myocardial infarction, 88% and 92% for cardiac arrest, 80% and 90% for pneumonia, 95% and 80% for urinary tract infection, and 77% and 63% for wound infection, respectively. A third of the complications occurred outside of the hospital setting. CONCLUSIONS: Computer algorithms on data extracted from the electronic health record produced respectable sensitivity and specificity across a large sample of patients seen in 6 different medical centers. This study demonstrates the utility of combining NLP with structured data for mining the information contained within the electronic health record.

Detection of infectious symptoms from VA emergency department and primary care clinical documentation.

OBJECTIVE: The majority of clinical symptoms are stored as free text in the clinical record, and this information can inform clinical decision support and automated surveillance efforts if it can be accurately processed into computer interpretable data. METHODS: We developed rule-based algorithms and evaluated a natural language processing (NLP) system for infectious symptom detection using clinical narratives. Training (60) and testing (444) documents were randomly selected from VA emergency department, urgent care, and primary care records. Each document was processed with NLP and independently manually reviewed by two clinicians with adjudication by referee. Infectious symptom detection rules were developed in the training set using keywords and SNOMED-CT concepts, and subsequently evaluated using the testing set. RESULTS: Overall symptom detection performance was measured with a precision of 0.91, a recall of 0.84, and an F measure of 0.87. Overall symptom detection with assertion performance was measured with a precision of 0.67, a recall of 0.62, and an F measure of 0.64. Among those instances in which the automated system matched the reference set determination for symptom, the system correctly detected 84.7% of positive assertions, 75.1% of negative assertions, and 0.7% of uncertain assertions. CONCLUSION: This work demonstrates how processed text could enable detection of non-specific symptom clusters for use in automated surveillance activities.

Detection of blood culture bacterial contamination using natural language processing.

Microbiology results are reported in semi-structured formats and have a high content of useful patient information. We developed and validated a hybrid regular expression and natural language processing solution for processing blood culture microbiology reports. Multi-center Veterans Affairs training and testing data sets were randomly extracted and manually reviewed to determine the culture and sensitivity as well as contamination results. The tool was iteratively developed for both outcomes using a training dataset, and then evaluated on the test dataset to determine antibiotic susceptibility data extraction and contamination detection performance. Our algorithm had a sensitivity of 84.8% and a positive predictive value of 96.0% for mapping the antibiotics and bacteria with appropriate sensitivity findings in the test data. The bacterial contamination detection algorithm had a sensitivity of 83.3% and a positive predictive value of 81.8%.

Direct comparison of MEDCIN and SNOMED CT for representation of a general medical evaluation template.

BACKGROUND: Two candidate terminologies to support entry of general medical data are SNOMED CT and MEDCIN. We compare the ability of SNOMED CT and MEDCIN to represent concepts and interface terms from a VA general medical examination template. METHODS: We parsed the VA general medical evaluation template and mapped the resulting expressions into SNOMED CT and MEDCIN. Internists conducted double independent reviews on 864 expressions. Exact concept level matches were used to evaluate reference coverage. Exact term level matches were required for interface terms. RESULTS: Sensitivity of SNOMED CT as a reference terminology was 83% vs. 25% for MEDCIN (p<0.001). The sensitivity of SNOMED CT as an interface terminology was 53% vs. 7% for MEDCIN (P< 0.001). DISCUSSION: The content coverage of SNOMED CT as a reference terminology and as an interface terminology outperformed MEDCIN. We did not evaluate other aspects of interface terminologies such as richness of clinical linkages.

eQuality: electronic quality assessment from narrative clinical reports.

OBJECTIVE: To evaluate an electronic quality (eQuality) assessment tool for dictated disability examination records. METHODS: We applied automated concept-based indexing techniques to automated quality screening of Department of Veterans Affairs spine disability examinations that had previously undergone gold standard quality review by human experts using established quality indicators. We developed automated quality screening rules and refined them iteratively on a training set of disability examination reports. We applied the resulting rules to a novel test set of spine disability examination reports. The initial data set was composed of all electronically available examination reports (N=125,576) finalized by the Veterans Health Administration between July and September 2001. RESULTS: Sensitivity was 91% for the training set and 87% for the test set (P-.02). Specificity was 74% for the training set and 71% for the test set (P=.44). Human performance ranged from 4% to 6% higher (P<.001) than the eQuality tool in sensitivity and 13% to 16% higher in specificity (P<.001). In addition, the eQuality tool was equivalent or higher in sensitivity for 5 of 9 individual quality indicators. CONCLUSION: The results demonstrate that a properly authored computer-based expert systems approach can perform quality measurement as well as human reviewers for many quality indicators. Although automation will likely always rely on expert guidance to be accurate and meaningful, eQuality is an important new method to assist clinicians in their efforts to practice safe and effective medicine.

SNOMED CT: utility for a general medical evaluation template.

BACKGROUND: The U.S. government has licensed SNOMED CT to permit broad-based evaluation and use of the terminology. We evaluated the ability of SNOMED CT to represent terms used for interface objects (e.g., labels and captions) and concepts used for data and branching logic in a general medical evaluation template in use within the Department of Veterans Affairs. METHODS: The general medical evaluation form definition, report definition, and script files were parsed and 1573 expressions were mapped into SNOMED CT. Compositional expressions required to represent 1171 concepts. Double independent reviews were conducted. Exact concept level matches were used to evaluate reference coverage. Exact term level matches were required for interface terms. Semantics were analyzed for a randomly selected subset of 20 terms. RESULTS: Sensitivity of SNOMED CT as a reference terminology was 63.8% , ranging from 29.3% for history items to 92.4% for exam items. SNOMED CT's sensitivity as an "interface terminology" was 55.0%. 80% of the necessary linking semantics for the subset were present. Subgroup statistics are presented. DISCUSSION: SNOMED CT is promising as a terminology for knowledge representation underlying a large general medical evaluation. Its performed less well as an interface terminology.

The effect of standardized, computer-guided templates on quality of VA disability exams.

BACKGROUND: The costs and limitations of clinical encounter documentation using dictation/transcription have provided impetus for increased use of computerized structured data entry to enforce standardization and improve quality. The purpose of the present study is to compare exam report quality of Veterans Affairs (VA) disability exams documented by computerized protocol-guided templates with exams documented in the usual fashion (dictation). METHODS: Exam report quality for 17,490 VA compensation and pension (C&P) disability exams reviewed in 2005 was compared for exam reports completed by template and exam reports completed in routine fashion (dictation). An additional set of 2,903 exams reviewed for quality the last three months of 2004 were used for baseline comparison. RESULTS: Mean template quality scores of 91 (95% CI 89, 92) showed significant improvement over routine exams conducted during the study period 78 (95% CI 77, 78) and at baseline 73 (95% CI 72, 75). The quality difference among examination types is presented. DISCUSSION: The results of the present study suggest that use of the standardized, guided documentation templates in VA disability exams produces significant improvement in quality compared with routinely completed exams (dictation). The templates demonstrate the opportunity and capacity for informatics tools to enhance delivery of care when operating in a health system with a sophisticated electronic medical record.