A Comparison of Natural Language Processing Methods for the Classification of Lumbar Spine Imaging Findings Related to Lower Back Pain.

RATIONALE AND OBJECTIVES: The use of natural language processing (NLP) in radiology provides an opportunity to assist clinicians with phenotyping patients. However, the performance and generalizability of NLP across healthcare systems is uncertain. We assessed the performance within and generalizability across four healthcare systems of different NLP representational methods, coupled with elastic-net logistic regression to classify lower back pain-related findings from lumbar spine imaging reports. MATERIALS AND METHODS: We used a dataset of 871 X-ray and magnetic resonance imaging reports sampled from a prospective study across four healthcare systems between October 2013 and September 2016. We annotated each report for 26 findings potentially related to lower back pain. Our framework applied four different NLP methods to convert text into feature sets (representations). For each representation, our framework used an elastic-net logistic regression model for each finding (i.e., 26 binary or "one-vs.-rest" classification models). For performance evaluation, we split data into training (80%, 697/871) and testing (20%, 174/871). In the training set, we used cross validation to identify the optimal hyperparameter value and then retrained on the full training set. We then assessed performance based on area under the curve (AUC) for the test set. We repeated this process 25 times with each repeat using a different random train/test split of the data, so that we could estimate 95% confidence intervals, and assess significant difference in performance between representations. For generalizability evaluation, we trained models on data from three healthcare systems with cross validation and then tested on the fourth. We repeated this process for each system, then calculated mean and standard deviation (SD) of AUC across the systems. RESULTS: For individual representations, n-grams had the best average performance across all 26 findings (AUC: 0.960). For generalizability, document embeddings had the most consistent average performance across systems (SD: 0.010). Out of these 26 findings, we considered eight as potentially clinically important (any stenosis, central stenosis, lateral stenosis, foraminal stenosis, disc extrusion, nerve root displacement compression, endplate edema, and listhesis grade 2) since they have a relatively greater association with a history of lower back pain compared to the remaining 18 classes. We found a similar pattern for these eight in which n-grams and document embeddings had the best average performance (AUC: 0.954) and generalizability (SD: 0.007), respectively. CONCLUSION: Based on performance assessment, we found that n-grams is the preferred method if classifier development and deployment occur at the same system. However, for deployment at multiple systems outside of the development system, or potentially if physician behavior changes within a system, one should consider document embeddings since embeddings appear to have the most consistent performance across systems.

Customizing clinical narratives for the electronic medical record interface using cognitive methods.

OBJECTIVE: As healthcare practice transitions from paper-based to computer-based records, there is increasing need to determine an effective electronic format for clinical narratives. Our research focuses on utilizing a cognitive science methodology to guide the conversion of medical texts to a more structured, user-customized presentation in the electronic medical record (EMR). DESIGN: We studied the use of discharge summaries by psychiatrists with varying expertise-experts, intermediates, and novices. Experts were given two hypothetical emergency care scenarios with narrative discharge summaries and asked to verbalize their clinical assessment. Based on the results, the narratives were presented in a more structured form. Intermediate and novice subjects received a narrative and a structured discharge summary, and were asked to verbalize their assessments of each. MEASUREMENTS: A qualitative comparison of the interview transcripts of all subjects was done by analysis of recall and inference made with respect to level of expertise. RESULTS: For intermediate and novice subjects, recall was greater with the structured form than with the narrative. Novices were also able to make more inferences (not always accurate) from the structured form than with the narrative. Errors occurred in assessments using the narrative form but not the structured form. CONCLUSIONS: Our cognitive methods to study discharge summary use enabled us to extract a conceptual representation of clinical narratives from end-users. This method allowed us to identify clinically relevant information that can be used to structure medical text for the EMR and potentially improve recall and reduce errors.