Formative evaluation of ontology learning methods for entity discovery by using existing ontologies as reference standards.

OBJECTIVE: Developing a two-step method for formative evaluation of statistical Ontology Learning (OL) algorithms that leverages existing biomedical ontologies as reference standards. METHODS: In the first step optimum parameters are established. A 'gap list' of entities is generated by finding the set of entities present in a later version of the ontology that are not present in an earlier version of the ontology. A named entity recognition system is used to identify entities in a corpus of biomedical documents that are present in the 'gap list', generating a reference standard. The output of the algorithm (new entity candidates), produced by statistical methods, is subsequently compared against this reference standard. An OL method that performs perfectly will be able to learn all of the terms in this reference standard. Using evaluation metrics and precision-recall curves for different thresholds and parameters, we compute the optimum parameters for each method. In the second step, human judges with expertise in ontology development evaluate each candidate suggested by the algorithm configured with the optimum parameters previously established. These judgments are used to compute two performance metrics developed from our previous work: Entity Suggestion Rate (ESR) and Entity Acceptance Rate (EAR). RESULTS: Using this method, we evaluated two statistical OL methods for OL in two medical domains. For the pathology domain, we obtained 49% ESR, 28% EAR with the Lin method and 52% ESR, 39% EAR with the Church method. For the radiology domain, we obtain 87% ESA, 9% EAR using Lin method and 96% ESR, 16% EAR using Church method. CONCLUSION: This method is sufficiently general and flexible enough to permit comparison of any OL method for a specific corpus and ontology of interest.

Effectiveness of lexico-syntactic pattern matching for ontology enrichment with clinical documents.

OBJECTIVE: To evaluate the effectiveness of a lexico-syntactic pattern (LSP) matching method for ontology enrichment using clinical documents. METHODS: Two domains were separately studied using the same methodology. We used radiology documents to enrich RadLex and pathology documents to enrich National Cancer Institute Thesaurus (NCIT). Several known LSPs were used for semantic knowledge extraction. We first retrieved all sentences that contained LSPs across two large clinical repositories, and examined the frequency of the LSPs. From this set, we randomly sampled LSP instances which were examined by human judges. We used a two-step method to determine the utility of these patterns for enrichment. In the first step, domain experts annotated medically meaningful terms (MMTs) from each sentence within the LSP. In the second step, RadLex and NCIT curators evaluated how many of these MMTs could be added to the resource. To quantify the utility of this LSP method, we defined two evaluation metrics: suggestion rate (SR) and acceptance rate (AR). We used these measures to estimate the yield of concepts and relationships, for each of the two domains. RESULTS: For NCIT, the concept SR was 24%, and the relationship SR was 65%. The concept AR was 21%, and the relationship AR was 14%. For RadLex, the concept SR was 37%, and the relationship SR was 55%. The concept AR was 11%, and the relationship AR was 44%. CONCLUSION: The LSP matching method is an effective method for concept and concept relationship discovery in biomedical domains.

A comparison of classification algorithms to automatically identify chest X-ray reports that support pneumonia.

We compared the performance of expert-crafted rules, a Bayesian network, and a decision tree at automatically identifying chest X-ray reports that support acute bacterial pneumonia. We randomly selected 292 chest X-ray reports, 75 (25%) of which were from patients with a hospital discharge diagnosis of bacterial pneumonia. The reports were encoded by our natural language processor and then manually corrected for mistakes. The encoded observations were analyzed by three expert systems to determine whether the reports supported pneumonia. The reference standard for radiologic support of pneumonia was the majority vote of three physicians. We compared (a) the performance of the expert systems against each other and (b) the performance of the expert systems against that of four physicians who were not part of the gold standard. Output from the expert systems and the physicians was transformed so that comparisons could be made with both binary and probabilistic output. Metrics of comparison for binary output were sensitivity (sens), precision (prec), and specificity (spec). The metric of comparison for probabilistic output was the area under the receiver operator characteristic (ROC) curve. We used McNemar's test to determine statistical significance for binary output and univariate z-tests for probabilistic output. Measures of performance of the expert systems for binary (probabilistic) output were as follows: Rules--sens, 0.92; prec, 0.80; spec, 0.86 (Az, 0.960); Bayesian network--sens, 0.90; prec, 0.72; spec, 0.78 (Az, 0.945); decision tree--sens, 0.86; prec, 0.85; spec, 0.91 (Az, 0.940). Comparisons of the expert systems against each other using binary output showed a significant difference between the rules and the Bayesian network and between the decision tree and the Bayesian network. Comparisons of expert systems using probabilistic output showed no significant differences. Comparisons of binary output against physicians showed differences between the Bayesian network and two physicians. Comparisons of probabilistic output against physicians showed a difference between the decision tree and one physician. The expert systems performed similarly for the probabilistic output but differed in measures of sensitivity, precision, and specificity produced by the binary output. All three expert systems performed similarly to physicians.

Quantifying the characteristics of unambiguous chest radiography reports in the context of pneumonia.

RATIONALE AND OBJECTIVES: The purpose of this study was to statistically identify some characteristics of unambiguous (ie, clear) chest radiography reports in the context of acute bacterial pneumonia. MATERIALS AND METHODS: Seven physicians individually read 292 chest radiography reports to determine if they contained radiologic evidence of pneumonia. Unambiguous reports were defined as those that physicians unanimously classified as supporting or not supporting the diagnosis of pneumonia. Ambiguous reports were assigned degrees of ambiguity on the basis of how much disagreement they caused among the physicians. Characteristics of unambiguous reports as described in the literature were manually quantified and assigned to every report. To identify characteristics that statistically distinguished unambiguous from ambiguous reports, the authors performed an ordinal logistic regression analysis for which the dependent variable was the number of dissenting votes the report received and the independent variables were the quantified characteristics of the report. RESULTS: Six independent variables were statistically significantly associated with unambiguous reports (P < .05). Three were positively associated: an interpretation of whether findings supported the diagnosis of pneumonia in reports with pneumonia-related observations, short sentences, and redundancy of pneumonia-related observations. Three were negatively associated: high use of uncertainty modifiers for pneumonia-related observations, use of only descriptive terms to describe pneumonia-related observations, and insufficient amount of pneumonia-related information. CONCLUSION: The most influential characteristic of an unambiguous chest radiography report was an interpretation of whether the radiograph supported the diagnosis of pneumonia when findings could be indicative.

A simple algorithm for identifying negated findings and diseases in discharge summaries.

Narrative reports in medical records contain a wealth of information that may augment structured data for managing patient information and predicting trends in diseases. Pertinent negatives are evident in text but are not usually indexed in structured databases. The objective of the study reported here was to test a simple algorithm for determining whether a finding or disease mentioned within narrative medical reports is present or absent. We developed a simple regular expression algorithm called NegEx that implements several phrases indicating negation, filters out sentences containing phrases that falsely appear to be negation phrases, and limits the scope of the negation phrases. We compared NegEx against a baseline algorithm that has a limited set of negation phrases and a simpler notion of scope. In a test of 1235 findings and diseases in 1000 sentences taken from discharge summaries indexed by physicians, NegEx had a specificity of 94.5% (versus 85.3% for the baseline), a positive predictive value of 84.5% (versus 68.4% for the baseline) while maintaining a reasonable sensitivity of 77.8% (versus 88.3% for the baseline). We conclude that with little implementation effort a simple regular expression algorithm for determining whether a finding or disease is absent can identify a large portion of the pertinent negatives from discharge summaries.

Combining decision support methodologies to diagnose pneumonia.

OBJECTIVE: To evaluate the performance of a computerized decision support system that combines two different decision support methodologies (a Bayesian network and a natural language understanding system) for the diagnosis of patients with pneumonia. DESIGN: Evaluation study using data from a prospective, clinical study. PATIENTS: All patients 18 years and older who presented to the emergency department of a tertiary care setting and whose chest x-ray report was available during the encounter. METHODS: The computerized decision support system calculated a probability of pneumonia using information provided by the two systems. Outcome measures were the area under the receiver operating characteristic curve, sensitivity, specificity, predictive values, likelihood ratios, and test effectiveness. RESULTS: During the 3-month study period there were 742 patients (45 with pneumonia). The area under the receiver operating characteristic curve was 0.881 (95% CI: 0.822, 0.925) for the Bayesian network alone and 0.916 (95% CI: 0.869, 0.949) for the Bayesian network combined with the natural language understanding system (p=0.01). CONCLUSION: Combining decision support methodologies that process information stored in different data formats can increase the performance of a computerized decision support system.

Evaluation of negation phrases in narrative clinical reports.

OBJECTIVE: Automatically identifying findings or diseases described in clinical textual reports requires determining whether clinical observations are present or absent. We evaluate the use of negation phrases and the frequency of negation in free-text clinical reports. METHODS: A simple negation algorithm was applied to ten types of clinical reports (n=42,160) dictated during July 2000. We counted how often each of 66 negation phrases was used to mark a clinical observation as absent. Physicians read a random sample of 400 sentences, and precision was calculated for the negation phrases. We measured what proportion of clinical observations were marked as absent. RESULTS: The negation algorithm was triggered by sixty negation phrases with just seven of the phrases accounting for 90% of the negations. The negation phrases received an overall precision of 97%, with "not" earning the lowest precision of 63%. Between 39% and 83% of all clinical observations were identified as absent by the negation algorithm, depending on the type of report analyzed. The most frequently used clinical observations were negated the majority of the time. CONCLUSION: Because clinical observations in textual patient records are frequently negated, identifying accurate negation phrases is important to any system processing these reports.

Automatic detection of acute bacterial pneumonia from chest X-ray reports.

OBJECTIVE: To evaluate the performance of a natural language processing system in extracting pneumonia-related concepts from chest x-ray reports. DESIGN: Four physicians, three lay persons, a natural language processing system, and two keyword searches (designated AAKS and KS) detected the presence or absence of three pneumonia-related concepts and inferred the presence or absence of acute bacterial pneumonia from 292 chest x-ray reports. Gold standard: Majority vote of three independent physicians. Reliability of the gold standard was measured. OUTCOME MEASURES: Recall, precision, specificity, and agreement (using Finn's R: statistic) with respect to the gold standard. Differences between the physicians and the other subjects were tested using the McNemar test for each pneumonia concept and for the disease inference of acute bacterial pneumonia. RESULTS: Reliability of the reference standard ranged from 0.86 to 0.96. Recall, precision, specificity, and agreement (Finn R:) for the inference on acute bacterial pneumonia were, respectively, 0.94, 0.87, 0.91, and 0.84 for physicians; 0.95, 0.78, 0.85, and 0.75 for natural language processing system; 0.46, 0.89, 0.95, and 0.54 for lay persons; 0.79, 0.63, 0.71, and 0.49 for AAKS; and 0.87, 0.70, 0.77, and 0.62 for KS. The McNemar pairwise comparisons showed differences between one physician and the natural language processing system for the infiltrate concept and between another physician and the natural language processing system for the inference on acute bacterial pneumonia. The comparisons also showed that most physicians were significantly different from the other subjects in all pneumonia concepts and the disease inference. CONCLUSION: In extracting pneumonia related concepts from chest x-ray reports, the performance of the natural language processing system was similar to that of physicians and better than that of lay persons and keyword searches. The encoded pneumonia information has the potential to support several pneumonia-related applications used in our institution. The applications include a decision support system called the antibiotic assistant, a computerized clinical protocol for pneumonia, and a quality assurance application in the radiology department.

Contribution of a speech recognition system to a computerized pneumonia guideline in the emergency department.

OBJECTIVE: Evaluate the effect of a radiology speech recognition system on a real-time computerized guideline in the emergency department. METHODS: We collected all chest x-ray reports (n = 727) generated for patients in the emergency department during a six-week period. We divided the concurrently generated reports into those generated with speech recognition and those generated by traditional dictation. We compared the two sets of reports for availability during the patient's emergency department encounter and for readability. RESULTS: Reports generated by speech recognition were available seven times more often during the patients' encounters than reports generated by traditional dictation. Using speech recognition reduced the turnover time of reports from 12 hours 33 minutes to 2 hours 13 minutes. Readability scores were identical for both kinds of reports. CONCLUSION: Using speech recognition to generate chest x-ray reports reduces turnover time so reports are available while patients are in the emergency department.

Automatic identification of pneumonia related concepts on chest x-ray reports.

A medical language processing system called SymText, two other automated methods, and a lay person were compared against an internal medicine resident for their ability to identify pneumonia related concepts on chest x-ray reports. Sensitivity (recall), specificity, and positive predictive value (precision) are reported with respect to an independent panel of physicians. Overall the performance of SymText was similar to the physician and superior to the other methods. The automatic encoding of pneumonia concepts will support clinical research, decision making, computerized clinical protocols, and quality assurance in a radiology department.

Comparing expert systems for identifying chest x-ray reports that support pneumonia.

We compare the performance of four computerized methods in identifying chest x-ray reports that support acute bacterial pneumonia. Two of the computerized techniques are constructed from expert knowledge, and two learn rules and structure from data. The two machine learning systems perform as well as the expert constructed systems. All of the computerized techniques perform better than a baseline keyword search and a lay person, and perform as well as a physician. We conclude that machine learning can be used to identify chest x-ray reports that support pneumonia.

Bayesian modeling for linking causally related observations in chest X-ray reports.

Our natural language understanding system outputs a list of diseases, findings, and appliances found in a chest x-ray report. The system described in this paper links those diseases and findings that are causally related. Using Bayesian networks to model the conceptual and diagnostic information found in a chest x-ray we are able to infer more specific information about the findings that are linked to diseases.