Evaluation of a method that supports pathology report coding.

OBJECTIVES: The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the agreement between pathologists in coding and the quality of a system that supports pathologists in coding pathology reports were evaluated. METHODS: Six sets of three pathologists each received a different set of 40 pathology reports. Five different SNOMED code lines accompanied each pathology report. Three pathologists evaluated the correctness of each of these code lines. Kappa values and values for the reliability coefficients were determined to gain insight in the variance observed when coding pathology reports. The system that is evaluated compares a newly entered report, represented as a multi-dimensional word vector, with reports in a library, represented in the same way. The reports in the library are already coded. The system presents the code lines belonging to the five library reports most similar to the newly entered one to the pathologist in this way supporting the pathologist in determining the correct codes. A high similarity between two reports is indicated by a large value of the inproduct of the vector of the newly entered report and the vector of a report in the library. RESULTS: Agreement between pathologists in coding was fair (average kappa of 0.44). The reliability coefficient varied from 0.81 to 0.89 for the six sets of pathology reports. The system gave correct suggestions in 50% of the reports. In another 30% it was helpful for the pathologists. CONCLUSIONS: On the basis of the level of the reliability coefficients it could be concluded that three pathologists are indeed sufficient for obtaining a gold standard for evaluating the system. The method used for comparing reports is not strong enough to allow fully automatic coding. It could be shown that the system induces a more uniform coding by pathologists. An evaluation of the incorrect suggestions of the system indicates that the performance of the system can still be improved.

Supporting the classification of pathology reports: comparing two information retrieval methods.

In this contribution two methods from the domain of information retrieval are compared. The goal of the retrieval is to select from a library of pathology reports those ones that are most similar to a given report. The SNOMED codes that accompany these reports are presented to the pathologist who has to code the given report with the aim to improve the quality of coding. The reports were represented either as a vector of words or as a vector of N-grams. Both 4-, 5- and 6-grams were used. The similarity of the reports was determined by comparing the SNOMED terms that were added to the reports. It could be concluded that the word-based method was consistently better than the N-gram method.

Automatic coding of diagnostic reports.

A method is presented for assigning classification codes to pathology reports by searching similar reports from an archive collection. The key for searching is textual similarity, which estimates the true, semantic similarity. This method does not require explicit modeling, and can be applied to any language or any application domain that uses natural language reporting. A number of simulation experiments was run to assess the accuracy of the method and to indicate the role of size of the archive and the transfer of document collections across laboratories. In at least 63% of the simulation trials, the most similar archive text offered a suitable classification on organ, origin and diagnosis. In 85 to 90% of the trials, the archive's best solution was found within the first five similar reports. The results indicate that the method is suitable for its purpose: suggesting potentially correct classifications to the reporting diagnostician.

Automatic SNOMED classification--a corpus-based method.

This paper presents a method of automatic classification of clinical narrative through text comparison. A diagnosis report can be classified by searching archive texts that show a high textual similarity, and the 'nearest neighbor classifies the case. This paper describes the method's theoretical background and gives implementation details. Large scale simulation experiments were run with a wide range of histology reports. Results showed that for 80-84% of the trials, relevant classification lines were included among the first five alternatives. In 5% of the cases, retrieval was unsuccessful due to the absence of relevant archive reports. From the results it is concluded that the method is a versatile approach for finding potentially good classifications.

Speech interfacing for diagnosis reporting systems: an overview.

Automatic speech recognition has since long been seen as an ideal method for innovation in diagnosis reporting. Speech technology now seems on the verge of introducing (commercially) attractive systems. The selection of a good speech recogniser is only one consideration in system design. Interface aspects, error handling, reporting method and implementation in the daily working routine are interwoven with the selection of an appropriate speech recognition technique, and should therefore be determined first.