Comparative evaluation of accuracy of extraction of medication information from narrative physician notes by commercial and academic natural language processing software packages.

Medication information is frequently only found in narrative physician notes. It is now possible to extract medication data from narrative documents using NLP technology. A number of commercial and academic NLP software packages can perform this function. In this paper we report the first comparative evaluation of their accuracy. Evaluation was carried out on 150 notes randomly selected from electronic medical record. NLP software results were compared to manual abstraction of medication data by two independent reviewers. Recall, precision and F-measure for identification of medication names, doses, frequencies, routes and inactive status were computed. For different data categories, recall ranged from 6.6% to 90.6%, and precision from 16.7% to 96.6%. Recall was highest for medication names and lowest for identification of inactive medications; there were no significant differences in precision between data categories. NLP software accuracy improved significantly over the last decade but further improvements are needed, particularly in analysis of complex sentences.