Differences in information accessed in a pharmacologic knowledge base using a conversational agent vs traditional search methods.

INTRODUCTION: Clinicians rely on pharmacologic knowledge bases to answer medication questions and avoid potential adverse drug events. In late 2018, an artificial intelligence-based conversational agent, Watson Assistant (WA), was made available to online subscribers to the pharmacologic knowledge base, Micromedex®. WA allows users to ask medication-related questions in natural language. This study evaluated search method-dependent differences in the frequency of information accessed by traditional methods (keyword search and heading navigation) vs conversational agent search. MATERIALS AND METHODS: We compared the proportion of information types accessed through the conversational agent to the proportion of analogous information types accessed by traditional methods during the first 6 months of 2020. RESULTS: Addition of the conversational agent allowed early adopters to access 22 different information types contained in the 'quick answers' portion of the knowledge base. These information types were accessed 117,550 times with WA during the study period, compared to 33,649,651 times using traditional search methods. The distribution across information types differed by method employed (c2 test, P < .0001). Single drug/dosing, FDA/non-FDA uses, adverse effects, and drug administration emerged as 4 of the top 5 information types accessed by either method. Intravenous compatibility was accessed more frequently using the conversational agent (7.7% vs. 0.6% for traditional methods), whereas dose adjustments were accessed more frequently via traditional methods (4.8% vs. 1.4% for WA). CONCLUSION: In a widely used pharmacologic knowledge base, information accessed through conversational agents versus traditional methods differed. User-centered studies are needed to understand these differences.

Artificial intelligence-based conversational agent to support medication prescribing.

OBJECTIVE: This article describes the system architecture, training, initial use, and performance of Watson Assistant (WA), an artificial intelligence-based conversational agent, accessible within Micromedex®. MATERIALS AND METHODS: The number and frequency of intents (target of a user's query) triggered in WA during its initial use were examined; intents triggered over 9 months were compared to the frequency of topics accessed via keyword search of Micromedex. Accuracy of WA intents assigned to 400 queries was compared to assignments by 2 independent subject matter experts (SMEs), with inter-rater reliability measured by Cohen's kappa. RESULTS: In over 126 000 conversations with WA, intents most frequently triggered involved dosing (N = 30 239, 23.9%) and administration (N = 14 520, 11.5%). SMEs with substantial inter-rater agreement (kappa = 0.71) agreed with intent mapping in 247 of 400 queries (62%), including 16 queries related to content that WA and SMEs agreed was unavailable in WA. SMEs found 57 (14%) of 400 queries incorrectly mapped by WA; 112 (28%) queries unanswerable by WA included queries that were either ambiguous, contained unrecognized typographical errors, or addressed topics unavailable to WA. Of the queries answerable by WA (288), SMEs determined 231 (80%) were correctly linked to an intent. DISCUSSION: A conversational agent successfully linked most queries to intents in Micromedex. Ongoing system training seeks to widen the scope of WA and improve matching capabilities. CONCLUSION: WA enabled Micromedex users to obtain answers to many medication-related questions using natural language, with the conversational agent facilitating mapping to a broader distribution of topics than standard keyword searches.