Artificial intelligence in virtual standardized patients: Combining natural language understanding and rule based dialogue management to improve conversational fidelity.

INTRODUCTION: Advances in natural language understanding have facilitated the development of Virtual Standardized Patients (VSPs) that may soon rival human patients in conversational ability. We describe herein the development of an artificial intelligence (AI) system for VSPs enabling students to practice their history taking skills. METHODS: Our system consists of (1) Automated Speech Recognition (ASR), (2) hybrid AI for question identification, (3) classifier to choose between the two systems, and (4) automated speech generation. We analyzed the accuracy of the ASR, the two AI systems, the classifier, and student feedback with 620 first year medical students from 2018 to 2021. RESULTS: System accuracy improved from ∼75% in 2018 to ∼90% in 2021 as refinements in algorithms and additional training data were utilized. Student feedback was positive, and most students felt that practicing with the VSPs was a worthwhile experience. CONCLUSION: We have developed a novel hybrid dialogue system that enables artificially intelligent VSPs to correctly answer student questions at levels comparable with human SPs. This system allows trainees to practice and refine their history-taking skills before interacting with human patients.

Using virtual standardized patients to accurately assess information gathering skills in medical students.

Introduction: Practicing a medical history using standardized patients is an essential component of medical school curricula. Recent advances in technology now allow for newer approaches for practicing and assessing communication skills. We describe herein a virtual standardized patient (VSP) system that allows students to practice their history taking skills and receive immediate feedback. Methods: Our VSPs consist of artificially intelligent, emotionally responsive 3D characters which communicate with students using natural language. The system categorizes the input questions according to specific domains and summarizes the encounter. Automated assessment by the computer was compared to manual assessment by trained raters to assess accuracy of the grading system. Results: Twenty dialogs chosen randomly from 102 total encounters were analyzed by three human and one computer rater. Overall scores calculated by the computer were not different than those provided by the human raters, and overall accuracy of the computer system was 87%, compared with 90% for human raters. Inter-rater reliability was high across 19 of 21 categories. Conclusions: We have developed a virtual standardized patient system that can understand, respond, categorize, and assess student performance in gathering information during a typical medical history, thus enabling students to practice their history-taking skills and receive immediate feedback.