Assessing ChatGPT's Mastery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study.

BACKGROUND: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to "hallucinations" (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom's taxonomy. OBJECTIVE: This study aims to explore how GPT-4 performs in terms of Bloom's taxonomy using psychosomatic medicine exam questions. METHODS: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom's taxonomy. RESULTS: GPT-4's performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4's lowest exam performance was 78.9% (15/19), thereby always surpassing the "pass" threshold. Our qualitative analysis of incorrect answers, based on Bloom's taxonomy, showed that errors were primarily in the "remember" (29/68) and "understand" (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. CONCLUSIONS: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom's taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

Attitudes Toward e-Health: The Otolaryngologists' Point of View.

INTRODUCTION: Online communication and the number of e-health applications have noticeably increased. However, little is known about the otolaryngologists' use behavior and their attitudes toward the potential of e-health. The aims of the study were to evaluate the documentation, information, and communication technologies used by otolaryngologists and to get a better understanding of their attitudes toward the potential of e-health for cross-sectoral patient care. METHODS: A survey was developed and tested by otolaryngologists, healthcare-information technology experts, and health services researchers. A total of 334 otolaryngologists in private practice were asked to participate in this cross-sectional study. In total, 234 of them took part in the study, and 157 returned completed questionnaires. Statistical analysis was performed by using crosstabs, including chi-square tests, and multivariate logistic regressions. Results and Materials: Digital technologies are widely used by otolaryngologists (e.g., 89.6% use an electronic health record). However, the majority of intersectoral communication is still based on analogue techniques (e.g., fax use in 63.7%). From the otolaryngologists' perspectives, the potential of e-health for intersectoral care is mostly in appointment scheduling, further referrals to hospitals, and automated appointment reminders. The physicians' attitudes toward e-health are associated with their Internet use behavior in daily life (odds ratio = 4.30, confidence interval 1.11-16.64, p = 0.035) but not with their demographics. DISCUSSION: The otolaryngologists are well prepared and have an overall positive attitude toward e-health for deeper use in cross-sectoral care. Therefore, e-health in otolaryngology needs more attention and resources for further studies, especially with a focus on quality and safety of care.

Communication skills of medical students during the OSCE: Gender-specific differences in a longitudinal trend study.

BACKGROUND: Communication skills are essential in a patient-centred health service and therefore in medical teaching. Although significant differences in communication behaviour of male and female students are known, gender differences in the performance of students are still under-reported. The aim of this study was to analyse gender differences in communication skills of medical students in the context of an OSCE exam (OSCE = Objective Structured Clinical Examination). METHODS: In a longitudinal trend study based on seven semester-cohorts, it was analysed if there are gender differences in medical students' communication skills. The students (self-perception) and standardized patients (SP) (external perception) were asked to rate the communication skills using uniform questionnaires. Statistical analysis was performed by using frequency analyses and t-tests in SPSS 21. RESULTS: Across all ratings in the self- and the external perception, there was a significant gender difference in favour of female students performing better in the dimensions of empathy, structure, verbal expression and non-verbal expression. The results of male students deteriorated across all dimensions in the external perception between 2011 and 2014. DISCUSSION & CONCLUSION: It is important to consider if gender-specific teaching should be developed, considering the reported differences between female and male students.

A Generative Pretrained Transformer (GPT)-Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study.

BACKGROUND: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions. OBJECTIVE: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. METHODS: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). RESULTS: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). CONCLUSIONS: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.

Embracing digital health: German otolaryngology patients' usage and prospects of digital information and communication technologies for cross-sectoral care.

Objective: The usage of digital information and communication technologies in European healthcare is growing. Unlike numerous technological possibilities, the present use of these technologies and perspectives towards them in relation to otolaryngology care have so far been of less interest. This study evaluates the utilisation of and attitudes towards digital information and communication technologies in cross-sectoral otolaryngology care among German patients. Methods: A structured interview-based study was conducted at the outpatient facility of a tertiary hospital in Germany. It focused on chief complaints, current use of digital technologies, estimated benefits of increased digital technology use in otolaryngology care, and sociodemographic data. The detailed statistical analysis employed Chi-squared tests and multivariate logistic regression. Results: A total of 208 otolaryngology patients completed the interview. Digital communication technologies exhibited a high penetration rate (91.8%) and were regularly used in daily life (78.7%) and for health reasons (73.3%). Younger age (p ≤ 0.003) and higher education levels (p ≤ 0.008) were significantly correlated with the increased digital communication technology use. The overall potential of eHealth technologies was rated significantly higher by younger patients (p ≤ 0.001). The patients' chief complaints showed no significant influence on the current and potential use of these technologies for cross-sectoral otolaryngology care. Conclusion: Regardless of their chief complaints, German otolaryngology patients regularly use digital information and communication technologies for health reasons and express interest in their further use for cross-sectoral care. To enhance digital patient communication in otolaryngology, attention should be given to treatment quality, usability, data security and availability and financial remuneration for service providers.

Communication training for advanced medical students improves information recall of medical laypersons in simulated informed consent talks--a randomized controlled trial.

BACKGROUND: Informed consent talks are mandatory before invasive interventions. However, the patients' information recall has been shown to be rather poor. We investigated, whether medical laypersons recalled more information items from a simulated informed consent talk after advanced medical students participated in a communication training aiming to reduce a layperson's cognitive load. METHODS: Using a randomized, controlled, prospective cross-over-design, 30 5th and 6th year medical students were randomized into two groups. One group received communication training, followed by a comparison intervention (early intervention group, EI); the other group first received the comparison intervention and then communication training (late intervention group, LI). Before and after the interventions, the 30 medical students performed simulated informed consent talks with 30 blinded medical laypersons using a standardized set of information. We then recorded the number of information items the medical laypersons recalled. RESULTS: After the communication training both groups of medical laypersons recalled significantly more information items (EI: 41 ± 9% vs. 23 ± 9%, p < .0001, LI 49 ± 10% vs. 35 ± 6%, p < .0001). After the comparison intervention the improvement was modest and significant only in the LI (EI: 42 ± 9% vs. 40 ± 9%, p = .41, LI 35 ± 6% vs. 29 ± 9%, p = .016). CONCLUSION: Short communication training for advanced medical students improves information recall of medical laypersons in simulated informed consent talks.

Does doctors' workload impact supervision and ward activities of final-year students? A prospective study.

BACKGROUND: Hospital doctors face constantly increasing workloads. Besides caring for patients, their duties also comprise the education of future colleagues. The aim of this study was to objectively investigate whether the workload arising from increased patient care interferes with student supervision and is associated with more non-medical activities of final-year medical students. METHODS: A total of 54 final-year students were asked to keep a diary of their daily activities over a three-week period at the beginning of their internship in Internal Medicine. Students categorized their activities--both medical and non-medical--according to whether they had: (1) only watched, (2) assisted the ward resident, (3) performed the activity themselves under supervision of the ward resident, or (4) performed the activity without supervision. The activities reported on a particular day were matched with a ward specific workload-index derived from the hospital information system, including the number of patients treated on the corresponding ward on that day, a correction factor according to the patient comorbidity complexity level (PCCL), and the number of admissions and discharges. Both students and ward residents were blinded to the study question. RESULTS: A total of 32 diaries (59 %, 442 recorded working days) were handed back. Overall, the students reported 1.2 ± 1.3 supervised, 1.8 ± 1.6 medical and 3.6 ± 1.7 non-medical activities per day. The more supervised activities were reported, the more the number of reported medical activities increased (p < .0001). No relationship between the ward specific workload and number of medical activities could be shown. CONCLUSIONS: There was a significant association between ward doctors' supervision of students and the number of medical activities performed by medical students. The workload had no significant effect on supervision or the number of medical or non-medical activities of final-year students.

"We Want Good Education for All of Us" - A Participatory Quality Improvement Approach.

Introduction: In ever changing conditions, medical faculties must face the challenge of preparing their medical students as best as possible for the demands of their future work. This requires involving all stakeholders, especially medical students in the constant redefinition of medical curricula. Using the idea of "Communities of Practice" as conceptual framework, this study looks at semester spokespeople as an example for participatory quality management. Methods: We conducted focus-group interviews with semester spokespeople at a German Medical Faculty. Data was recorded, transcribed, and analysed using MAXQDA. The interviews were analysed using meaning condensation method. Results: Eleven out of 48 semester spokespeople took part. We found seven topics that fell within three main categories: (1) role of the semester spokesperson, (2) role of the fixed meeting, and (3) contact and commitment. Communities of Practice principles could be aligned to topics and categories. Discussion: The idea of semester spokespeople based on the concept of Communities of Practice are useful in the quality management processes of a medical school and lead to greater involvement of medical students, identifying their needs. The reciprocal commitment among all stakeholders fosters mutual understanding and collaboration. Future studies could investigate the underlying motivational factors of dedicated students and how to transfer these characteristics to a larger cohort.

Evaluation of a Language Translation App in an Undergraduate Medical Communication Course: Proof-of-Concept and Usability Study.

BACKGROUND: Language barriers in medical encounters pose risks for interactions with patients, their care, and their outcomes. Because human translators, the gold standard for mitigating language barriers, can be cost- and time-intensive, mechanical alternatives such as language translation apps (LTA) have gained in popularity. However, adequate training for physicians in using LTAs remains elusive. OBJECTIVE: A proof-of-concept pilot study was designed to evaluate the use of a speech-to-speech LTA in a specific simulated physician-patient situation, particularly its perceived usability, helpfulness, and meaningfulness, and to assess the teaching unit overall. METHODS: Students engaged in a 90-min simulation with a standardized patient (SP) and the LTA iTranslate Converse. Thereafter, they rated the LTA with six items-helpful, intuitive, informative, accurate, recommendable, and applicable-on a 7-point Likert scale ranging from 1 (don't agree at all) to 7 (completely agree) and could provide free-text responses for four items: general impression of the LTA, the LTA's benefits, the LTA's risks, and suggestions for improvement. Students also assessed the teaching unit on a 6-point scale from 1 (excellent) to 6 (insufficient). Data were evaluated quantitatively with mean (SD) values and qualitatively in thematic content analysis. RESULTS: Of 111 students in the course, 76 (68.5%) participated (59.2% women, age 20.7 years, SD 3.3 years). Values for the LTA's being helpful (mean 3.45, SD 1.79), recommendable (mean 3.33, SD 1.65) and applicable (mean 3.57, SD 1.85) were centered around the average of 3.5. The items intuitive (mean 4.57, SD 1.74) and informative (mean 4.53, SD 1.95) were above average. The only below-average item concerned its accuracy (mean 2.38, SD 1.36). Students rated the teaching unit as being excellent (mean 1.2, SD 0.54) but wanted practical training with an SP plus a simulated human translator first. Free-text responses revealed several concerns about translation errors that could jeopardize diagnostic decisions. Students feared that patient-physician communication mediated by the LTA could decrease empathy and raised concerns regarding data protection and technical reliability. Nevertheless, they appreciated the LTA's cost-effectiveness and usefulness as the best option when the gold standard is unavailable. They also reported wanting more medical-specific vocabulary and images to convey all information necessary for medical communication. CONCLUSIONS: This study revealed the feasibility of using a speech-to-speech LTA in an undergraduate medical course. Although human translators remain the gold standard, LTAs could be valuable alternatives. Students appreciated the simulated teaching and recognized the LTA's potential benefits and risks for use in real-world clinical settings. To optimize patients' and health care professionals' experiences with LTAs, future investigations should examine specific design options for training interventions and consider the legal aspects of human-machine interaction in health care settings.

Attitude and potential benefits of modern information and communication technology use and telemedicine in cross-sectoral solid organ transplant care.

Situations like the COVID-19 pandemic urgently require the implementation of eHealth for vulnerable patient populations. Here we quantitatively evaluate use and potential of modern information and communication technology (ICT) in solid organ transplant (SOT) recipients. We conducted a structured, questionnaire-based, cross-sectional study that was addressed to patients after kidney, liver, pancreas, or combined transplantation. We focused on: sociodemographic data, present use of digital technologies in daily life and for health reasons, patients' eHealth literacy, and their overall attitude towards eHealth. A total of 234 patients completed the questionnaire. Most of the patients (90%) have a web-enabled computer, 78.2% have a smartphone, and 71.8% regularly search the internet for health-related information. Sixty-eight percent would like to receive discharge summaries online, and 54% would like to chat online with their physicians. Even though ICT use in daily life was age-related, no significant difference could be shown for health reasons or the type of transplanted organ. Modern ICT use is predominantly accepted for health reasons by SOT recipients. Regardless of the transplanted organ, a deeper integration of eHealth has potential for improving cross-sectoral care. To successfully implement eHealth technologies in cross-sectoral care future research should include online physician-patient communication, data security, data safety, and the aspects of quality and safety of care.

Electronic charts do not facilitate the recognition of patient hazards by advanced medical students: A randomized controlled study.

Chart review is an important tool to identify patient hazards. Most advanced medical students perform poorly during chart review but can learn how to identify patient hazards context-independently. Many hospitals have implemented electronic health records, which enhance patient safety but also pose challenges. We investigated whether electronic charts impair advanced medical students' recognition of patient hazards compared with traditional paper charts. Fifth-year medical students were randomized into two equal groups. Both groups attended a lecture on patient hazards and a training session on handling electronic health records. One group reviewed an electronic chart with 12 standardized patient hazards and then reviewed another case in a paper chart; the other group reviewed the charts in reverse order. The two case scenarios (diabetes and gastrointestinal bleeding) were used as the first and second case equally often. After each case, the students were briefed about the patient safety hazards. In total, 78.5% of the students handed in their notes for evaluation. Two blinded raters independently assessed the number of patient hazards addressed in the students' notes. For the diabetes case, the students identified a median of 4.0 hazards [25%-75% quantiles (Q25-Q75): 2.0-5.5] in the electronic chart and 5.0 hazards (Q25-Q75: 3.0-6.75) in the paper chart (equivalence testing, p = 0.005). For the gastrointestinal bleeding case, the students identified a median of 5.0 hazards (Q25-Q75: 4.0-6.0) in the electronic chart and 5.0 hazards (Q25-Q75: 3.0-6.0) in the paper chart (equivalence testing, p < 0.001). We detected no improvement between the first case [median 5.0 (Q25-Q75: 3.0-6.0)] and second case [median, 5.0 (Q25-Q75: 3.0-6.0); p < 0.001, test for equivalence]. Electronic charts do not seem to facilitate advanced medical students' recognition of patient hazards during chart review and may impair expertise formation.

Navigating Through Electronic Health Records: Survey Study on Medical Students' Perspectives in General and With Regard to a Specific Training.

BACKGROUND: An electronic health record (EHR) is the state-of-the-art method for ensuring all data concerning a given patient are up to date for use by multidisciplinary hospital teams. Therefore, medical students need to be trained to use health information technologies within this environment from the early stages of their education. OBJECTIVE: As little is known about the effects of specific training within the medical curriculum, this study aimed to develop a course module and evaluate it to offer best practice teaching for today's students. Moreover, we looked at the acceptance of new technologies such as EHRs. METHODS: Fifth-year medical students (N=104) at the University of Tübingen took part in a standardized two-day training procedure about the advantages and risks of EHR use. After the training, students performed their own EHR entries on hypothetical patient cases in a safe practice environment. In addition, questionnaires-standardized and with open-ended questions-were administered to assess students' experiences with a new teaching module, a newly developed EHR simulator, the acceptance of the health technology, and their attitudes toward it before and after training. RESULTS: After the teaching, students rated the benefit of EHR training for medical knowledge significantly higher than before the session (mean 3.74, SD 1.05). However, they also had doubts about the long-term benefit of EHRs for multidisciplinary coworking after training (mean 1.96, SD 0.65). The special training with simulation software was rated as helpful for preparing students (88/102, 86.2%), but they still did not feel safe in all aspects of EHR. CONCLUSIONS: A specific simulated training on using EHRs helped students improve their knowledge and become more aware of the risks and challenges of such a system. Overall, students welcomed the new training module and supported the integration of EHR teaching into the medical curriculum. Further studies are needed to optimize training modules and make use of long-term feedback opportunities a simulated system offers.

Portfolios Enhance Clinical Activity in Surgical Clerks.

OBJECTIVES: A change in German licensing legislation imposed a portfolio for surgical clerks. We aimed to analyze whether the implementation of the portfolio changed the amount of clinical exposure and activities during surgical clerkships. DESIGN: The study was conducted with a modified pre-post design at the University Hospital of Tuebingen, Germany. Before and after the implementation of the portfolio on April 1, 2013, final-year students (n = 557) who had just finished their surgical clerkship were interviewed with an online questionnaire. A total of 21 basic surgical skills were evaluated. RESULTS: Overall, 230 questionnaires were returned and analyzed; 51% were preintervention. Overall clinical activity for the whole study cohort varied for different activities between 98% and 32%. For 16 of 21 parameters, there was more clinical activity in the postintervention (portfolio) group. This difference was statistically significant for the following 7 activities: discharge, analgesia, local infiltration, patient positioning, drain in, blood transfusion, and emergency diagnostics. CONCLUSION: The implementation of the portfolio did enhance clinical activity for surgical clerks in the study cohort. Nevertheless, overall exposure is still unsatisfactory low for some activities. Additional changes and studies are necessary to further improve surgical education.

Problem-based training improves recognition of patient hazards by advanced medical students during chart review: a randomized controlled crossover study.

BACKGROUND: Patient chart review is the gold standard for detection of potential patient hazards (i.e. medication errors or failure to follow up actionable results) in both routine clinical care and patient safety research. However, advanced medical students' ability to read patient charts and to identify patient hazards is rather poor. We therefore investigated whether it is possible to teach advanced medical students how to identify patient hazards independent of context (i.e. cancer versus cardiac failure) in patient charts. METHODS: All fifth-year medical students in one semester (n = 123) were randomized into two groups. One group (IC) received a patient chart review-training first and then a control-intervention and the other group (CI) received the control-intervention first and then the patient chart review-training. Before and after the teaching sessions, students reviewed different scenarios with standardized fictional patient charts containing 12 common patient hazards. Two blinded raters rated the students' notes for any patient hazard addressed in the notes using a checklist. The students were blinded to the study question and design. There was no external funding and no harm for the participating students. RESULTS: A total of 35 data sets had to be excluded because of missing data. Overall, the students identified 17% (IQR 8-29%) of the patient hazards before the training and 56% (IQR 41-66%) of the patient hazards after the training. At the second assessment students identified more patient hazards than at the first. They identified even more in the third. The effect was most pronounced after the patient chart review training (all p<.01). CONCLUSION: Patient chart review exercises and problem-based patient chart review training improve students' abilities to recognize patient hazards independent of context during patient chart review.

Approaches to family-friendliness at the Medical Faculty of Tübingen (MFT).

The feminization of the medical profession, demographic change with an impending shortage of physicians, Generation Y - these issues are new challenges for medical schools in terms of their social responsibility and the training of the next generation of highly qualified scientists. This study, conducted by the University Hospital of Ulm throughout Baden-Württemberg via an online survey provides a valuable data basis which can be used to optimise support activities. A disproportionately high percentage of students with children in Tübingen must, as an additional challenge, cope with a lower than average monthly income. Students with children need organisational support in this doubly challenging situation. The Dean's Office can provide valuable assistance as an advocate (contact with the departments/institutions), provide organisational help (course guidance, individual support) and infrastructure (childcare/changing rooms, etc.). In Tübingen, high intrinsic motivation (willingness to engage in self-managed care networks) is also found amongst students, something that needs to be integrated into the development of the existing support networks.

Effect of supervised students' involvement on diagnostic accuracy in hospitalized medical patients--a prospective controlled study.

BACKGROUND: During internships most medical students engage in history taking and physical examination during evaluation of hospitalized patients. However, the students' ability for pattern recognition is not as developed as in medical experts and complete history taking is often not repeated by an expert, so important clues may be missed. On the other hand, students' history taking is usually more extensive than experts' history taking and medical students discuss their findings with a Supervisor. Thus the effect of student involvement on diagnostic accuracy is unclear. We therefore compared the diagnostic accuracy for patients in the medical emergency department with and without student involvement in the evaluation process. METHODOLOGY/PRINCIPAL FINDINGS: Patients in the medical emergency department were assigned to evaluation by either a supervised medical student or an emergency department physician. We only included patients who were admitted to our hospital and subsequently cared for by another medical team on the ward. We compared the working diagnosis from the emergency department with the discharge diagnosis. A total of 310 patients included in the study were cared for by 41 medical students and 21 emergency department physicians. The working diagnosis was changed in 22% of the patients evaluated by physicians evaluation and in 10% of the patients evaluated by supervised medical students (p = .006). There was no difference in the expenditures for diagnostic procedures, length of stay in the emergency department or patient comorbidity complexity level. CONCLUSION/SIGNIFICANCE: Involvement of closely supervised medical students in the evaluation process of hospitalized medical patients leads to an improved diagnostic accuracy compared to evaluation by an emergency department physician alone.