Assessing ChatGPT's Mastery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study.

BACKGROUND: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to "hallucinations" (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom's taxonomy. OBJECTIVE: This study aims to explore how GPT-4 performs in terms of Bloom's taxonomy using psychosomatic medicine exam questions. METHODS: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom's taxonomy. RESULTS: GPT-4's performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4's lowest exam performance was 78.9% (15/19), thereby always surpassing the "pass" threshold. Our qualitative analysis of incorrect answers, based on Bloom's taxonomy, showed that errors were primarily in the "remember" (29/68) and "understand" (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. CONCLUSIONS: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom's taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

Submitting artificial intelligence in health professions education papers to medical teacher.

As any field evolves, so do journals' expectations from authors. As Artificial Intelligence (AI) usage in Higher Professions Education (HPE) has evolved, Medical Teacher's expectations have changed, and previously-accepted paper types are now routinely rejected. This commentary gives some guidance for authors currently submitting AI in HPE papers to Medical Teacher.

A checklist for reporting, reading and evaluating Artificial Intelligence Technology Enhanced Learning (AITEL) research in medical education.

Advances in Artificial Intelligence (AI) have led to AI systems' being used increasingly in medical education research. Current methods of reporting on the research, however, tend to follow patterns of describing an intervention and reporting on results, with little description of the AI in the system, or the many concerns about the use of AI. In essence, the readers do not actually know anything about the system itself. This paper proposes a checklist for reporting on AI systems, and covers the initial protocols and scoping, modelling and code, algorithm design, training data, testing and validation, usage, comparisons, real-world requirements, results and limitations, and ethical considerations. The aim is to have a systematic reporting process so that readers can have a comprehensive understanding of the AI system that was used in the research.

Online learning in Health Professions Education. Part 1: Teaching and learning in online environments: AMEE Guide No. 161.

Online learning in Health Professions Education (HPE) has been evolving over decades, but COVID-19 changed its use abruptly. Technology allowed necessary HPE during COVID-19, but also demonstrated that many HP educators and learners had little knowledge and experience of these complex sociotechnical environments. Due to the educational benefits and flexibility that technology can afford, many higher education experts agree that online learning will continue and evolve long after COVID-19. As HP educators stand at the crossroads of technology integration, it is important that we examine the evidence, theories, advantages/disadvantages, and pedagogically informed design of online learning. This Guide will provide foundational concepts and practical strategies to support HPE educators and institutions toward advancing pedagogically informed use of online HPE. This Guide consists of two parts. The first part will provide an overview of evidence, theories, formats, and educational design in online learning, including contemporary issues and considerations such as learner engagement, faculty development, inclusivity, accessibility, copyright, and privacy. The second part (to be published as a separate Guide) focuses on specific technology tool types with practical examples for implementation and integration of the concepts discussed in Guide 1, and will include digital scholarship, learning analytics, and emerging technologies. In sum, both guides should be read together, as Guide 1 provides the foundation required for the practical application of technology showcased in Guide 2.Please refer to the video abstract for Part 1 of this Guide at https://bit.ly/AMEEGuideOnlineLearning.

Online learning in health professions education. Part 2: Tools and practical application: AMEE Guide No. 163.

Part 1 of the AMEE Guide Online learning in health professions education focused on foundational concepts such as theory, methods, and instructional design in online learning. Part 2 builds upon Part 1, introducing technology tools and applications of these foundational concepts by exploring the various levels (from beginner to advanced) of utilisation, while describing how their usage can transform Health Professions Education. This Part covers Learning Management Systems, infographics, podcasting, videos, websites, social media, online discussion forums, simulation, virtual patients, extended and virtual reality. Intertwined are other topics, such as online small group teaching, game-based learning, FOAM, online social and collaboration learning, and virtual care teaching. We end by discussing digital scholarship and emerging technologies. Combined with Part 1, the overall aim of Part 2 is to produce a comprehensive overview to help guide effective use online learning in Health Professions Education.

Twelve tips on creating and using custom GPTs to enhance health professions education.

The custom GPT is the latest powerful feature added to ChatGPT. Non-programmers can create and share their own GPTs ("chat bots"), allowing Health Professions Educators to apply the capabilities of ChatGPT to create administrative assistants, online tutors, virtual patients, and more, to support their clinical and non-clinical teaching environments. To achieve this correctly, however, requires some skills, and this 12-Tips paper provides those: we explain how to construct data sources, build relevant GPTs, and apply some basic security.

Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide No. 158.

Health Professions Education (HPE) has benefitted from the advances in Artificial Intelligence (AI) and is set to benefit more in the future. Just as any technological advance opens discussions about ethics, so the implications of AI for HPE ethics need to be identified, anticipated, and accommodated so that HPE can utilise AI without compromising crucial ethical principles. Rather than focussing on AI technology, this Guide focuses on the ethical issues likely to face HPE teachers and administrators as they encounter and use AI systems in their teaching environment. While many of the ethical principles may be familiar to readers in other contexts, they will be viewed in light of AI, and some unfamiliar issues will be introduced. They include data gathering, anonymity, privacy, consent, data ownership, security, bias, transparency, responsibility, autonomy, and beneficence. In the Guide, each topic explains the concept and its importance and gives some indication of how to cope with its complexities. Ideas are drawn from personal experience and the relevant literature. In most topics, further reading is suggested so that readers may further explore the concepts at their leisure. The aim is for HPE teachers and decision-makers at all levels to be alert to these issues and to take proactive action to be prepared to deal with the ethical problems and opportunities that AI usage presents to HPE.

Medical Teacher's first ChatGPT's referencing hallucinations: Lessons for editors, reviewers, and teachers.

Students' inappropriate use of ChatGPT is a concern. There is also, however, the potential for academics to use ChatGPT inappropriately. After explaining ChatGPT's "hallucinations" regarding citing and referencing, this commentary illustrates the problem by describing the detection of the first known Medical Teacher submission using ChatGPT inappropriately, the lessons that can be drawn from it for journal editors, reviewers, and teachers, and then the wider implications if this problem is left unchecked.

The effect of watching lecture videos at 2× speed on memory retention performance of medical students: An experimental study.

AIM: This study aimed to determine how watching lecture videos at 1× and 2× speeds affects memory retention in medical students. METHODS: A posttest-only experimental design was utilized. The participants were 60 Year-1 and Year-2 medical students. The participants were assigned to one of two groups through stratified randomization: Group 1 would watch the video at 1× speed, and Group 2 at 2× speed. Their performance was assessed using a test comprising 20 multiple-choice questions. The test has been applied immediately after watching the video (Immediate test), and, again after one week (Delayed test). Parametric and non-parametric statistical tests were performed. RESULTS: In the Immediate test, the mean score of the 1× speed group was 11.26 ± 4.06, while 2× speed group's mean score was 10.16 ± 2.46. The difference was not significant t(58) = 1.26, p > .05. In the Delayed test, the mean score of 1× speed group was 9.66 ± 3.94, while 2× speed group's mean score was 8.36 ± 2.80. The difference was not significant t(55) = 1.42, p > .05. CONCLUSIONS: Watching the video lecture at 2× speed did not impair memory retention in medical students. This may help students to save time in their dense curricula.[Box: see text].

AMEE Guide to ethical teaching in online medical education: AMEE Guide No. 146.

The Covid-19 pandemic necessitated Emergency Remote Teaching (ERT): the sudden move of educational materials online. While ERT served its purpose, medical teachers are now faced with the long-term and complex demands of formal online teaching. One of these demands is ethical online teaching. Although ethical teaching is practiced in face-to-face situations, online teaching has new ethical issues that must be accommodated, and medical teachers who wish to teach online must be aware of these and need to teach ethically. This Guide leads the medical teacher through this maze of complex ethical issues to transform ERT into ethical online teaching. It begins by setting the context and needs and identifies the relevant fundamental ethical principles and issues. It then guides the medical teacher through the practical application of these ethical principles, covering course design and layout (including the curriculum document, implementation, on-screen layouts, material accessibility), methods of interaction (synchronous and asynchronous), feedback, supervision and counselling, deeper accessibility issues, issues specific to clinical teaching, and assessment. It then discusses course reviews (peer-review and student evaluations), student monitoring and analytics, and archiving. The Guide aims to be a useful tool for medical teachers to solidly ground their online teaching practices in ethical principles.

Edgar Dale's Pyramid of Learning in medical education: Further expansion of the myth.

INTRODUCTION: A mythical Pyramid of Learning, usually attributed to Edgar Dale (or the National Training Laboratories [NTL]) and giving student learning retention rates, has been cited in a wide range of educational literature. A 2013 literature review indicated that medical education literature similarly cites this Pyramid. It was hoped that highlighting this myth in that review would reduce references to the Pyramid in future medical education literature. This study aimed at determining what change in Pyramid citation has occurred in the past 5 years. METHODS: A documented literature review, following the same process as the original review, was conducted. The search dates were September 2012 to April 2018, and the databases were Academic Search Complete, CINAHL, Medline and Google Scholar. Sources were from peer-reviewed journals or conferences, in English. RESULTS: From an initial search result of 992 documents, 41 were found to match the criteria. Trends discovered are: the number of Pyramid citations in medical education literature is increasing dramatically, new sources of the Pyramid are now being used, refutations of the Pyramid are being used to support it, and even researchers who acknowledge the weakness of the Pyramid still cite it. DISCUSSION AND CONCLUSION: In spite of the 2013 review, the situation has become worse. One possible reason is that refutations use too polite academic wording, and other researchers then consider the Pyramid to be merely "disputed" or "debated." To kill the myth of the Pyramid, it is necessary for this article's Abstract to state unequivocally: The Pyramid is rubbish, the statistics are rubbish, and they do not come from Edgar Dale. Until the NTL can provide details about the original research, their version must also be treated as rubbish.

Surgeons' Interactions With and Attitudes Toward E-Patients: Questionnaire Study in Germany and Oman.

BACKGROUND: Doctors' interactions with and attitudes toward e-patients have an overall impact on health care delivery. OBJECTIVE: This study aimed to gauge surgeons' interactions with e-patients, their attitudes toward those e-patient activities, the possible impact on the delivery of health care, and the reasons behind those activities and attitudes. METHODS: We created a paper-based and electronic survey form based on pertinent variables identified in the literature, and from March 2018 to July 2018 we surveyed 49 surgeons in Germany and 59 surgeons in Oman, asking them about their interactions with and attitudes toward e-patients. Data were stored in Microsoft Excel and SPSS, and descriptive statistics, Pearson correlations, and chi-square tests were performed on the data. RESULTS: Of our sample, 71% (35/49) of the German surgeons and 56% (33/59) of the Omani surgeons communicated electronically with their patients. Although the German surgeons spent a greater percentage of Internet usage time on work-related activities (χ218=32.5; P=.02) than the Omani surgeons, there were many similarities in their activities. An outstanding difference was that the German surgeons used email with their patients more than the Omani surgeons (χ21=9.0; P=.003), and the Omani surgeons used social media, specifically WhatsApp, more than the German surgeons (χ21=18.6; P<.001). Overall, the surgeons were equally positive about the most common e-patient activities such as bringing material from the internet to the consultation (mean 4.11, SD 1.6), although the German surgeons (mean 3.43, SD 1.9) were more concerned (P=.001) than the Omani surgeons (mean 2.32, SD 1.3) about the potential loss of control and time consumption (German: mean 5.10, SD 1.4 and Omani: mean 3.92, SD 1.6; P<.001). CONCLUSIONS: The interactions show a high degree of engagement with e-patients. The differences between the German and the Omani surgeons in the preferred methods of communication are possibly closely linked to cultural differences and recent historical events. These differences may, moreover, indicate e-patients' desired method of electronic communication to include social media. The low impact of surgeons' attitudes on the activities may also result from a normalization of many e-patient activities, irrespective of the doctors' attitudes and influences.

Ethics in medical education digital scholarship: AMEE Guide No. 134.

Ethics has long been a concern in medicine, education and scholarship. In the digital age, new complexities have arisen, and many medical education researchers are unprepared for the pitfalls ahead, often negotiating these in the absence of guidelines, and unaware of the many tools that can be used to assist them. This Guide takes the medical education scholar through a journey in which issues of ethics are discussed in all stages of digital scholarship: research preparation, research subject monitoring and data gathering, securing one's data (and balancing security against accessibility), anonymising textual and non-textual data, third party identifiability in digital data, writing one's own work (including plagiarism and paper mills), copyright (including issues of Creative Commons and royalty-free), accessing inaccessible reference material, ethically citing electronic material, and manuscript submission (including issues of selecting journals, open access and data sharing). The Guide ends with a brief look to the future. This Guide aims to be a useful tool to alert the readers to some of the most important ethical issues that need to be considered, and some practical solutions to ethical problems faced, when engaging in medical education digital scholarship.

Artificial intelligence in medical education.

Artificial intelligence (AI) is a growing phenomenon, and will soon facilitate wide-scale changes in many professions, including medical education. In order for medical educators to be properly prepared for AI, they will need to have at least a fundamental knowledge of AI in relation to learning and teaching, and the extent to which it will impact on medical education. This Guide begins by introducing the broad concepts of AI by using fairly well-known examples to illustrate AI's implications within the context of education. It then considers the impact of AI on medicine and the implications of this impact for educators trying to educate future doctors. Drawing on these strands, it then identifies AI's direct impact on the methodology and content of medical education, in an attempt to prepare medical educators for the changing demands and opportunities that are about to face them because of AI.

Preparing medical students for the e-patient.

The emergence of the e-patient has resulted in many medical practitioners' being ill-equipped to deal with the 21st-century medical practice. This Guide is a teaching guide for medical educators so that they can prepare their students for the new environment that has resulted from the emergence of the e-patient. Within the context of theoretical perspectives, the Guide begins by defining the concept, and examining the history of the e-patient, detailing typical e-patient activities and some complexities raised by these activities. Finally, the Guide details the topic areas that should be covered in a course aimed at preparing medical students for e-patients. The result is a theoretical and practical teaching Guide that equips medical teachers and their students with the necessary background information, and also assists teachers in the teaching of that information so that their students may become health practitioners fully equipped to deal with the problems and potential of the e-patient.

The e-patient and medical students.

The recent publicity around the tragic case of Bronte Doyne has highlighted a pressing need in healthcare delivery: the need for doctors to know that their patients, "e-patients," know medicine. In turn, this requires our medical students to be trained in how best to utilise the potential of e-patients in healthcare delivery. "I can't begin to tell you how it feels to have to tell an oncologist they are wrong, it's a young person's cancer. I had to, I'm fed up of trusting them." - Bronte Doyne (Vize 2015).

Mobile technologies in medical education: AMEE Guide No. 105.

Mobile technologies (including handheld and wearable devices) have the potential to enhance learning activities from basic medical undergraduate education through residency and beyond. In order to use these technologies successfully, medical educators need to be aware of the underpinning socio-theoretical concepts that influence their usage, the pre-clinical and clinical educational environment in which the educational activities occur, and the practical possibilities and limitations of their usage. This Guide builds upon the previous AMEE Guide to e-Learning in medical education by providing medical teachers with conceptual frameworks and practical examples of using mobile technologies in medical education. The goal is to help medical teachers to use these concepts and technologies at all levels of medical education to improve the education of medical and healthcare personnel, and ultimately contribute to improved patient healthcare. This Guide begins by reviewing some of the technological changes that have occurred in recent years, and then examines the theoretical basis (both social and educational) for understanding mobile technology usage. From there, the Guide progresses through a hierarchy of institutional, teacher and learner needs, identifying issues, problems and solutions for the effective use of mobile technology in medical education. This Guide ends with a brief look to the future.