A checklist for reporting, reading and evaluating Artificial Intelligence Technology Enhanced Learning (AITEL) research in medical education.

Advances in Artificial Intelligence (AI) have led to AI systems' being used increasingly in medical education research. Current methods of reporting on the research, however, tend to follow patterns of describing an intervention and reporting on results, with little description of the AI in the system, or the many concerns about the use of AI. In essence, the readers do not actually know anything about the system itself. This paper proposes a checklist for reporting on AI systems, and covers the initial protocols and scoping, modelling and code, algorithm design, training data, testing and validation, usage, comparisons, real-world requirements, results and limitations, and ethical considerations. The aim is to have a systematic reporting process so that readers can have a comprehensive understanding of the AI system that was used in the research.

Assessing ChatGPT's Mastery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study.

BACKGROUND: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to "hallucinations" (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom's taxonomy. OBJECTIVE: This study aims to explore how GPT-4 performs in terms of Bloom's taxonomy using psychosomatic medicine exam questions. METHODS: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom's taxonomy. RESULTS: GPT-4's performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4's lowest exam performance was 78.9% (15/19), thereby always surpassing the "pass" threshold. Our qualitative analysis of incorrect answers, based on Bloom's taxonomy, showed that errors were primarily in the "remember" (29/68) and "understand" (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. CONCLUSIONS: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom's taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

Twelve tips on creating and using custom GPTs to enhance health professions education.

The custom GPT is the latest powerful feature added to ChatGPT. Non-programmers can create and share their own GPTs ("chat bots"), allowing Health Professions Educators to apply the capabilities of ChatGPT to create administrative assistants, online tutors, virtual patients, and more, to support their clinical and non-clinical teaching environments. To achieve this correctly, however, requires some skills, and this 12-Tips paper provides those: we explain how to construct data sources, build relevant GPTs, and apply some basic security.

Online learning in Health Professions Education. Part 1: Teaching and learning in online environments: AMEE Guide No. 161.

Online learning in Health Professions Education (HPE) has been evolving over decades, but COVID-19 changed its use abruptly. Technology allowed necessary HPE during COVID-19, but also demonstrated that many HP educators and learners had little knowledge and experience of these complex sociotechnical environments. Due to the educational benefits and flexibility that technology can afford, many higher education experts agree that online learning will continue and evolve long after COVID-19. As HP educators stand at the crossroads of technology integration, it is important that we examine the evidence, theories, advantages/disadvantages, and pedagogically informed design of online learning. This Guide will provide foundational concepts and practical strategies to support HPE educators and institutions toward advancing pedagogically informed use of online HPE. This Guide consists of two parts. The first part will provide an overview of evidence, theories, formats, and educational design in online learning, including contemporary issues and considerations such as learner engagement, faculty development, inclusivity, accessibility, copyright, and privacy. The second part (to be published as a separate Guide) focuses on specific technology tool types with practical examples for implementation and integration of the concepts discussed in Guide 1, and will include digital scholarship, learning analytics, and emerging technologies. In sum, both guides should be read together, as Guide 1 provides the foundation required for the practical application of technology showcased in Guide 2.Please refer to the video abstract for Part 1 of this Guide at https://bit.ly/AMEEGuideOnlineLearning.

Online learning in health professions education. Part 2: Tools and practical application: AMEE Guide No. 163.

Part 1 of the AMEE Guide Online learning in health professions education focused on foundational concepts such as theory, methods, and instructional design in online learning. Part 2 builds upon Part 1, introducing technology tools and applications of these foundational concepts by exploring the various levels (from beginner to advanced) of utilisation, while describing how their usage can transform Health Professions Education. This Part covers Learning Management Systems, infographics, podcasting, videos, websites, social media, online discussion forums, simulation, virtual patients, extended and virtual reality. Intertwined are other topics, such as online small group teaching, game-based learning, FOAM, online social and collaboration learning, and virtual care teaching. We end by discussing digital scholarship and emerging technologies. Combined with Part 1, the overall aim of Part 2 is to produce a comprehensive overview to help guide effective use online learning in Health Professions Education.

Medical Teacher's first ChatGPT's referencing hallucinations: Lessons for editors, reviewers, and teachers.

Students' inappropriate use of ChatGPT is a concern. There is also, however, the potential for academics to use ChatGPT inappropriately. After explaining ChatGPT's "hallucinations" regarding citing and referencing, this commentary illustrates the problem by describing the detection of the first known Medical Teacher submission using ChatGPT inappropriately, the lessons that can be drawn from it for journal editors, reviewers, and teachers, and then the wider implications if this problem is left unchecked.

Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide No. 158.

Health Professions Education (HPE) has benefitted from the advances in Artificial Intelligence (AI) and is set to benefit more in the future. Just as any technological advance opens discussions about ethics, so the implications of AI for HPE ethics need to be identified, anticipated, and accommodated so that HPE can utilise AI without compromising crucial ethical principles. Rather than focussing on AI technology, this Guide focuses on the ethical issues likely to face HPE teachers and administrators as they encounter and use AI systems in their teaching environment. While many of the ethical principles may be familiar to readers in other contexts, they will be viewed in light of AI, and some unfamiliar issues will be introduced. They include data gathering, anonymity, privacy, consent, data ownership, security, bias, transparency, responsibility, autonomy, and beneficence. In the Guide, each topic explains the concept and its importance and gives some indication of how to cope with its complexities. Ideas are drawn from personal experience and the relevant literature. In most topics, further reading is suggested so that readers may further explore the concepts at their leisure. The aim is for HPE teachers and decision-makers at all levels to be alert to these issues and to take proactive action to be prepared to deal with the ethical problems and opportunities that AI usage presents to HPE.

The effect of watching lecture videos at 2× speed on memory retention performance of medical students: An experimental study.

AIM: This study aimed to determine how watching lecture videos at 1× and 2× speeds affects memory retention in medical students. METHODS: A posttest-only experimental design was utilized. The participants were 60 Year-1 and Year-2 medical students. The participants were assigned to one of two groups through stratified randomization: Group 1 would watch the video at 1× speed, and Group 2 at 2× speed. Their performance was assessed using a test comprising 20 multiple-choice questions. The test has been applied immediately after watching the video (Immediate test), and, again after one week (Delayed test). Parametric and non-parametric statistical tests were performed. RESULTS: In the Immediate test, the mean score of the 1× speed group was 11.26 ± 4.06, while 2× speed group's mean score was 10.16 ± 2.46. The difference was not significant t(58) = 1.26, p > .05. In the Delayed test, the mean score of 1× speed group was 9.66 ± 3.94, while 2× speed group's mean score was 8.36 ± 2.80. The difference was not significant t(55) = 1.42, p > .05. CONCLUSIONS: Watching the video lecture at 2× speed did not impair memory retention in medical students. This may help students to save time in their dense curricula.[Box: see text].

Can Medical Students Evaluate Medical Websites?: A mixed-methods study from Oman.

Objectives: This study aimed to discover the extent to which medical students can evaluate medical websites, evaluation criteria used, factors affecting their abilities and whether a teaching intervention could rectify problems. Medical students and practitioners are required to evaluate medical information available on the Internet. Most current medical students are familiar with the Internet, but their ability to evaluate material may require improvement. Methods: A class of undergraduate medical students evaluated an unreliable medical website, received a teaching intervention on website evaluation criteria and re-evaluated the same site. This mixed-methods study was conducted at Sultan Qaboos University, Muscat, Oman, from September to December 2018. Results: A total of 149 (response rate: 82.3%) students participated. Students spent, on average, 4.69 hours per day on the Internet. No significant correlations were found between demographic indicators and Internet time. On a 10-point Likert scale, students' scores ranged from 5-6, with no significant differences between the pre- and post-intervention evaluations, except for increased polarisation away from the mean. Qualitative comments indicated an awareness of relevant criteria but an overall inability to critically apply them. Conclusion: The results indicate that one cannot make a blanket statement about medical students' ability to evaluate medical websites despite their familiarity with technology. Moreover, website evaluation should be viewed primarily from the information perspective and that critical thinking ability may play a major role. Due to these overriding factors, short interventions are unlikely to have an impact, and other educational strategies should be developed. These are necessary to ensure that medical students can function independently as life-long learners and medical professionals.

AMEE Guide to ethical teaching in online medical education: AMEE Guide No. 146.

The Covid-19 pandemic necessitated Emergency Remote Teaching (ERT): the sudden move of educational materials online. While ERT served its purpose, medical teachers are now faced with the long-term and complex demands of formal online teaching. One of these demands is ethical online teaching. Although ethical teaching is practiced in face-to-face situations, online teaching has new ethical issues that must be accommodated, and medical teachers who wish to teach online must be aware of these and need to teach ethically. This Guide leads the medical teacher through this maze of complex ethical issues to transform ERT into ethical online teaching. It begins by setting the context and needs and identifies the relevant fundamental ethical principles and issues. It then guides the medical teacher through the practical application of these ethical principles, covering course design and layout (including the curriculum document, implementation, on-screen layouts, material accessibility), methods of interaction (synchronous and asynchronous), feedback, supervision and counselling, deeper accessibility issues, issues specific to clinical teaching, and assessment. It then discusses course reviews (peer-review and student evaluations), student monitoring and analytics, and archiving. The Guide aims to be a useful tool for medical teachers to solidly ground their online teaching practices in ethical principles.

Designing and developing an app to perform Hofstee cut-off calculations.

Determining a Hofstee cut-off point in medical education student assessment is problematic: traditional methods can be time-consuming, inaccurate, and inflexible.  To counter this, we developed a simple Android app that receives raw, unsorted student assessment data in .csv format, allows for multiple judges' inputs, mean or median inputs, calculates the Hofstee cut-off mathematically, and outputs the results with other guiding information. The app contains a detailed description of its functionality.

How Have Researchers Acknowledged and Controlled for Academic Work Activity When Measuring Medical Students' Internet Addiction? A Systematic Literature Review.

Internationally, medical students' Internet Addiction (IA) is widely studied. As medical students use the Internet extensively for work, we asked how researchers control for work-related Internet activity, and the extent to which this influences interpretations of "addiction" rates. A search of PubMed, CINAHL, Web of Science, Scopus, and Google Scholar was conducted on the search phrase of "medical students" and "internet addiction" in March 2020. In total, 98 studies met our criteria, 88 (90%) used Young's Internet Addiction Test, and the studies' IA rates ranged widely. Little note was taken of work-related activity, and, when discussed, had little to no impact on the interpretation of Internet "addiction". Studies seldom accounted for work-related activities, researcher bias appears to influence their position, "usage" appears conflated with "addiction", and correlations between "addiction" and negative behaviours are frequently confused with one-way causation. In spite of IA's not being officially recognised, few researchers questioned its validity. While IA may exist among medical students, its measurement is flawed; given the use of the Internet as a crucial medical education tool, there is the risk that conscientious students will be labelled "addicted", and poor academic performance may be attributed to this "addiction".

They Shoot Horses, Don't They? A warning to medical schools about medical teacher burnout during COVID-19.

This article was migrated. The article was marked as recommended. The 1969 film, They Shoot Horses, Don't They? provides a contextual background for what appears to be happening to medical teachers as they attempt to cope with teaching during the COVID-19 pandemic, staving off the threat of exhaustion. This short piece argues that it is necessary for medical education institutions to recognise the changing demands made on, and by, their teachers, so that they can prevent burnout, and provide the support required to take online teaching to the levels that will now be expected. It traces the medical teachers' changes across three stages of development, commenting on the overall mood, attitudes towards students, teaching focus, research focus, computer usage, theoretical knowledge and self-growth, assessment and institutional support. The aim is to provide some degree of insight into medical teachers' needs so that adequate anticipation and responses can occur before it is too late.

Medical student Internet usage: is the literature correct to call it addiction? An opinion piece.

Over the past few years, there has been a sharp rise in the number of academic articles examining "Internet addiction" among medical students. This opinion piece views the Internet as a communication environment and a medical information tool within medical education. Within this context, the paper investigates the Internet Addiction Test (IAT) and criteria used in those articles, and questions their assumptions and conclusions. It then argues that what is often viewed as "addiction" may actually be dedication to work.

Surgical patients' use of, and attitudes towards, the internet for e-patient activities in Germany and Oman.

INTRODUCTION: E-patient activities are known to impact upon the patient-doctor relationship and on surgical outcomes. In Oman, there is no published information about the e-patient. The aim of this study, conducted at two surgical sites, was to investigate surgical e-patients' use of, and attitudes towards, the Internet, and the possible impact on the delivery of healthcare. MATERIALS AND METHODS: A convenience sample of 83 German and 93 Omani patients at the two surgical sites were given an in-house paper-based questionnaire, based on e-patient activities described in the literature. Descriptive statistics like means, standard deviations and frequencies were calculated. RESULTS: There were many similarities in usage and attitudes. Omani patients showed much greater knowledge and usage of sites and apps, used the Internet more for health-related activities (26.9% vs. 12.0%), and had a greater proportion of their physician encounters through email (10.0% vs. 4.0%) and social media (15.2% vs. 1.8%). More Omani patients brought information from the Internet than German patients (13.5% vs. 6.0%). Patients from both countries were generally positive about bringing material from the Internet to the consultation. DISCUSSION AND CONCLUSION: Both sites indicated typical e-patient activity and attitudes as described in the literature. Age and type of condition (chronic vs. acute) may explain the differences to some extent, but this was not consistent. Socio-cultural differences between the two countries may have a great influence on the usage.

Surgeons' Interactions With and Attitudes Toward E-Patients: Questionnaire Study in Germany and Oman.

BACKGROUND: Doctors' interactions with and attitudes toward e-patients have an overall impact on health care delivery. OBJECTIVE: This study aimed to gauge surgeons' interactions with e-patients, their attitudes toward those e-patient activities, the possible impact on the delivery of health care, and the reasons behind those activities and attitudes. METHODS: We created a paper-based and electronic survey form based on pertinent variables identified in the literature, and from March 2018 to July 2018 we surveyed 49 surgeons in Germany and 59 surgeons in Oman, asking them about their interactions with and attitudes toward e-patients. Data were stored in Microsoft Excel and SPSS, and descriptive statistics, Pearson correlations, and chi-square tests were performed on the data. RESULTS: Of our sample, 71% (35/49) of the German surgeons and 56% (33/59) of the Omani surgeons communicated electronically with their patients. Although the German surgeons spent a greater percentage of Internet usage time on work-related activities (χ218=32.5; P=.02) than the Omani surgeons, there were many similarities in their activities. An outstanding difference was that the German surgeons used email with their patients more than the Omani surgeons (χ21=9.0; P=.003), and the Omani surgeons used social media, specifically WhatsApp, more than the German surgeons (χ21=18.6; P<.001). Overall, the surgeons were equally positive about the most common e-patient activities such as bringing material from the internet to the consultation (mean 4.11, SD 1.6), although the German surgeons (mean 3.43, SD 1.9) were more concerned (P=.001) than the Omani surgeons (mean 2.32, SD 1.3) about the potential loss of control and time consumption (German: mean 5.10, SD 1.4 and Omani: mean 3.92, SD 1.6; P<.001). CONCLUSIONS: The interactions show a high degree of engagement with e-patients. The differences between the German and the Omani surgeons in the preferred methods of communication are possibly closely linked to cultural differences and recent historical events. These differences may, moreover, indicate e-patients' desired method of electronic communication to include social media. The low impact of surgeons' attitudes on the activities may also result from a normalization of many e-patient activities, irrespective of the doctors' attitudes and influences.