The foundations of measurement and assessment in medical education.

As a medical educator, you may be directly or indirectly involved in the quality of assessments. Measurement has a substantial role in developing the quality of assessment questions and student learning. The information provided by psychometric data can improve pedagogical issues in medical education. Through measurement we are able to assess the learning experiences of students. Standard setting plays an important role in assessing the performance quality of students as doctors in the future. Presentation of performance data for standard setters may contribute towards developing a credible and defensible pass mark. Validity and reliability of test scores are the most important factors for developing quality assessment questions. Analysis of the answers to individual questions provides useful feedback for assessment leads to improve the quality of each question, and hence make students' marks fair in terms of diversity and ethnicity. Item Characteristic Curves (ICC) can send signals to assessment leads to improve the quality of individual questions.

Level of Confidence in the 12 Roles of Medical Teacher. A Descriptive Study at Faculty of Medicine, Srinakharinwirot University, Thailand.

BACKGROUND: An appropriate faculty development program is imperative for every medical school and should be tailored to suit the needs of individuals and the institution. OBJECTIVE: To measure the average level of confidence in each of the 12 roles of medical teachers and analyse if any of these factors: age, gender duration of teaching, and teaching in pre-clinical or clinical years has effect on the level of confidence in the teacher roles. MATERIAL AND METHOD: The online questionnaire invitations were sent via email to all 211 faculty members at Faculty of Medicine, Srinakharinwirot University, and 118 (55.9%) responded. The questionnaire asked about demographic data, teaching experience and their specialty or disciplines. The respondents were also asked to rate their level of confidence in each of the 12 teacher roles on a 5 point Likert scale ranging from 1 (none) to 5 (great). RESULTS: The three most highly rated roles on the level of confidence were the clinical teacher (4.11), the on the job role model (4.11) and the lecturer (3.97). The three roles with the lowest rating were the curriculum planner (3.08), the curriculum assessor (3.23) and the mentor (3.31). Age and teaching experience were positively correlated with the level of confidence in nearly all of the 12 teacher roles. The pre-clinical year teachers had a higher mean level of confidence than clinical teachers in 6 out of the 12 roles. CONCLUSION: The future faculty development programs should aim towards supporting self-evaluation of teaching, mentoring as well as promoting facilitative roles of the teacher Retention of faculty members at the institution is also important as the teaching experience significantly correlate with confidence in the teacher roles.

Postexamination analysis of objective tests using the three-parameter item response theory.

BACKGROUND AND PURPOSE: Although the importance of item response theory (IRT) has been emphasized in health and medical education, in practice, few psychometricians in nurse education have used these methods to create tests that discriminate well at any level of student ability. The purpose of this study is to evaluate the psychometric properties of a real objective test using three-parameter IRT. METHODS: Three-parameter IRT was used to monitor and improve the quality of the test items. RESULTS: Item parameter indices, item characteristic curves (ICCs), test information functions, and test characteristic curves reveal aberrant items which do not assess the construct being measured. CONCLUSIONS: The results of this study provide useful information for educators to improve the quality of assessment, teaching strategies, and curricula.

Analysing the hidden curriculum: use of a cultural web.

CONTEXT: Major influences on learning about medical professionalism come from the hidden curriculum. These influences can contribute positively or negatively towards the professional enculturation of clinical students. The fact that there is no validated method for identifying the components of the hidden curriculum poses problems for educators considering professionalism. The aim of this study was to analyse whether a cultural web, adapted from a business context, might assist in the identification of elements of the hidden curriculum at a UK veterinary school. METHODS: A qualitative approach was used. Seven focus groups consisting of three staff groups and four student groups were organised. Questioning was framed using the cultural web, which is a model used by business owners to assess their environment and consider how it affects their employees and customers. The focus group discussions were recorded, transcribed and analysed thematically using a combination of a priori and emergent themes. RESULTS: The cultural web identified elements of the hidden curriculum for both students and staff. These included: core assumptions; routines; rituals; control systems; organisational factors; power structures, and symbols. Discussions occurred about how and where these issues may affect students' professional identity development. CONCLUSIONS: The cultural web framework functioned well to help participants identify elements of the hidden curriculum. These aspects aligned broadly with previously described factors such as role models and institutional slang. The influence of these issues on a student's development of a professional identity requires discussion amongst faculty staff, and could be used to develop learning opportunities for students. The framework is promising for the analysis of the hidden curriculum and could be developed as an instrument for implementation in other clinical teaching environments.

Psychometric evaluation of a knowledge based examination using Rasch analysis: an illustrative guide: AMEE guide no. 72.

Classical test theory has traditionally been used to carry out post-examination analysis of objective test data. It uses descriptive methods and aggregated data to help identify sources of measurement error and unreliability in a test, in order to minimise them. Item response theory (IRT), and in particular Rasch analysis, uses more complex methods to produce outputs that not only identify sources of measurement error and unreliability, but also identify the way item difficulty interacts with student ability. In this guide, a knowledge-based test is analysed by the Rasch method to demonstrate the variety of useful outputs that can be provided. IRT provides a much deeper analysis giving a range of information on the behaviour of individual test items and individual students as well as the underlying constructs being examined. Graphical displays can be used to evaluate the ease or difficulty of items across the student ability range as well as providing a visual method for judging how well the difficulty of items on a test match student ability. By displaying data in this way, problem test items are more easily identified and modified allowing medical educators to iteratively move towards the 'perfect' test in which the distribution of item difficulty is mirrored by the distribution of student ability.

Medical students' understanding of empathy: a phenomenological study.

CONTEXT: Empathy towards patients is associated with improved health outcomes. However, quantitative studies using self-reported data have not provided an in-depth opportunity to explore the lived experiences of medical students concerning empathy. OBJECTIVES: This study was designed to investigate undergraduate medical students' experiences of the phenomenon of empathy during the course of their medical education and to explore the essence of their empathy. METHODS: This was a descriptive, phenomenological study of medical student interviews conducted using the method of Colaizzi and Giorgi. The sample (n = 10) was drawn from medical students in Years 4 and 5. In-depth interviews were used to obtain a clear understanding of their experiences of empathy in the context of patient care. Interviews continued until no new information could be identified from transcripts. RESULTS: Five themes were identified from analysis: the meaning of empathy; willingness to empathise; innate empathic ability; empathy decline or enhancement, and empathy education. Empathic ability was manifested through two factors: innate capacity for empathy, and barriers to displaying empathy. Different experiences and explanations concerning the decline or enhancement of empathy during medical education were explored. CONCLUSIONS: Empathic ability was identified as an important innate attribute which nevertheless can be enhanced by educational interventions. Barriers to the expression of empathy with patients were identified. Role-modelling by clinical teachers was seen as the most important influence on empathy education for students engaged in experiential learning.

Twelve tips for incorporating educational theory into teaching practices.

BACKGROUND: Although there is no single overarching theory of learning, there is a group of learning theories that shares some common elements which can provide useful guidance on a range of teaching and learning practices. AIM: This article aims to describe Constructivist, Experiential and Humanistic learning theories, to explain how these three theories are fundamentally related and to demonstrate how each of them suggests teaching and learning practices. CONCLUSION: Common educational theories can be combined to provide 12 practical tips for teachers and facilitators. This demonstrates how theoretical ideas lead to practical consequences.

Post-examination interpretation of objective test data: monitoring and improving the quality of high-stakes examinations--a commentary on two AMEE Guides.

As great emphasis is rightly placed upon the importance of assessment to judge the quality of our future healthcare professionals, it is appropriate not only to choose the most appropriate assessment method, but to continually monitor the quality of the tests themselves, in a hope that we may continually improve the process. This article stresses the importance of quality control mechanisms in the exam cycle and briefly outlines some of the key psychometric concepts including reliability measures, factor analysis, generalisability theory and item response theory. The importance of such analyses for the standard setting procedures is emphasised. This article also accompanies two new AMEE Guides in Medical Education (Tavakol M, Dennick R. Post-examination Analysis of Objective Tests: AMEE Guide No. 54 and Tavakol M, Dennick R. 2012. Post examination analysis of objective test data: Monitoring and improving the quality of high stakes examinations: AMEE Guide No. 66) which provide the reader with practical examples of analysis and interpretation, in order to help develop valid and reliable tests.

Post-examination interpretation of objective test data: monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66.

The purpose of this Guide is to provide both logical and empirical evidence for medical teachers to improve their objective tests by appropriate interpretation of post-examination analysis. This requires a description and explanation of some basic statistical and psychometric concepts derived from both Classical Test Theory (CTT) and Item Response Theory (IRT) such as: descriptive statistics, explanatory and confirmatory factor analysis, Generalisability Theory and Rasch modelling. CTT is concerned with the overall reliability of a test whereas IRT can be used to identify the behaviour of individual test items and how they interact with individual student abilities. We have provided the reader with practical examples clarifying the use of these frameworks in test development and for research purposes.

Post-examination analysis of objective tests.

One of the key goals of assessment in medical education is the minimisation of all errors influencing a test in order to produce an observed score which approaches a learner's 'true' score, as reliably and validly as possible. In order to achieve this, assessors need to be aware of the potential biases that can influence all components of the assessment cycle from question creation to the interpretation of exam scores. This Guide describes and explains the processes whereby objective examination results can be analysed to improve the validity and reliability of assessments in medical education. We cover the interpretation of measures of central tendency, measures of variability and standard scores. We describe how to calculate the item-difficulty index and item-discrimination index in examination tests using different statistical procedures. This is followed by an overview of reliability estimates. The post-examination analytical methods described in this guide enable medical educators to construct reliable and valid achievement tests. They also enable medical educators to develop question banks using the collection of appropriate questions from existing examination tests in order to use computerised adaptive testing.

Psychometric properties and confirmatory factor analysis of the Jefferson Scale of Physician Empathy.

BACKGROUND: Empathy towards patients is considered to be associated with improved health outcomes. Many scales have been developed to measure empathy in health care professionals and students. The Jefferson Scale of Physician Empathy (JSPE) has been widely used. This study was designed to examine the psychometric properties and the theoretical structure of the JSPE. METHODS: A total of 853 medical students responded to the JSPE questionnaire. A hypothetical model was evaluated by structural equation modelling to determine the adequacy of goodness-of-fit to sample data. RESULTS: The model showed excellent goodness-of-fit. Further analysis showed that the hypothesised three-factor model of the JSPE structure fits well across the gender differences of medical students. CONCLUSIONS: The results supported scale multi-dimensionality. The 20 item JSPE provides a valid and reliable scale to measure empathy among not only undergraduate and graduate medical education programmes, but also practising doctors. The limitations of the study are discussed and some recommendations are made for future practice.

Corpus analysis of problem-based learning transcripts: an exploratory study.

CONTEXT: Problem-based learning (PBL) is one of the most researched areas of medical education; numerous studies and meta-analyses have examined its overall effectiveness compared with that of traditional methods. Fewer studies have looked at the processes occurring within the PBL group, considering that a key difference between PBL and traditional teaching is the amount of talking in which students engage: a typical PBL week might involve 3-5 hours of talk. Little research has attempted to capture the vocabulary used or to analyse the cognitive processes that occur during PBL discussions and that might contribute to the advantages of this method of learning. METHODS: We professionally transcribed a recording of a facilitator and seven students engaging in a PBL case consisting of three sessions. The text was analysed by using Wmatrix2 (UCREL, Lancaster University, UK), a powerful linguistic program used for corpus analysis. This allowed the inter-sessional comparative quantification of the use of technical vocabulary, as well as of reasoning, questioning and explaining episodes. RESULTS: We have compared the frequency of technical words and of reasoning, explaining and questioning episodes between sessions and related these differences to the focus of each session. We have been able to build concordances of important words and phrases within sentences which can be used to identify and monitor examples of questioning, explaining and reasoning. CONCLUSIONS: In this exploratory study we have shown that corpus analysis can be a useful tool with which to analyse PBL transcriptions. This technique can be used to monitor the development of a technical vocabulary and skills in scientific and clinical reasoning as students progress through a PBL curriculum. We propose this methodology will become a powerful tool to help explore the much wider cognitive and linguistic developments of students and facilitators as they engage in PBL discourse.

Are Asian international medical students just rote learners?

A wide variety of countries are seeking to attract international medical students. This could be due to the fact that their universities not only receive the economic benefit from these students, but also because they recognise the issues of cultural diversity and pedagogical practice. This review paper draws on literature to understand more fully the learning process of Asian international students. Whereas views on learning are different across cultures, medical school teachers must understand how Asian international students learn based on their culture. Two general themes emerged from the literature review: firstly culture's influence on learning and secondly memorisation versus understanding, both of which relate to the learning process of Asian international students. This study shows that Asian international students have a different approach to learning, which is not just about rote learning. Changes in attitudes towards Asian international students may stimulate the internationalisation of a more culturally sensitive form of medical education. The paper suggests further work on the area of appreciative thinking in order to identify the epistemological and ontological dimensions for a flexible approach to learning.

Empathy in Iranian medical students: A preliminary psychometric analysis and differences by gender and year of medical school.

BACKGROUND: It has been well documented that effective empathic communication in the context of patient care is associated with improved health care outcomes. However, the emphasis given to empathy in medical education in Iran is limited, and the state of such teaching is unknown in many countries. AIMS: To determine the psychometric properties of an Iranian translation of the Jefferson Scale of Physician Empathy (JSPE) among medical students, and to examine the differences on mean empathy scores by gender and the different years of medical school. METHOD: A cross-sectional study was conducted among medical students. Data analysis was based on 181 questionnaires. Principal component analysis (PCA) with Varimax rotation was used to identify the number and composition of components constituting the developed constructs. RESULTS: The PCA yielded three factors: Compassionate care, perspective-taking, and the ability to walk in the patient's shoes. No statistically significant differences in the empathy means scores were found by gender and the different years of medical school. CONCLUSIONS: The Persian version of JSPE is a psychometrically sound instrument to measure empathy. Cultural backgrounds and pedagogical practice may influence medical students' attitudes towards empathy. Some recommendations are made, and the study limitations are discussed.

A quantitative study of Iranian nursing students' knowledge and attitudes towards pain: implication for education.

It is well documented that pain management and pain assessment is an indispensible part of the nursing care of patients. This study sought to quantify the current knowledge and attitudes of nursing students in Iran about pain management. We conducted a cross-sectional study using a well-validated questionnaire entitled the 'Knowledge and Attitudes Regarding Pain Tool', previously used to evaluate undergraduate nursing students. Results from the survey questionnaire showed that there was a severe deficit in knowledge relating to pain and its management. It is argued that there is a real need for improving the content of pain and its management in the undergraduate nursing education curriculum, which might improve the delivery of optimal nursing care of patients. The limitations of the study are discussed and some recommendations are made for reforming pain management education for future practice.

Online eAssessment: AMEE guide no. 39.

In this guide, the authors outline the advantages of online eAssessment and examine the intellectual, technical, legal and cost issues that arise from its use. This guide outlines the major assessment types that are suitable for online assessment and makes a key distinction between formative and summative assessment. The focus is primarily on the latter since that is where the difficulties are most acute and robust systems most critical. A range of practical issues relating to the key stages in running a summative e-exam are explored and advice given on system requirements and on how to ensure that the exam runs smoothly when you 'go live'. This section includes consideration of the way that using eAssessment might affect the standard setting and results analysis process. The section on future trends in online assessment explores possibilities such as computer adaptive testing and the automated assessment of free text answers. Finally, there is a consideration of the implications of these trends for management.

Evidence of gender bias in True-False-Abstain medical examinations.

BACKGROUND: There is evidence that males and females differ in their attainment on a variety of assessments in general and in medical education. It has been suggested that the True-False-Abstain (TFA) format with negative marking is biased against females. METHODS: Eight years worth of examination data from the first two years of an undergraduate medical curriculum was analysed. 359 courses were evaluated for statistically significant differences between the genders using ANOVA. Logistic regression was used to test if subject area, calendar year or exam format predicted that males or females do better (termed male advantage or female advantage). RESULTS: Statistically significant differences between the genders were found in 111 (31%) of assessments with females doing better than males in 85 and males better in 26. Female advantage was associated with a particular year (2001), the Personal and Professional Development strand of the curriculum, in course assessment and short answer questions. Male advantage was associated with the anatomy and physiology strand of the curriculum and examinations containing TFA formats, where the largest gender difference was noted. Males were 16.7 times more likely than females to do better on an assessment if it had any questions using the TFA format. CONCLUSION: Although a range of statistically significant gender differences was found, they were concentrated in TFA and short answer formats. The largest effect was for TFA formats where males were much more likely to do better than females. The gender bias of TFA assessments in medical education is yet another reason why caution should be exercised in their use.

A descriptive study of medical educators' views of problem-based learning.

BACKGROUND: There is a growing amount of literature on the benefits and drawbacks of Problem-Based Learning (PBL) compared to conventional curricula. However, it seems that PBL research studies do not provide information rigorously and formally that can contribute to making evidence-based medical education decisions. The authors performed an investigation aimed at medical education scholars around the question, "What are the views of medical educators concerning the PBL approach?" METHODS: After framing the question, the method of data collection relied on asking medical educators to report their views on PBL. Two methods were used for collecting data: the questionnaire survey and an online discussion forum. RESULTS: The descriptive analysis of the study showed that many participants value the PBL approach in the practice and training of doctors. However, some participants hold contrasting views upon the importance of the PBL approach in basic medical education. For example, more than a third of participants (38.5%) had a neutral stance on PBL as a student-oriented educational approach. The same proportion of participants also had a neutral view of the efficiency of traditional learning compared to a PBL tutorial. The open-ended question explored the importance of faculty development in PBL. A few participants had negative perceptions of the epistemological assumptions of PBL. Two themes emerged from the analysis of the forum repliers: the importance of the faculty role and self-managed education. CONCLUSION: Whilst many participants valued the importance of the PBL approach in the practice and training of doctors and agreed with most of the conventional descriptions of PBL, some participants held contrasting views on the importance of the PBL approach in undergraduate medical education. However there was a strong view concerning the importance of facilitator training. More research is needed to understand the process of PBL better.

Present and future of the undergraduate ophthalmology curriculum: a survey of UK medical schools.

OBJECTIVES: To investigate the current undergraduate ophthalmology curricula provided by the UK medical schools, evaluate how they compare with the guidelines of the Royal College of Ophthalmologists (RCOphth) and International Council for Ophthalmology (ICO), and determine the views of the UK ophthalmology teaching leads on the future direction of the curriculum. METHODS: A cross-sectional questionnaire was sent to teaching leads in 31 medical schools across the UK. The questionnaire evaluated eight themes of the curriculum: content and learning outcomes, communication of learning outcomes, organisation of the curriculum, assessment, educational resources, teaching methods used, and the educational environment. The ophthalmology teaching leads were also asked their opinion on the current and future management of the curriculum. These were compared with RCOphth and ICO guidelines and descriptive statistical analysis performed. RESULTS: A response rate of 93% (n=29/31) was achieved. The knowledge and clinical skills taught by the UK medical schools match the  RCOphth guidelines, but fail to meet the ICO recommendations. A diverse range of assessment methods are used by UK medical schools during ophthalmology rotations. Variation was also observed in the organisation and methods of ophthalmology teaching.  However, a significant consensus about the future direction of the curriculum was reported by teaching leads. CONCLUSIONS: Comprehensive RCOphth guidance, and resource sharing between medical schools could help to ensure ophthalmology's continuing presence in the medical curriculum and improve the effectiveness of undergraduate ophthalmology teaching, while reducing the workload of local teaching departments and medical schools.