Impact of student choice on academic performance: cross-sectional and longitudinal observations of a student cohort.

BACKGROUND: Student choice plays a prominent role in the undergraduate curriculum in many contemporary medical schools. A key unanswered question relates to its impact on academic performance. METHODS: We studied 301 students who were in years 2 and 3 of their medical studies in 2005/06. We investigated the relationship between SSC grade and allocated preference. Separately, we examined the impact of 'self-proposing' (students designing and completing their own SSC) on academic performance in other, standard-set, summative assessments throughout the curriculum. The chi-squared test was used to compare academic performance in SSC according to allocated preference. Generalised estimating equations were used to investigate the effect of self-proposing on performance in standard-set examinations. RESULTS: (1) Performance in staff-designed SSC was not related to allocated preference. (2) Performance in year 1 main examination was one of the key predictors of performance in written and OSCE examinations in years 2, 3 and 4 (p<0.001). (3) The higher the score in the year 1 examination, the more likely a student was to self-propose in subsequent years (OR [CI] 1.07 [1.03-1.11], p<0.001). (4) Academic performance of students who self-proposed at least once in years 2 and/or 3 varied according to gender and year of course. CONCLUSION: In this study, no association was observed between allocated preference and SSC grade. The effect of self-proposing on academic performance in standard-set examinations was small. Our findings suggest instead that academically brighter students are more likely to design their own modules. Although student choice may have educational benefits, this report does not provide convincing evidence that it improves academic performance.

Examiner perceptions of a portfolio assessment process.

BACKGROUND: The portfolio assessment process is important for assessing learner achievement. AIMS: To study examiner perceptions of Dundee Medical School's portfolio assessment process, in years 4 and 5 of the 5-year curriculum, in relation to: outcomes as a framework for the portfolio assessment process; portfolio content; portfolio assessment process; end points of the portfolio assessment process; appropriateness of the two part final exam format and examiner training. METHODS: A questionnaire containing statements and open questions was used to obtain examiner feedback. Responses to each statement were compared over 3 years: 1999, 2000 and 2003. RESULTS: Response rates were 100%, 88% and 61% in 1999, 2002 and 2003, respectively. Examiners were positive about the ability of institutionally set learning outcomes (Dundee 12 exit learning outcomes) to provide a framework for the portfolio assessment process. They found difficulties, however, with the volume of portfolio content and the time allocated to assess it. Agreeing a grade for each learning outcome for the candidate with their co-examiner did not present difficulties. The comprehensive, holistic picture of the candidate provided by the portfolio assessment process was perceived to be one of its strengths. Examiners were supportive of the final examination format, and were satisfied with their briefing about the process. CONCLUSIONS: The 12 exit learning outcomes of Dundee curriculum provide an appropriate framework for the portfolio assessment process, but the content of the portfolio requires fine-tuning particularly with regard to quantity. Time allocated to examiners for the portfolio assessment process needs to be balanced against practicability. The holistic picture of the candidate provided by the process was one of its strengths.

'Hawks' and 'doves': effect of feedback on grades awarded by supervisors of student selected components.

BACKGROUND: Supervisors of some student selected components (SSCs) may appear to give higher grades than others. It is not known if feedback can influence the behaviour of supervisors in the grades they award. We have introduced feedback letters in our institution. AIMS: (1) To assess the feasibility of objectively identifying SSCs where grades awarded are consistently higher or lower than the average; (2) To assess the effect of feedback on the grades awarded by supervisors of SSCs. METHODS: The breakdown of SSC grades was examined over four consecutive years, before and after feedback letters were introduced in 2005. The grades awarded globally, and in five individual SSCs, were compared using the chi(2) goodness-of-fit test. RESULTS: (1) Individual SSCs were identified which awarded grades that were consistently different from the average. (2) Overall grades awarded in 2003/04 and 2004/05 (before feedback) were similar (chi2=0.37, df=2, p=0.83). Likewise, overall grades awarded in 2005/06 and 2006/07 (after feedback) were similar (chi2=1.72, df=2, p=0.42). Comparison of 2003/04 with 2005/06 (chi2=16.0, df=2, p<0.001), and 2006/07 (chi2=26.6, df=2, p<0.001), and of 2004/05 with 2005/06 (chi2=13.5, df=2, p=0.001), and 2006/07 (chi2=23.7, df=2, p<0.001), revealed highly significant differences. The grades awarded after feedback were higher than the grades awarded before feedback. CONCLUSIONS: The chi2 goodness-of-fit test may be used to identify individual SSCs where the grades awarded are different from the average, although the interpretation of the results thus obtained is fraught with difficulty. Our data also suggest that it is possible to influence assessors in the grades they award.

Student selected components: student-designed modules are associated with closer alignment of planned and learnt outcomes.

BACKGROUND: Student selected components (SSCs) are staff-designed modules selected by students from a menu of options provided separately from the 'core' curriculum. Students completing these do not always learn what teachers think they teach. Some medical schools also allow students to design their own modules. It is not known whether greater student input into planning of modules is associated with closer alignment of planned and learnt outcomes. AIMS: To compare student perception of learning outcomes addressed by student-designed ('self-proposed') SSCs, before and after completion, using the 'Dundee learning outcomes' template that we apply to all components of the undergraduate curriculum. METHODS: Students were required at the time of self-proposal, and subsequently as part of feedback on completed modules, to indicate which of twelve learning outcomes they felt were addressed by their self-proposed SSC. The chi2 test was used to compare student perceptions of learning outcomes before and after completion. RESULTS: More students thought that learning outcome 10 (appropriate decision making skills, clinical reasoning and judgement) was addressed after completion than before (96.3% versus 90.0%, chi2 4.99, p=0.02); for all other learning outcomes global perceptions were not significantly different after completion. Individual changes in perception ranged from 2.1% for outcome 12 (aptitude for personal development) to 19.6% for outcome 2 (competent to perform practical procedures). CONCLUSION: Greater student input into planning of modules is associated with closer alignment of planned and learnt outcomes. Our findings provide further evidence for the benefit of student-directed learning.

The long case and its modifications: a literature review.

CONTEXT: This review provides a summary of the published literature on the suitability of the long case and its modifications for high-stakes assessment. METHODS: Databases related to medicine were searched for articles published from 2000 to 2008, using the keywords 'long case', 'clinical examinations' and 'clinical assessment'. Reference lists of review articles were hand-searched. Articles related to the objective structured clinical examination were eliminated. Research-based articles with hard data were given more emphasis in this review than those based on opinion. RESULTS: Eighteen articles were identified. The main disadvantage of the long case is its inability to sample the curriculum widely, resulting in low reliability. The main advantage of the long case is its ability to assess the candidate's overall (holistic) approach to the patient. Modifications to the long case attempt to: structure the format and the marking scheme; increase the number of examiners; observe the candidate's behaviour, and increase the number of cases. CONCLUSIONS: The long case is a traditional clinical examination format for the assessment of clinical competence and assessment at this level is important. The starting point for the majority of recent research on the long case has been an acceptance of its low reliability and modifications to the format have been proposed. Further evidence of the efficacy of these modifications is required, however, before they can be recommended for summative assessment. If further research is to be undertaken on the long case, it should focus on finding practicable ways of sampling the curriculum widely to increase reliability while maintaining the holistic approach towards the patient, which represents the attraction of the long case.

Student perceptions of a portfolio assessment process.

OBJECTIVES: The objectives of this study were to identify and analyse students' attitudes to the portfolio assessment process over time. METHODS: A questionnaire containing statements and open questions was used to obtain feedback from students at the University of Dundee Medical School, Scotland. The responses to each statement were compared over 4 years (1999, 2000, 2002 and 2003). RESULTS: Response rates were 83% in 1999, 70% in 2000, 89% in 2002 and 88% in 2003. A major finding is that students perceived that portfolio building heightened their understanding of the exit learning outcomes and enabled reflection on their work. Student reactions to the portfolio process were initially negative, although they appreciated that senior staff took time to become familiar with their work through reviewing their portfolios. Student attitudes became more positive over the 4 years as the process evolved. Although portfolio assessment was recognised as supporting student learning, portfolio building was perceived to interfere with clinical learning as a result of the excessive amounts of paper evidence required. CONCLUSIONS: Paperwork should be kept within manageable limits. A student induction process that highlights the importance of providing evidence for achieving all learning outcomes, not just theoretical knowledge and skills, may be helpful in allaying student concern over portfolio building and assessment and support preparation for lifelong learning and reflective clinical practice.

Student selected components: do students learn what teachers think they teach?

BACKGROUND: It is well recognized that what teachers teach and what students learn may not be the same. This applies to all parts of the undergraduate medical curriculum, but may be especially relevant to student selected components, which vary substantially in their educational content. This has not been studied previously. AIMS: To compare perceptions of students and supervisors in relation to learning outcomes addressed by student selected components, and thus to examine differences between what is taught and what is learned. METHODS: Supervisors (n = 69) were asked to indicate which of twelve learning outcomes they felt were components of teaching and assessment. Upon completion of each SSC, students were required to complete the same outcomes template as part of their feedback (n = 644). Perceptions were compared in two ways: (1) a colour-coded 'traffic-light' system was used to record agreement/disagreement between students and supervisors of individual SSCs; (2) differences in perception of outcomes across the entire SSC programme were compared using the chi(2) statistic. RESULTS: (1) The 'traffic-light' system readily identified individual SSCs where significant disagreement existed and which were subject to further scrutiny. (2) More students than supervisors thought that outcome 2 (competent to perform practical procedures) was a component of teaching and assessment (41.8% v 27.5%, chi(2) = 5.24, p = 0.02), whereas more supervisors than students thought that outcome 6 (competent in communication skills) (97.1% v 82.1%, chi(2) = 6.91, p = 0.009) and outcome 7 (competent to retrieve and handle information) (100% v 93.7%, chi(2) = 4.8, p = 0.02) were. CONCLUSIONS: Significant disagreement exists about the outcomes addressed by SSCs, suggesting that students do not always learn what teachers think they teach. The use of two complementary approaches allows global and individual comparisons to be drawn and thus provides a powerful tool to address this important issue.

Case studies in outcome-based education.

Outcome-based education is one of the most significant global developments in medical education in recent years. This paper presents four case studies of outcome-based education from medical schools in different parts of the world; Scotland; USA; Pakistan; and Singapore. The outcome-based curricula have either been in place for some time, are evolving or are at the planning proposal stage. The outcomes, change process and implementation of the outcome-based approach are described. Variation in the extent to which each medical school has implemented outcome-based education is discussed and key points for successful implementation are highlighted. This paper is based on the pre-conference symposium "outcome-based curricula: global perspectives" presented by the authors at the 4th Asia Pacific Medical Education Conference (APMEC) in Singapore, 8-11 February, 2007.

The Objective Structured Clinical Examination (OSCE) as a determinant of veterinary clinical skills.

The Objective Structured Clinical Examination (OSCE) has become an excellent tool to evaluate many elements of a student's clinical skills, especially including communication with the patient (human medicine) or client (veterinary medicine); eliciting clinical information from these conversations; some aspects of the physical examination; and many areas of clinical evaluation and assessment. One key factor is that the examination can be structured to compare different students' abilities.

AMEE Education Guide no. 28: the development and role of departments of medical education.

A department of medical education is becoming an essential requirement for a medical school. This publication is intended for those wishing to establish or develop a medical education department. It may also prove useful to teachers in medicine by providing information on how such a department can support their activities. This will vary with the local context but the principles are generalizable. Medical education departments are established in response to increased public expectations relating to healthcare, societal trends towards increased accountability, educational developments, increased interest in what to teach and how to educate doctors and the need to train more doctors. The functions of a department of medical education include research, teaching, service provision and career development of the staff. The scope of its activities includes undergraduate and postgraduate education, continuing professional development and continuing medical education. These activities may be extended to other healthcare professions. Flexibility is the key to staffing a department of medical education. Various contractual arrangements, affiliations and support from non-affiliated personnel are needed to provide a multi-professional team with a range of expertise. The precise structure of the department will depend on the individual institution. The name of the department may suggest its position within the university structure. The director provides academic leadership for the department and his/her responsibilities include promotion of staff collaboration, fostering career development of the staff and establishing local, regional and international links. Financial support may come from external funding agencies, government or university sources. Some departments of medical education are financially self-supporting. The department should be closely integrated with the medical school. Support for the department from the dean is an essential factor for sustainability. Several case studies of medical education departments throughout the world are included as examples of the different roles and functions of a department of medical education.

Outcome-based education.

The move to outcome-based education has been one of the most important trends in health-profession education in recent years. This paper defines outcomes and outcome-based education, describes the development of outcome-based education, identifies several different ways that outcomes have been presented, and discusses the advantages and disadvantages of the outcome-based educational approach. The implementation of outcome-based education at the University of Dundee medical school, Scotland, UK is described as a case study for curriculum planners in veterinary medical education. The lessons learned in Dundee from six years' experience with outcome-based education are identified to aid veterinary medical educators wishing to implement the approach.

OSCE: the Dundee experience.

The Dundee Medical School has bean running OSCEs since 1977. In 1995, an integrated systems-based spiral curriculum on the core and options model was introduced. In 1997, outcome-based education was introduced as the basis for instruction, with a task-based educational strategy employed for students in years 4 and 5. This blend of educational strategies was considered in the design of the student assessment process. Assessment instruments, appropriate for use at each of the four levels of Miller's pyramid, were identified and included in the assessment process. The OSCE was used for summative assessment of students at the level of 'shows how' or simulation in years 2, 3 and 4. A year 2 OSCE is described here. Features of the Dundee OSCE are identified, relating to number and length of individual stations, practicalities or assessing a year group of students without student contamination with examination information and the blueprints used to design the examinations. Suggestions made for future development of the OSCE include the OSSE, the Objective Structured Selection Examination, and an exploration of the potential of the OSCE to assess attitudes, personal attributes and professionalism. The need is identified for a platform to debate issues such as should individual medical schools attempt to achieve national test centre standards with their examinations.

Planning and implementing an undergraduate medical curriculum: the lessons learned.

In 1995 Dundee medical school introduced an integrated, systems-based spiral curriculum with a number of innovative features. The medical school has now had eight years' experience of the curriculum. This paper describes the changes that have taken place in the curriculum over the eight years. Evidence from internal and external reviews and student examination data are used to identify the lessons learned from implementing the curriculum. The Dundee experience, the approaches to the curriculum described and the conclusions reached are relevant to all with an interest in medical education.