Assessing the technical skills of surgical trainees.

BACKGROUND: The aims were to determine whether tests of technical skill on simple simulations can predict competence in the operating theatre and whether objective assessment in the operating theatre by direct observation and video recording is feasible and reliable. METHODS: Thirty-three general surgical trainees undertook five simple skill simulations (knotting, skin incision and suturing, tissue dissection, vessel ligation and small bowel anastomosis). The operative competence of each trainee was then assessed during two or three saphenofemoral disconnections (SFDs) by a single surgeon. Video recordings of the operations were also assessed by two surgeons. RESULTS: The inter-rater reliability between direct observation and blinded videotape assessment was high (alpha = 0.96 (95 per cent confidence interval 0.92 to 0.98)). Backward stepwise regression analysis revealed that the best predictors of operative competence were the number of SFDs performed previously plus the simulation scores for dissection and ligation, the key components of SFD (64 per cent of variance explained; P = 0.001). CONCLUSION: Deconstruction of operations into their component parts enables trainees to practise on simple simulations representing each component, and be assessed as competent, before undertaking the actual operation. Assessment of surgical competence by direct observation and video recording is feasible and reliable; such assessments could be used for both formative and summative assessment.

Comparing and contrasting undergraduate competence in musculoskeletal medicine with cardiovascular medicine and neurology.

OBJECTIVES: With an increasing prevalence of musculoskeletal conditions within the UK, specialty bodies are concerned that graduating medical students may lack appropriate knowledge in this system. We investigated the knowledge base of final year Sheffield medical students in the musculoskeletal system, compared with other major body systems. METHODS: A computer-based assessment was designed covering core topics that a pre-registration house officer should know about in musculoskeletal medicine, cardiology and neurology, using a predesigned testing format. The test was blueprinted against internal and external guidelines. It comprised 24 extended matching questions, each with three stems. A sample of 74 volunteer students from the final year (year 5) of the medical course at the University of Sheffield took part in the assessment. RESULTS: Overall scores of students on the test ranged from a baseline of 45% to a maximum of 85%. Test reliability was 0.75 (Cronbach's alpha). On stratifying the overall percentages into marks for individual systems, it was found that there were no significant differences between scores in musculoskeletal medicine, cardiovascular medicine or neurology. CONCLUSIONS: Despite the disparity of teaching between musculoskeletal medicine and other major organ systems within Sheffield's integrated medical curriculum, the knowledge base of medical students in the basic and clinical musculoskeletal sciences appears to be similar to that for cardiovascular medicine and neurology by the time of graduation. Nevertheless, several important issues must be addressed before these findings can be generalized.

Selecting performance assessment methods for experienced physicians.

BACKGROUND: While much is now known about how to assess the competence of medical practitioners in a controlled environment, less is known about how to measure the performance in practice of experienced doctors working in their own environments. The performance of doctors depends increasingly on how well they function in teams and how well the health care system around them functions. METHODS: This paper reflects the combined experiences of a group of experienced education researchers and the results of literature searches on performance assessment methods. CONCLUSION: Measurement of competence is different to measurement of performance. Components of performance could be re-conceptualised within a different domain structure. Assessment methods may be of a different utility to that in competence assessment and, indeed, of different utility according to the purpose of the assessment. An exploration of the utility of potential performance assessment methods suggests significant gaps that indicate priority areas for research and development.

The use of portfolios for assessment of the competence and performance of doctors in practice.

BACKGROUND: The use of portfolios can potentially provide flexibility in the summative assessment of doctors in practice. An assessment system should reflect and reinforce the active and planned professional development goals of individual doctors. This paper discusses some of the issues involved in developing such a system. RESULTS: To provide a complete picture of an individual doctor's practice, we suggest that a portfolio should encompass: (1) evidence covering all three domains of patient care, personal development and context management; (2) evidence that the person continuously undertakes critical assessment of their own performance, identifies and prioritises areas requiring enhanced performance and takes action to improve them as appropriate; (3) evidence that has been generated by assessments that are acceptably reliable, and (4) evidence which, taken in its entirety, is sufficient, valid, current and authentic. We include a suggested outline of the components of such a portfolio and suggest some criteria to determine the effectiveness of learning cycles. Portfolio reliability and validity requires sufficient evidence on which to base a judgement combined with reliable processes. CONCLUSION: Carefully specified portfolios can contribute to a system that ensures all doctors take an active part in identifying and meeting their own learning needs. Such a system, if properly implemented, would have a greatly beneficial impact on continuous quality improvement for the profession in general.

Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score.

BACKGROUND: Objective structured clinical examination (OSCE) standard-setting procedures are not well developed and are often time-consuming and complex. We report an evaluation of a simple 'contrasting groups' method, applied to an OSCE conducted simultaneously in three separate schools. SUBJECTS: Medical students undertaking an end-of-fifth year multidisciplinary OSCE. METHODS: Using structured marking sheets, pairs of examiners independently scored student performance at each OSCE station. Examiners also provided a global rating of overall performance. The actual scores of any borderline candidates at each station were averaged to provide a passing score for each station. The passing scores for all stations were combined to become the passing score for the whole exam. Validity was determined by making comparisons with performance on other fifth-year assessments. Reliability measures comprised interschool agreement, interexaminer agreement and interstation variability. RESULTS: The approach was simple and had face validity. There was a stronger association between the performance of borderline candidates on the OSCE and their in-course assessments than with their performance on the written exam, giving a weak measure of construct validity in the absence of a better 'gold standard'. There was good agreement between examiners in identifying borderline candidates. There were significant differences between schools in the borderline score for some stations, which disappeared when more than three stations were aggregated. CONCLUSION: This practical method provided a valid and reliable competence-based pass mark. Combining marks from all stations before determining the pass mark was more reliable than making decisions based on individual stations.

Development of a three-centre simultaneous objective structured clinical examination.

OBJECTIVE: To describe the development, organization, implementation and evaluation of a yearly multicentre, identical and simultaneous objective structured clinical examination (OSCE). SUBJECTS: All fifth-year medical students in a 6-year undergraduate medical programme. SETTING: The Christchurch, Dunedin and Wellington Schools of Medicine of the University of Otago, New Zealand. METHOD: One practice and two full 18-station OSCEs have been completed over 2 years, for up to 72 students per centre, in three centres. The process of development and logistics is described. Data are presented on validity, reliability and fairness. RESULTS: Face and content validity were established. Internal consistency was 0.83-0. 86 and interexaminer reliability, as assessed by the coefficient of correlation, averaged 0.78. Students rated the OSCE highly on relevance. Of the total variance in total OSCE marks, the schools contributed 6.9%, and the students 93.1%, in the first year. In the second year the schools contributed 6.2% and the students 93.8%. CONCLUSION: Implementation of a psychometrically sound, multicentre, simultaneous and identical OSCE is possible with a low level of interschool variation.

Assessing clinical competence at the undergraduate level.

This booklet aims to provide relevant background information and guidelines for medical school teachers in clinical departments charged with assessing the clinical competence of undergraduate students. It starts by emphasizing the difference between clinical competence and clinical performance. An approach to defining what should be assessed is outlined. The technical considerations of validity, reliability and practicability are discussed with reference to the ward- or practice-based setting and to the examination setting. The various methods available to assess aspects of competence are described and their strengths and weaknesses reviewed. The paper concludes with a discussion of the important issues of scoring and standard setting. The conclusion is reached that the quality of many current assessments could be improved. To do so will require a multi-format approach using both the practice and examination settings. Some of the traditional methods will have to be abandoned or modified and new methods introduced.

The observed long-case in clinical assessment.

The University of Adelaide has developed a comprehensive approach to final-year clinical assessment through the use of ward ratings during student internships and the administration of an objective examination of clinical competence at the end of the year. The traditional clinical viva has been retained only to make pass-fail decisions on borderline students. Problems in making these decisions prompted a modification of the viva so that each student undertook two long-cases, each being observed by a pair of examiners. Better agreement among examiners and more effective decision-making seemed to ensue. The success of this approach led to the introduction of observed long-cases into the ward setting. This move has been strongly supported by staff and students. The opportunities for remedial action during the course were regarded as particularly valuable. This approach may offer a practical alternative to more sophisticated assessment techniques, such as those involving standardized patients, without the need for additional resources. However, psychometric studies will have to be performed before the approach can be fully recommended.

Physicians' approaches to continuing education.

A questionnaire survey was conducted of a random sample of 308 physicians to determine their actual and preferred approach to continuing education. Demographic data raised questions about the possible influence of the undergraduate institution attended, the future impact of the growing proportion of women physicians and the effect of increasing subspecialisation. The average physician spends eight-nine hours a week on educational activities and 2.5-4.0 hours a week teaching. Unstructured ad hoc reading and postgraduate activities predominate over methods based on specific, individual needs or on current patient problems. The educational desirability of this is questioned. Physicians are avid attenders at conferences with 10-15% being at a national or international meeting in any one survey week. It appears physicians would prefer to use a wider range of educational activities including more refresher courses and innovative methods. By comparison, Canadian physicians showed similar, but more strongly evident, preferences. These findings have important implications for the College. It appears physicians need to be better informed about potentially more effective learning methods and account ought to be taken of their preference for methods which are not currently widely available.

The approaches to learning of specialist physicians.

Recent studies have provided information about the approaches to studying and learning used by medical students. However, no published work is available on the approaches of practising doctors. The Adelaide Diagnostic Learning Inventory for Medical Students (ADLIMS) was modified and administered to a random sample of 308 physicians (internists). Generally speaking, physicians seem to have lower scores on surface approach and higher scores on deep approach than students. Level of clinical experience did not appear to influence this finding. However, marked differences were apparent between the approaches adopted by physicians with additional postgraduate academic training and those without. The former had a much stronger tendency to use the more desirable deep approach. The latter seemed to rely more on the less desirable surface approach. Further longitudinal studies will be required to determine whether this difference is attributable to self-selection of those who have already developed a deep approach or is causally related to a training in research. Although these findings must be interpreted with caution, some implications are clear. Physicians should become more aware of the way they learn and about the way they teach. Inappropriate patterns of learning may be entrenched during the undergraduate and immediate postgraduate years. If validated, these findings may provide support for the inclusion of research projects in medical school and during specialist training.

Towards the identification of student learning problems: the development of a diagnostic inventory.

An understanding of how students approach their learning has important implications for medical education. Of particular interest is the fact that the approach students use in their study has a significant impact on both the quality of the learning and their academic success. It would clearly be of value to identify students whose approach to learning was predictive of unsatisfactory performance. This paper describes the initial development of two versions of an inventory (questionnaire) which was designed to aid in the diagnosis of student learning problems. Preliminary information is given on the reliability and validity of these instruments. Factor analyses support the underlying design. Correlation of subscales with academic performance has provided encouraging evidence of the potential of these inventories in identifying students with specific learning or study problems.

Psychometric characteristics of the objective structured clinical examination.

The objective structured clinical examination (OSCE) is increasingly being used as a method of clinical assessment yet its measurement characteristics have not been well documented. Evidence is accumulating that many OSCEs may be too short to achieve reliable results. This paper reports detailed psychometric analyses of OSCEs which were administered as part of a well-established final-year examination. Generalizability theory guided investigation of test reliability. At the present test length the OSCE components showed low reliabilities relative to written components. Satisfactory reliabilities could potentially be achieved if test length was increased to approximately 6 hours, a time which would create significant logistic problems for most medical schools. Several strategies for dealing with this practical problem have been explored. Firstly, it was shown that more careful selection of stations based on their psychometric characteristics can significantly improve reliability. Secondly, where rater availability is a limiting factor to increasing test length, more can be gained by using one rater per station and having more stations than using two raters per station. Finally, OSCE scores can, with advantage, be combined with other test scores which are obtained by using less resource-intensive methods. By adopting such strategies, a reliable assessment of clinical competence could be obtained in about 4 hours of testing time which was equally divided between an OSCE constructed of practical and clinical stations and a written test.

Eight years' experience with a structured clinical examination.

A structured clinical examination has been an integral component of the final-year examinations conducted by the Departments of Medicine and Surgery at the University of Adelaide for the last 8 years. It has been used as an alternative to the traditional clinical examination. This paper describes the results of ongoing student and examiner surveys carried out to determine their views and satisfaction with this new approach. It also briefly discusses the feasibility of introducing such an examination into a conventional medical school environment. The surveys showed a remarkable level of acceptance and support by both students and examiners. This positive reaction has been maintained over the 8-year period. The main reasons seem to be its perceived relevance and fairness. Students also appear to be directing their learning in a direction thought desirable by teachers. No significant problems have been encountered with the practical implementation of the method.

The approaches to learning of students in a traditional and in an innovative problem-based medical school.

The way in which students approach their learning is dependent on a variety of factors including the characteristics of the departments and teaching to which they are exposed. These factors appear to influence whether the students adopt a surface, deep or strategic approach. In order to explore further the relationship between educational context and approach to learning, a comparison was made between students attending a traditional medical school and those attending a problem-based medical school. The results showed marked differences, with the problem-based school being higher on deep approach and lower on surface approach than the traditional school. This study provides one of the first pieces of evidence of a difference between students in the two types of medical school which can be directly attributed to the educational environment. Students in the problem-based school appear to have an approach to learning which more closely approximates the aims of most medical schools. The results provide support for the philosophies and strategies of the problem-based schools.

Learning styles and approaches: implications for medical education.

This paper reviews the recent literature on learning styles and approaches to learning. It identifies two separate streams of research, one originating from mainstream cognitive and psychometric psychology and one from research undertaken within the everyday learning environment. The latter is dealt with in greater detail as it seems to have more immediate practical relevance. A simple model of the teaching-learning process is presented showing how students learn in different ways which are partly attributable to their preferred learning style and partly to the context in which the learning takes place. Three basic approaches have been identified: surface, deep and strategic, each resulting in a different learning outcome. The most desirable and successful is the deep approach. The way in which the teaching and the policies of the department and school influence the students' approach to learning are reviewed in some detail. A consideration of these characteristics in medical schools suggested that many may hinder rather than assist in the development of the desired approach. The work reviewed here suggests that the remedy will require not only substantial changes in the teaching, curriculum and, particularly, assessment, but also a new strategy based on identifying and assisting individual students whose approaches to study are not those expected of a competent university-educated doctor.