The threat of the dyscompetent resident: A plea to make the implicit more explicit!

Although several examples of frameworks dealing with students' unprofessional behaviour are available, guidance on how to deal locally or regionally with dysfunctional residents is limited (Hickson et al. in Acad Med 82(11):1040-1048, 2007b; Leape and Fromson in Ann Intern Med 144(2):107-115, 2006). Any 'rules' are mostly unwritten, and often emerge by trial and error within the specialty training programme (Stern and Papadakis in N Engl J Med 355(17):1794-1799, 2006). It is nevertheless of utmost importance that objectives, rules and guidelines comparable to those existing in undergraduate training (Project Team Consilium Abeundi van Luijk in Professional behaviour: teaching, assessing and coaching students. Final report and appendices. Mosae Libris, 2005; van Mook et al. in Neth J Crit Care 16(4):162-173, 2010a) are developed for postgraduate training. And that implicit rules are made explicit. This article outlines a framework based on the lessons learned from contemporary postgraduate medical training programmes.

An evaluation of a computer based education program for the diagnosis and management of dementia in primary care. An international study of the transcultural adaptations necessary for European dissemination.

OBJECTIVES: The aim of this study is to make an inventory of the changes that are needed to make an interactive computer based training program (ICBT) with a specific educational content, acceptable to professional communities with different linguistic,cultural and health care backgrounds in different European countries. METHODS: Existing educational software, written in two languages was reviewed by GPs and primary care professionals in three different countries. Reviewers worked through the program using a structured critical reading grid. RESULTS: A 'simple' translation of the program is not sufficient. Minor changes are needed to take account of linguistic differences and medical semantics. Major changes are needed in respect of the existing clinical guidelines in every country related to differences in the existing health care systems. CONCLUSIONS: ICTB programs cannot easily be used in different countries and cultures. The development of a structured educational program needs collaboration between educationalists, domain experts, information technology advisers and software engineers. Simple validation of the content by local expert groups will not guarantee the program's exportability. It is essential to involve different national expert groups at every phase of the development process in order to disseminate it in other countries.

Assessing the technical skills of surgical trainees.

BACKGROUND: The aims were to determine whether tests of technical skill on simple simulations can predict competence in the operating theatre and whether objective assessment in the operating theatre by direct observation and video recording is feasible and reliable. METHODS: Thirty-three general surgical trainees undertook five simple skill simulations (knotting, skin incision and suturing, tissue dissection, vessel ligation and small bowel anastomosis). The operative competence of each trainee was then assessed during two or three saphenofemoral disconnections (SFDs) by a single surgeon. Video recordings of the operations were also assessed by two surgeons. RESULTS: The inter-rater reliability between direct observation and blinded videotape assessment was high (alpha = 0.96 (95 per cent confidence interval 0.92 to 0.98)). Backward stepwise regression analysis revealed that the best predictors of operative competence were the number of SFDs performed previously plus the simulation scores for dissection and ligation, the key components of SFD (64 per cent of variance explained; P = 0.001). CONCLUSION: Deconstruction of operations into their component parts enables trainees to practise on simple simulations representing each component, and be assessed as competent, before undertaking the actual operation. Assessment of surgical competence by direct observation and video recording is feasible and reliable; such assessments could be used for both formative and summative assessment.

Corridor consultations and the medical microbiological record: is patient safety at risk?

The performance procedures of the General Medical Council are aimed at identifying seriously deficient performance in a doctor. The performance procedures require the medical record to be of a standard that enables the next doctor seeing the patient to give adequate care based on the available information. Setting standards for microbiological record keeping has proved difficult. Over one fifth of practising medical microbiologists (including virologists) in the UK (139 of 676) responded to a survey undertaken by the working group developing the performance procedures for microbiology, to identify current practice and to develop recommendations for agreement within the profession about the standards of the microbiological record. The cumulative frequency for the surveyed recording methods used indicated that at various times 65% (90 of 139) of respondents used a daybook, 62% (86 of 139) used the back of the clinical request card, 57% (79 of 139) used a computer record, and 22% (30 of 139) used an index card system to record microbiological advice, suggesting wide variability in relation to how medical microbiologists maintain clinical records.

The relationship between competence and performance: implications for assessing practice performance.

OBJECTIVE: This paper aims to describe current views of the relationship between competence and performance and to delineate some of the implications of the distinctions between the two areas for the purpose of assessing doctors in practice. METHODS: During a 2-day closed session, the authors, using their wide experiences in this domain, defined the problem and the context, discussed the content and set up a new model. This was developed further by e-mail correspondence over a 6-month period. RESULTS: Competency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as measures of what doctors do in practice. The distinction between competency-based and performance-based methods leads to a three-stage model for assessing doctors in practice. The first component of the model proposed is a screening test that would identify doctors at risk. Practitioners who 'pass' the screen would move on to a continuous quality improvement process aimed at raising the general level of performance. Practitioners deemed to be at risk would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice. CONCLUSION: We propose a new model, designated the Cambridge Model, which extends and refines Miller's pyramid. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence, the influences of the individual (e.g. health, relationships), and the influences of the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.

When enough is enough: a conceptual basis for fair and defensible practice performance assessment.

INTRODUCTION: An essential element of practice performance assessment involves combining the results of various procedures in order to see the whole picture. This must be derived from both objective and subjective assessment, as well as a combination of quantitative and qualitative assessment procedures. Because of the severe consequences an assessment of practice performance may have, it is essential that the procedure is both defensible to the stakeholders and fair in that it distinguishes well between good performers and underperformers. LESSONS FROM COMPETENCE ASSESSMENT: Large samples of behaviour are always necessary because of the domain specificity of competence and performance. The test content is considerably more important in determining which competency is being measured than the test format, and it is important to recognise that the process of problem-solving process is more idiosyncratic than its outcome. It is advisable to add some structure to the assessment but to refrain from over-structuring, as this tends to trivialise the measurement. IMPLICATIONS FOR PRACTICE PERFORMANCE ASSESSMENT: A practice performance assessment should use multiple instruments. The reproducibility of subjective parts should not be increased by over-structuring, but by sampling through sources of bias. As many sources of bias may exist, sampling through all of them may not prove feasible. Therefore, a more project-orientated approach is suggested using a range of instruments. At various timepoints during any assessment with a particular instrument, questions should be raised as to whether the sampling is sufficient with respect to the quantity and quality of the observations, and whether the totality of assessments across instruments is sufficient to see 'the whole picture'. This policy is embedded within a larger organisational and health care context.

Setting performance standards for medical practice: a theoretical framework.

BACKGROUND: The assessment of performance in the real world of medical practice is now widely accepted as the goal of assessment at the postgraduate level. This is largely a validity issue, as it is recognised that tests of knowledge and in clinical simulations cannot on their own really measure how medical practitioners function in the broader health care system. However, the development of standards for performance-based assessment is not as well understood as in competency assessment, where simulations can more readily reflect narrower issues of knowledge and skills. This paper proposes a theoretical framework for the development of standards that reflect the more complex world in which experienced medical practitioners work. METHODS: The paper reflects the combined experiences of a group of education researchers and the results of literature searches that included identifying current health system data sources that might contribute information to the measurement of standards. CONCLUSION: Standards that reflect the complexity of medical practice may best be developed through an "expert systems" analysis of clinical conditions for which desired health care outcomes reflect the contribution of several health professionals within a complex, three-dimensional, contextual model. Examples of the model are provided, but further work is needed to test validity and measurability.

Length of uninterrupted repeats determines instability at the unstable mouse expanded simple tandem repeat family MMS10 derived from independent SINE B1 elements.

Mouse expanded simple tandem repeats (ESTRs) provide highly informative loci for analyzing spontaneous and induced germline mutation. We have conducted an extensive sequence database search and identified 17 new members of the highly unstable rodent-specific ESTR family called MMS10. This family has arisen by independent expansions of a common GGCAGA repeat unit from within a subset of both ancestral and modern SINE B1 elements during the course of mouse evolution. Analysis of the interspersion patterns of variant repeats along alleles of 20 of these MMS10 loci revealed two distinct classes of tandem arrays: one composed of uninterrupted GGCAGA repeats and the second with generally larger arrays interrupted by variant units. Surveys of allelic diversity at 11 representative members of these two classes of loci in various laboratory strains and BXD recombinant inbred lines revealed that the level of repeat instability was positively correlated with the length of uninterrupted repeats. Turnover processes at MMS10 loci, therefore, appear similar to the type of mechanism observed at human microsatellites. The MMS10 family thus provides a potentially useful murine model for studying dynamic mutation at simple tandem repeats.

CeMENT: evaluation of a regional development programme integrating hospital and general practice clinical teaching for medical undergraduates. The Community-Based Medical Education in North Thames.

OBJECTIVES: To determine the feasibility and effectiveness of shared hospital and general practice clinical teaching for medical undergraduates. DESIGN: A multifaceted approach employing quantitative and qualitative techniques. SETTING: All medical schools in North Thames Region. SUBJECTS: Students, GP tutors and hospital specialists. RESULTS: The model was successfully adopted in a broad range of clinical specialties in all of the participating medical schools, resulting in a doubling of the involvement of general practice in clinical teaching. Participating students provided an overwhelmingly positive evaluation of the attachments and there was a clear perception of benefit amongst the participating GPs. However, the views of the participating hospital clinicians were less positive and the true nature and extent of the educational impact proved difficult to assess. CONCLUSIONS: This model of collaborative clinical teaching between hospital and general practice can be implemented in accordance with the project's key aims, but the enthusiastic involvement of hospital clinicians may be difficult to secure.

The assessment of poorly performing doctors: the development of the assessment programmes for the General Medical Council's Performance Procedures.

BACKGROUND: Modernization of medical regulation has included the introduction of the Professional Performance Procedures by the UK General Medical Council in 1995. The Council now has the power to assess any registered practitioner whose performance may be seriously deficient, thus calling registration (licensure) into question. Problems arising from ill health or conduct are dealt with under separate programmes. METHODS: This paper describes the development of the assessment programmes within the overall policy framework determined by the Council. Peer review of performance in the workplace (Phase 1) is followed by tests of competence (Phase 2) to reflect the relationship between clinical competence and performance. The theoretical and research basis for the approach are presented, and the relationship between the qualitative methods in Phase 1 and the quantitative methods in Phase 2 explored. CONCLUSIONS: The approach is feasible, has been implemented and has stood legal challenge. The assessors judge and report all the evidence they collect and may not select from it. All their judgements are included and the voice of the lay assessor is preserved. Taken together, the output from both phases forms an important basis for remediation and training should it be required.

The General Medical Council's Performance Procedures: the development and implementation of tests of competence with examples from general practice.

OBJECTIVE: This paper describes the development of the tests of competence used as part of the General Medical Council's assessment of potentially seriously deficient doctors. It is illustrated by reference to tests of knowledge and clinical and practical skills created for general practice. SUBJECTS AND TESTS: A notional sample of 30 volunteers in 'good standing' in the specialty (reference group), 27 practitioners referred to the procedures and four practitioners not referred but who were the focus of concern over their performance. Tests were constructed using available guidelines and a specially convened working group in the specialty. METHODS: Standards were set using Angoff, modified contrasting group and global judgement methods, as appropriate. RESULTS: Tests performed highly reliably, showed evidence of construct validity, intercorrelated at appropriate levels and, at the standards employed, demonstrated good separation of reference and referred groups. Likelihood ratios for above and below standard performance based on competence were large for each test. Seven of 27 doctors referred were shown not to be deficient in both phases of the performance assessment.

Training the assessors for the General Medical Council's Performance Procedures.

From July 1997, the General Medical Council (GMC) has had the power to investigate doctors whose performance is considered to be seriously deficient. Assessment procedures have been developed for all medical specialties to include peer review of performance in practice and tests of competence. Peer review is conducted by teams of at least two medical assessors and one lay assessor. A comprehensive training programme for assessors has been developed that simulates the context of a typical practice-based assessment and has been tailored for 12 medical specialties. The training includes the principles of assessment, familiarization with the assessment instruments and supervised practice in assessment methods used during the peer review visit. High fidelity is achieved through the use of actors who simulate third party interviewees and trained doctors who role play the assessee. A subgroup of assessors, selected to lead the assessment teams, undergo training in handling group dynamics, report writing and in defending the assessment report against legal challenge. Debriefing of assessors following real assessments has been strongly positive with regard to their preparedness and confidence in undertaking the assessment.

The reproducibility of assessing radiological reporting: studies from the development of the General Medical Council's Performance Procedures.

OBJECTIVES: To investigate the reproducibility of peer ratings of consultant radiologists' reports, as part of the new General Medical Council (GMC) Performance Procedures. DESIGN: An evaluation protocol was piloted, used in a blocked, balanced, randomized generalizability analysis with three blocks of three judges (raters), each rating 30 reports from 10 radiologists, and re-rated to estimate intrarater reliability with conventional statistics (kappa). SETTING: Rating was performed at the Royal College of Radiologists. Volunteers were sampled from 23 departments of radiology in university teaching and district general hospitals. PARTICIPANTS: A nationally drawn non-random sample of 30 consultant radiologists contributing a total of 900 reports. Three trained and six non-trained judges were used in the rating analysis. RESULTS: A protocol was generated that was usable by judges. Generalizable results would be obtained with not less than three judges all rating the same 60 reports from a radiologist. CONCLUSIONS: Any assessment of performance of technical abilities in this field will need to use multiple assessors, basing judgements on an adequate sample of reports.

The General Medical Council's Performance Procedures: peer review of performance in the workplace.

The General Medical Council procedures to assess the performance of doctors who may be seriously deficient include peer review of the doctor's practice at the workplace and tests of competence and skills. Peer reviews are conducted by three trained assessors, two from the same speciality as the doctor being assessed, with one lay assessor. The doctor completes a portfolio to describe his/her training, experience, the circumstances of practice and self rate his/her competence and familiarity in dealing with the common problems of his/her own discipline. The assessment includes a review of the doctor's medical records; discussion of cases selected from these records; observation of consultations for clinicians, or of relevant activities in non-clinicians; a tour of the doctor's workplace; interviews with at least 12 third parties (five nominated by the doctor); and structured interviews with the doctor. The content and structure of the peer review are designed to assess the doctor against the standards defined in Good Medical Practice, as applied to the doctor's speciality. The assessment methods are based on validated instruments and gather 700-1000 judgements on each doctor. Early experience of the peer review visits has confirmed their feasibility and effectiveness.

Learning together in medical and nursing training: aspirations and activity.

CONTEXT: Policy documents about service innovation, education priorities and professional development exhort professions to learn together and work collaboratively. However, the literature suggests that the existence of shared learning in medical and nursing pre-qualifying education is patchy. AIM: This paper does not claim to be research. It sets out to reflect on the trends and tensions in key policy directions, relating these to aspirations and a mapping of current intiatives in the sphere of medical and nursing pre-qualifying education. APPROACH: A limited national information gathering exercise was conducted during the planning phase of seminars hosted by the Centre for the Advancement of Interprofessional Education (CAIPE) in 1996 and 1997. This involved directly contacting all medical schools and departments of nursing and midwifery in geographical proximity, or with an institutional relationship. Information was sought on current or planned activity in shared learning, defined as medical and nursing students and/or working together. EMERGING THEMES: There were a few examples of shared learning identified by the mapping exercise. The paper discusses these and draws on the consensus that emerged from the seminars on objectives and topics for shared learning. It concludes with a discussion of what makes for success or failure in such ventures with suggestions for future educational policy development.

Patient-oriented learning: a review of the role of the patient in the education of medical students.

AIM: To explore the contribution patients can make to medical education from both theoretical and empirical perspectives, to describe a framework for reviewing and monitoring patient involvement in specific educational situations and to generate suggestions for further research. METHODS: Literature review. RESULTS: Direct contact with patients can be seen to play a crucial role in the development of clinical reasoning, communication skills, professional attitudes and empathy. It also motivates through promoting relevance and providing context. Few studies have explored this area, including effects on the patients themselves, although there are examples of good practice in promoting more active participation. CONCLUSION: The Cambridge framework is a tool for evaluating the involvement of patients in the educational process, which could be used by curriculum planners and teachers to review and monitor the extent to which patients are actively involved. Areas for further research include looking at the 'added value' of using real, as opposed to simulated, patients; more work on outcomes for patients (other than satisfaction); the role of real patients in assessment; and the strengths and weaknesses of different models of patient involvement.