Development of a GMC aligned curriculum for internal medicine including a qualitative study of the acceptability of 'capabilities in practice' as a curriculum model.

The Shape of Training review and the General Medical Council (GMC) requirements in demonstrating generic professional capabilities were major drivers for the Joint Royal Colleges of Physicians Training Board (JRCPTB) to develop the new internal medicine curriculum. In particular, the GMC required progression to a more outcomes-based curriculum. The present curricula for physician training are based on demonstrating a large number of individual competencies that are assessed by a variety of different methods. It was felt that current system was overwhelming and had become a 'tick box' exercise. The new curriculum is based on a much smaller number of outcomes, called capabilities in practice, which reflect the key professional work activities of a fully trained physician. The aim is to re-emphasise the role of professional judgement in 'trusting' the work a trainee does and thus make assessment more realistic and meaningful for both trainees and trainers. The proof of concept study explored the feasibility of using this outcomes-based model of assessment in a UK NHS setting. The learning from the study has enabled us to make significant changes to the internal medicine curriculum. The GMC has recently approved the curriculum and the JRCPTB is implementing the programme from August 2019.

Lessons learnt from a pilot of assessment for learning.

A 12-month pilot was carried out on assessments for learning and assessments of learning as part of workplace-based assessments in postgraduate medical education. This was carried out in three regions and core medical trainees and higher specialty medical trainees participated. Focus groups and questionnaires were utilised to investigate the trainees' and trainers' experiences and perceptions of assessments for learning. The study demonstrated that the trainees and trainers perceived the newly introduced assessments for learning--supervised learning events (SLEs)--as learning tools. However, SLEs were often undertaken with no previous organisation and with no direct observation, regardless of the underlying purposes and methods of the WPBAs. There was a lack of, or delayed or non-specific, feedback following SLEs, which would have impeded its educational value. Trainee and trainer disengagement was one of the contributing factors. These findings are valuable in informing and facilitating future successful implementation of assessments for learning.

Do assessor comments on a multi-source feedback instrument provide learner-centred feedback?

OBJECTIVES: Free-text comments in multi-source feedback are intended to facilitate change in the assessee's practice. This study was designed to utilise a large dataset of free-text comments obtained in a national pilot study in order to investigate how helpful these free-text comments may be to assessees. METHODS: We investigated: (i) which areas of performance are usually addressed by free-text comments; (ii) to what extent assessors' (doctors, nurses, allied health professionals and clerical or managerial staff) comments correspond to assessees' (career-grade doctors) self-assessments, and (iii) whether the comments contain specific behavioural evidence and suggestions for change. Initially comments were read through to identify commonly recurring themes. A strong theme was 'respondent-centredness', which refers to the extent to which comments focus on issues that are of value to the assessor rather than to the assessee's personal development. In response to this, the data were re-evaluated against predefined research questions to assess how constructive comments were for the assessee's personal development. RESULTS: Of 11,483 assessor forms, 4777 (42%) included free-text comments. A total of 513 forms contained at least one below average score and 286 (56%) of these forms contained the assessor's free-text feedback. Free-text comments were mostly rater-centred and addressed the effect of the assessee on the colleague's working life rather than areas of relevance to the assessee's personal development. A total of 1806 assessor/assessee pairs of comments were compared; most demonstrated clear differences of opinion or interpretation. Reliability and supportiveness were over-represented; clinical performance and personal development were under-represented. The comments were unlikely to provide specific behavioural evidence or to address how change might be initiated. CONCLUSIONS: Our data indicate that, in their current form, the overwhelming majority of free-text comments add little to facilitate improvement in assessees' personal development and performance.

Assessment of professionalism: recommendations from the Ottawa 2010 Conference.

Over the past 25 years, professionalism has emerged as a substantive and sustained theme, the operationalization and measurement of which has become a major concern for those involved in medical education. However, how to go about establishing the elements that constitute appropriate professionalism in order to assess them is difficult. Using a discourse analysis approach, the International Ottawa Conference Working Group on Professionalism studied some of the dominant notions of professionalism, and in particular the implications for its assessment. The results presented here reveal different ways of thinking about professionalism that can lead towards a multi-dimensional, multi-paradigmatic approach to assessing professionalism at different levels: individual, inter-personal, societal-institutional. Recommendations for research about professionalism assessment are also presented.

Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales.

CONTEXT: Assessment in the workplace is important, but many evaluations have shown that assessor agreement and discrimination are poor. Training discussions suggest that assessors find conventional scales invalid. We evaluate scales constructed to reflect developing clinical sophistication and independence in parallel with conventional scales. METHODS: A valid scale should reduce assessor disagreement and increase assessor discrimination. We compare conventional and construct-aligned scales used in parallel to assess approximately 2000 medical trainees by each of three methods of workplace-based assessment (WBA): the mini-clinical evaluation exercise (mini-CEX); the acute care assessment tool (ACAT), and the case-based discussion (CBD). We evaluate how scores reflect assessor disagreement (V(j) and V(j*p) ) and assessor discrimination (V(p) ), and we model reliability using generalisability theory. RESULTS: In all three cases the conventional scale gave a performance similar to that in previous evaluations, but the construct-aligned scales substantially reduced assessor disagreement and substantially increased assessor discrimination. Reliability modelling shows that, using the new scales, the number of assessors required to achieve a generalisability coefficient ≥0.70 fell from six to three for the mini-CEX, from eight to three for the CBD, from 10 to nine for 'on-take' ACAT, and from 30 to 12 for 'post-take' ACAT. CONCLUSIONS: The results indicate that construct-aligned scales have greater utility, both because they are more reliable and because that reliability provides evidence of greater validity. There is also a wider implication: the disappointing reliability of existing WBA methods may reflect not assessors' differing assessments of performance, but, rather, different interpretations of poorly aligned scales. Scales aligned to the expertise of clinician-assessors and the developing independence of trainees may improve confidence in WBA.

Assessing trainees in the workplace: results of a pilot study.

This paper outlines the development and evaluation of the utility of workplace-based assessments in higher medical training: case-based discussion (CbD); the acute care assessment tool (ACAT); audit assessment; teaching observation and patient survey (PS). The study population included trainees in higher medical training (ST3+) from physician specialties in the U.K. The pilot consisted of a prospective study of the use of the new assessments using local study coordinators (LSCs) and volunteer trainees. In total, 169 LSCs were recruited and 134 trainees returned at least one assessment. The end-of-pilot questionnaire was returned by 44 assessors and 57 trainees. Questionnaire data and qualitative feedback were used to evaluate the validity, impact and feasibility of the new tools. For adequate reliability (co-efficient 0.7) a total of 12 CbDs; three ACATs and 16 PS raters are required. There was evidence for the validity and positive educational impact of all the tools. There were difficulties with the feasibility of the PS.

A single generic multi-source feedback tool for revalidation of all UK career-grade doctors: does one size fit all?

BACKGROUND: The UK Department of Health is considering a single, generic multi-source feedback (MSF) questionnaire to inform revalidation. METHOD: Evaluation of an implementation pilot, reporting: response rates, assessor mix, question redundancy and participants' perceptions. Reliability was estimated using Generalisability theory. RESULTS: A total of 12,540 responses were received on 977 doctors. The mean time taken to complete an MSF exercise was 68.2 days. The mean number of responses received per doctor was 12.0 (range 1-17) with no significant difference between specialties. Individual question response rates and participants' comments about questions indicate that some questions are less appropriate for some specialities. There was a significant difference in the mean score between specialities. Despite guidance, there were significant differences in the mix of assessors across specialties. More favourable scores were given by progressively more junior doctors. Nurses gave the most reliable scores. CONCLUSIONS: It is feasible to electronically administer a generic questionnaire to a large population of doctors. Generic content is appropriate for most but not all specialties. The differences in mean scores and the reliability of the MSF between specialties may be in part due to the specialty differences in assessor mix. Therefore the number and assessor mix should be standardised at specialty level and scores should not be compared across specialties.

A blueprint to assess professionalism: results of a systematic review.

PURPOSE: Assessing professionalism is hampered by varying definitions and these definitions' lack of a clear breakdown of the elements of professionalism into aspects that can be measured. Professionalism is multidimensional, so a combination of assessment tools is required. In this study, conducted during 2007-2008, the authors aimed to match assessment tools to definable elements of professionalism and to identify gaps where professionalism elements are not well addressed by existing assessment tools. METHOD: The authors conducted literature reviews of definitions of professionalism and of relevant assessment tools, clustered the definitions of professionalism into assessable components, and clustered assessment tools of a similar nature. They then created a "blueprint" whereby the elements of professionalism are matched to relevant assessment tools. RESULTS: Five clusters of professionalism were formed: adherence to ethical practice principles, effective interactions with patients and with people who are important to those patients, effective interactions with people working within the health system, reliability, and commitment to autonomous maintenance / improvement of competence in oneself, others, and systems. Nine clusters of assessment tools were identified: observed clinical encounters, collated views of coworkers, records of incidents of unprofessionalism, critical incident reports, simulations, paper-based tests, patients' opinions, global views of supervisor, and self-administered rating scales. CONCLUSIONS: Professionalism can be assessed using a combination of observed clinical encounters, multisource feedback, patients' opinions, paper-based tests or simulations, measures of research and/or teaching activities, and scrutiny of self-assessments compared with assessments by others. Attributes that require more development in their measurement are reflectiveness, advocacy, lifelong learning, dealing with uncertainty, balancing availability to others with care for oneself, and seeking and responding to results of an audit.

Feedback from educational supervisors and trainees on the implementation of curricula and the assessment system for core medical training.

A pilot of core medical training (CMT) was conducted in 2006-7 with 160 trainees and 130 supervisors in the 10 hospitals within the Mersey Deanery. Questionnaires and focus groups were used to gain feedback from trainees and supervisors in relation to the components of CMT (the curricula, workplace-based assessments, appraisal, and the e-portfolio). There was generally a positive attitude to the CMT package. In particular the opportunities to give and receive feedback were appreciated; the e-portfolio was identified as helpful for recording assessment outcomes and supporting educational development for the trainees. The workplace-based assessments were well received. Many of the benefits of the components of CMT depended on the skill of the supervisor. The time required for effective training supervision and workplace-based assessments was identified as an important issue. This pilot was invaluable in informing the widespread implementation of CMT in 2007.

Implementing workplace-based assessment across the medical specialties in the United Kingdom.

OBJECTIVES: To evaluate the reliability and feasibility of assessing the performance of medical specialist registrars (SpRs) using three methods: the mini-clinical evaluation exercise (mini-CEX), directly observed procedural skills (DOPS) and multi-source feedback (MSF) to help inform annual decisions about the outcome of SpR training. METHODS: We conducted a feasibility study and generalisability analysis based on the application of these assessment methods and the resulting data. A total of 230 SpRs (from 17 specialties) in 58 UK hospitals took part from 2003 to 2004. Main outcome measures included: time taken for each assessment, and variance component analysis of mean scores and derivation of 95% confidence intervals for individual doctors' scores based on the standard error of measurement. Responses to direct questions on questionnaires were analysed, as were the themes emerging from open-comment responses. RESULTS: The methods can provide reliable scores with appropriate sampling. In our sample, all trainees who completed the number of assessments recommended by the Royal Colleges of Physicians had scores that were 95% certain to be better than unsatisfactory. The mean time taken to complete the mini-CEX (including feedback) was 25 minutes. The DOPS required the duration of the procedure being assessed plus an additional third of this time for feedback. The mean time required for each rater to complete his or her MSF form was 6 minutes. CONCLUSIONS: This is the first attempt to evaluate the use of comprehensive workplace assessment across the medical specialties in the UK. The methods are feasible to conduct and can make reliable distinctions between doctors' performances. With adaptation, they may be appropriate for assessing the workplace performance of other grades and specialties of doctor. This may be helpful in informing foundation assessment.

Laying the foundations for good medical practice: a new educational resource.

Balancing service commitments with educational development is never easy. The European Working Time Directive and the Chief Medical Officer's report on senior house officer training bring new challenges. The Royal College of Physicians has developed 'Laying the Foundations for Good Medical Practice', a resource that enables tutors to teach generic skills to small groups of trainees in a fun and interactive way.

Assessing the performance of specialist registrars.

Assessing the performance of doctors while they are engaged in clinical work is a challenging concept. The introduction of objective-based curricula provides the stimulus and opportunity for the Royal Colleges of Physicians to develop relevant and reliable methods of in-service assessment. We propose to pilot a study investigating the validity, reliability and feasibility of three assessment methods--direct observation of the clinical encounter using an adapted mini-CEX, direct observation of the performance of practical procedures (DOPS), and the doctor's ability to perform effectively as part of a team using 360 degree assessment. The methods will be studied in the setting of routine clinical care. Whilst demanding of time from both trainees and trainers, they will represent a significant advance on the current system which is characterised by a lack of evidence in the assessment process.