Expert validation of fit-for-purpose guidelines for designing programmes of assessment.

BACKGROUND: An assessment programme, a purposeful mix of assessment activities, is necessary to achieve a complete picture of assessee competence. High quality assessment programmes exist, however, design requirements for such programmes are still unclear. We developed guidelines for design based on an earlier developed framework which identified areas to be covered. A fitness-for-purpose approach defining quality was adopted to develop and validate guidelines. METHODS: First, in a brainstorm, ideas were generated, followed by structured interviews with 9 international assessment experts. Then, guidelines were fine-tuned through analysis of the interviews. Finally, validation was based on expert consensus via member checking. RESULTS: In total 72 guidelines were developed and in this paper the most salient guidelines are discussed. The guidelines are related and grouped per layer of the framework. Some guidelines were so generic that these are applicable in any design consideration. These are: the principle of proportionality, rationales should underpin each decisions, and requirement of expertise. Logically, many guidelines focus on practical aspects of assessment. Some guidelines were found to be clear and concrete, others were less straightforward and were phrased more as issues for contemplation. CONCLUSIONS: The set of guidelines is comprehensive and not bound to a specific context or educational approach. From the fitness-for-purpose principle, guidelines are eclectic, requiring expertise judgement to use them appropriately in different contexts. Further validation studies to test practicality are required.

Factors that might undermine the validity of patient and multi-source feedback.

CONTEXT: Multi-source feedback (MSF) and patient feedback (PF) are used increasingly around the world to assess and quality-assure clinical practice. However, concerns about the evidence for their utility pertain to their ability to identify poor performance, the impact of allowing assessees to select their own assessors and the many confounders that may undermine validity. METHODS: This study was conducted in conjunction with the National Clinical Assessment Service (NCAS) in the UK and used established MSF and PF instruments to assess doctors in potential difficulty. Multi-source feedback assessors were nominated by both the practitioner (Pnom) and the referring body (RBnom). Demographics were collected to elucidate any differences found. Ratings generated by MSF and PF were compared with one another and with findings of a previous study that provided a normative cohort. RESULTS: Using MSF, NCAS-assessed doctors scored significantly lower than the reference cohort. Nineteen (28%) NCAS-assessed doctors achieved scores that were less than satisfactory. This rose to 50% when only RBnom assessors were used. Overall, ratings awarded by RBnom assessors were significantly lower than those awarded by Pnom assessors. Collected demographics did not help to explain the difference. Only one NCAS-assessed doctor scored below average according to PF. Doctors in the NCAS-assessed group did not score significantly lower than the reference cohort in PF. Doctor assessment scores awarded by patients were significantly higher than those awarded by colleagues. CONCLUSIONS: Although colleagues appear to report poor performance using MSF, patients fail to report concurrent findings. This challenges the validity of PF as it is currently constructed. Scores in MSF differ significantly depending on whether they are practitioner- or third party-nominated. Previously recognised confounding factors do not help to explain this difference.

Can a district hospital assess its doctors for re-licensure?

CONTEXT: The Chief Medical Officer's recommendations on medical regulation in the UK suggest that National Health Service (NHS) trusts should assess their doctors and confirm whether they remain fit to practise medicine. OBJECTIVE: We set out to evaluate the utility of hospital trust-based assessment in a 'best-case scenario' within existing resources. METHODS: We carried out a generalisability analysis, and feasibility and validity evaluation, based on an assessment process for 137 career-grade doctors at Chesterfield Royal Hospital, Chesterfield, UK, using validated multi-source feedback (MSF) and patient rating (PR) instruments. RESULTS: Uptake and response rates were good for MSF (91% and 85%, respectively). However, only 6% of non-clinical doctors and anaesthetists, and 48% of clinical doctors, obtained sufficient PR ratings. Aggregate scores were acceptably reliable. Nine combined MSF ratings and 15 PR ratings produce standard errors of measurement of 0.19 on a 6-point scale and 0.15 on a 5-point scale, respectively. Overall aggregate scores did not identify any doctor as unsatisfactory, but 6 doctors were scored as unsatisfactory by 2 or more colleagues or patients. These performance concerns appear to merit further investigation. Patients rated female doctors better than male doctors (4.61 versus 4.46; P < 0.05). Colleagues rated UK graduates better than non-UK graduates (5.31 versus 5.15; P < 0.05). CONCLUSIONS: This study shows that the commissioning of professional services makes the implementation of an assessment process linked to appraisal feasible. However, trust-based assessment requires significant development: developmental appraisal needs protection; new instruments are needed for non-clinical specialties; PR requires specific administrative support, and guidance is required over concern thresholds and demographic effects. Disaggregated assessment data may help identify doctors with potential performance problems.

Assessing the competence of practicing physicians in New Zealand, Canada, and the United Kingdom: progress and problems.

Members of the public expect practicing physicians to be competent. They expect poorly performing physicians to be identified and either helped or removed from practice. "Maintenance of professional standards" by continuing education does not identify the poorly performing physician; assessment of clinical performance is necessary for that. Assessment may be responsive-ie, following a complaint- or periodic, either for all physicians or for an identified high-risk group. A thorough review using a range of tools is appropriate for a responsive assessment but is not practical for periodic assessment for all. A single, valid, reliable, and practical screening tool has yet to be devised to identify physicians whose practice is suboptimal. Further, articulate commentators are concerned about the harm that too-intensive scrutiny of professional performance may cause. We conclude that high performance by all physicians throughout their careers cannot be fully ensured, but it is nonetheless the responsibility of licensing bodies to use reasonable methods to determine whether performance remains acceptable. Such methods should be shown scientifically to be accurate, valid, and reliable for practicing physicians. Such an approach is likely to encourage the agreement and cooperation of the profession. To do less risks losing the trust of the public.

Defining insight: a challenge that matters.

BACKGROUND: Insight is a fundamental consideration in the maintenance and improvement of performance. What is required is a definition that can be used consistently along with an understanding of the dynamic nature of insight, and therefore its potential for improvement. CONTEXT: The work presented is a review of contemporary literature about insight, self-awareness and emotional intelligence, and experience from the National Clinical Assessment Service (NCAS) in the assessment of 300 doctors with performance problems. SUMMARY OF RESULTS: This work has led to the construction of a working definition of insight that incorporates reflection, emotional intelligence, self-awareness and motivation. The validity and utility of this definition was tested using data from NCAS's growing experience. CONCLUSIONS: The assessment of insight in NCAS performance assessments has involved the use of information from a variety of sources, including psychometric profiling, multisource feedback and self-assessment. This has been corroborated with findings from clinical performance assessment and the outcomes of remediation plans. IMPLICATIONS: A definition of insight is offered that is greater than just self-awareness, and that involves emotional intelligence and motivation. The use of this broad definition is of fundamental importance in the production of educational and development programmes at all levels.

A retrospective study of cognitive function in doctors and dentists with suspected performance problems: an unsuspected but significant concern.

OBJECTIVE: To examine the performance assessments and cognitive function of practitioners referred to the National Clinical Assessment Service (NCAS). DESIGN: Retrospective observational study. SETTING: Practitioners referred to NCAS for performance assessment due to suspected performance problems. PARTICIPANTS: One hundred and nine practitioners over the age of 45 years referred to NCAS between 1 September 2008 and 30 June 2012. MAIN OUTCOME MEASURES: Reasons for referral of practitioners and their characteristics; details of their assessments including screening for cognition using Addenbrooke's Cognitive Examination Revised (ACE-R); outcome of the process. RESULTS: Reasons for referral included 'clinical difficulties' and 'governance or safety issues'. Eighty-seven practitioners scored above 88 on ACE-R. Twenty-two were found to have an ACE-R score of ≤88. On further assessment, 14 of these 22 practitioners were found to have cognitive impairment. The majority of all practitioners were found to be performing below the expected level of practice for someone at their grade and specialty. Of those scoring ≤88 on the screening, only seven continued in clinical practice. CONCLUSIONS: A high proportion of practitioners scoring poorly on ACE-R were found to have cognitive impairment following detailed neuropsychological testing, the youngest aged 46 years. Many were working in isolation. Nearly all practitioners scoring poorly on ACE-R were international medical graduates; reasons for this are unclear. Performance assessment results showed persisting failings in the practitioners' record keeping and in their assessment of patients. Our findings highlight the need for increased vigilance and training of responsible officers to recognise performance problems and emphasise the importance of comprehensive assessment.

Identification of poor performance in a national medical workforce over 11 years: an observational study.

BACKGROUND: Poorly performing doctors are a source of harm but do not commonly feature in discussions of patient safety. Few countries have national mechanisms to deal with these doctors; most opt for suspension and/or exclusion from clinical practice. This study reports on the 11-year experience of dealing with concerns about doctors' performance in the UK National Health Service (NHS). The aim of this study was to describe the frequency with which doctors were referred due to performance-related concerns, examine demographic and specialty differences, and identify the nature of the concerns prompting referral. METHODS: This observational study uses data collected by the National Clinical Assessment Service for each referral (n=6179 doctors) over an 11-year period (April 2001-March 2012) in England to examine the rate at which concerns about doctors' performance occur, understand differences in rates between practitioner groups, and changes over time. FINDINGS: The annual referral rate was five per 1000 doctors (95% CI 4.6 to 5.4). Doctors whose first medical qualification was gained outside the UK were more than twice as likely to be referred as UK-qualified doctors; male doctors were more than twice as likely to be referred as women doctors; and doctors in the late stages of their career were nearly six times as likely to be referred as early career doctors. DISCUSSION: The UK holds a consistently collected national dataset on performance concerns about doctors. This allows risk groups to be identified so that preventive action and early intervention can be targeted most effectively to reduce harm to patients. A feature of past handling of poor clinical performance has been late presentation and a lack of thematic study of causation.