Reflective Practice for Patient Benefit: An Analysis of Doctors' Appraisal Portfolios in Scotland.

INTRODUCTION: Reflective practice has become the cornerstone of continuing professional development for doctors, with the expectation that it helps to develop and sustain the workforce for patient benefit. Annual appraisal is mandatory for all practicing doctors in the United Kingdom as part of medical revalidation. Doctors submit a portfolio of supporting information forming the basis of their appraisal discussion where reflection on the information is mandated and evaluated by a colleague, acting as an appraiser. METHODS: Using an in-depth case study approach, 18 online portfolios in Scotland were examined with a template developed to record the types of supporting information submitted and how far these showed reflection and/or changes to practice. Data from semistructured interviews with the doctors (n = 17) and their appraisers (n = 9) were used to contextualize and broaden our understanding of the portfolios. RESULTS: Portfolios generally showed little written reflection, and most doctors were unenthusiastic about documenting reflective practice. Appraisals provided a forum for verbal reflection, which was often detailed in the appraisal summary. Portfolio examples showed that reflecting on continued professional development, audits, significant events, and colleague multisource feedback were sometimes considered to be useful. Reflecting on patient feedback was seen as less valuable because feedback tended to be uncritical. DISCUSSION: The written reflection element of educational portfolios needs to be carefully considered because it is clear that many doctors do not find it a helpful exercise. Instead, using the portfolio to record topics covered by a reflective discussion with a facilitator would not only prove more amenable to many doctors but would also allay fears of documentary evidence being used in litigation.

Appraisal; the next steps?

Appraisal offers an opportunity for both the clinician and the employer to demonstrate commitment to their professional responsibilities in a tangible manner that can benefit patients, the individual clinician and the service. One unintentional consequence of revalidation may have been to foster a belief that it has undermined the educational and developmental aspects of appraisal. The objective of promoting transparency and accountability through revalidation may have led some clinicians to feel they work in a culture of greater scrutiny and assessment, leading to an erosion of their engagement with the process. Some appraisers previously comfortable with the educational and supportive ethos of appraisal may be less likely to continue in that role if they are viewed or see themselves as GMC assessors. Priorities include a need for research to evaluate the impact and outcomes of appraisal and revalidation, to evaluate how appraisal can meaningfully feed into improving patient care and the contribution that medical education can make to the process.

How should we train consultant appraisers? Description and evaluation of a pilot training model developed in Scotland.

OBJECTIVES: This paper outlines a model developed by NHS Education for Scotland to pilot consultant appraiser training. The training was aimed at both new/inexperienced appraisers and experienced appraisers. A Course for New/Inexperienced Appraisers was held over 2 days and the Experienced Course over 1 day. Both are described, as is an initial tutor induction day designed to recruit consultant tutors who could contribute to delivering the courses. A thorough qualitative evaluation of the courses was undertaken and this is reported. The paper also discusses the desirability and implications of rolling out the pilot on a wider scale. METHOD: A qualitative evaluation design using interviews to collect feedback from participants who had attended all three courses - the Induction event, the Inexperienced Appraiser Course and the Experienced Appraiser Course - was employed. Telephone interviews were transcribed and analysed using framework analysis and aided by NVivo 7. RESULTS: The response of those interviewed for this evaluation of the three courses was very positive. The content, organization, skills of the tutors and usefulness of the courses were commended. Suggested improvements were relatively minor and included: introducing more material on the requirements of revalidation, more small group discussions and making the course more secondary care-orientated. However, it was recognized that with the requirements for 'enhanced' appraisal and revalidation still uncertain, this created difficulties and the course would evolve to take account of further developments. CONCLUSIONS: The training piloted represents an effective way to train consultants in appraisal skills. Interviewees were supportive of the pilot being continued: either nationally or cascaded down locally. Exactly what model is achievable and desirable will depend on the number of appraisers to be trained and the availability of resources to support 'enhanced' appraisal in secondary care.

Pilot study to develop and test a generic peer feedback instrument to review videotaped consultations in primary care.

BACKGROUND: The ability to consult effectively is key to the delivery of quality patient care. External peer review of consultations has been available to general medical practitioners (GPs) in the west of Scotland for several years. Pharmacists are expected to provide increasingly complex advice for patients. This study describes the development and first steps in the evaluation of a generic tool to be used to inform feedback about both GP and pharmacist consultations with patients. METHOD: The tool was developed by a small group of GPs and tested for validity using a content validity inventory. An item correlation pilot (ICP) was carried out. Three experienced reviewers reviewed and scored three consultations in each of six GP tapes, a further three reviewers reviewed and scored three consultations in each of four pharmacists' tapes. The results underwent statistical analysis. RESULTS: Pharmacists do not examine patients and so two similar instruments were developed, whereby two questions pertinent to patient examination were omitted from the pharmacist instrument. The content validity inventory demonstrated aspects of face validity and content validity of instruments. Statistical analysis of GP tapes suggested that the instrument could discriminate between GP consultations. Skills demonstrated by pharmacists were given lower scores, were less consistent than the GPs and the instrument did not discriminate between those performing at different levels. CONCLUSION: The results suggest that the instrument is useful only when applied to practitioners who have been taught how to consult in a patientcentred manner, and have led to the introduction of training in consultation skills for pharmacists throughout Scotland. Potential reliability in providing peer feedback for GP consultations is important in the context of the provision of consistent, meaningful evidence for GP appraisal and revalidation in the UK.

The implications of enhancing appraisal to meet the requirements of revalidation, as perceived by appraisers: a qualitative study in Scotland.

This paper describes a qualitative study which explored the possible impact of enhanced appraisal and the requirements of revalidation upon GP appraisers in Scotland. So far there has been little research examining the impact 'enhanced' appraisal may have on the appraisee or appraiser population. Key objectives were to identify potential benefits and problems with the introduction of enhanced appraisal for the appraiser population and to explore what they perceive its impact will be on the GP workforce more generally. Three focus groups were held with a purposive sample of 17 highly experienced GP appraisers from across Scotland between February and April 2010. The results showed that this group of appraisers perceived that they would need to become more prescriptive about the standard of basic paperwork and evidence that is submitted by an appraisee. This created some anxieties about taking on a more judgemental role. They were concerned about a possible need to inform a high-achieving GP that their evidence was insufficient. Concerns were also raised about appraisees becoming less open and candid during appraisal once the links to revalidation are in place. Uncertainty was expressed about the relationship between the appraiser and responsible officer (RO) and how appraisers should decide which appraisees ought to be flagged up to the RO. The need for clear, standardised summary forms (known as GP Scot 4 in Scotland) to be passed to the RO was highlighted. Some of the appraisers were unsure about the value of new types of evidence which will be required--notably MSF and continuing professional development (CPD) credits. However, a few appraisers noted that some early adopters of CPD credits liked this way of recording their learning whilst MSF could be valuable and was seldom controversial. Potential benefits of 'enhanced appraisal' were highlighted--particularly increased engagement from some GPs who have hitherto been reluctant appraisees. The appraisers in this study wanted further training for their role, particularly training and calibration in delivering MSF feedback and in verifying CPD credits. They were frustrated by delays to revalidation and concerned that enhanced appraisal might be implemented without sufficient support (remediation and IT systems) being in place. However, they remained cautiously optimistic that some of the formative elements of appraisal can be maintained and were content to continue as appraisers provided they receive appropriate training and support and provided adequate remediation systems are in place for those GPs requiring help.

What are the concerns of prospective responsible officers about their role in medical revalidation?

INTRODUCTION: The Health and Social Care Act 2008 sets out the requirement for the introduction of responsible officers for the NHS. The paper states that these individuals will be responsible for ensuring that doctors who are revalidated meet the required standards. It is not known how well prepared prospective responsible officers are for their new role. METHODS: Semi-structured interviews were conducted with 12 medical directors from Scottish health boards who agreed to participate. RESULTS: Prospective responsible officers remain uncertain about their role in medical revalidation. Specific concerns were raised: (1) What will responsible officers be responsible for? (2) How can appraisal be quality assured? (3) How will the information requirements for revalidation be met? and (4) How can organizations meet the requirement for revalidation? CONCLUSION: We found important issues regarding the arrangements for revalidation which have not previously been described which may impact on the effectiveness of prospective responsible officers.

A review of further training for GP appraisers in Scotland.

A one-day further intensive skills (FIS) course has been developed to provide additional training for existing general practitioner (GP) appraisers in Scotland. The course focuses on skills in developing the appraisee's personal development portfolio (PDP) and skills in responding to significant issues (with emotional content) presented by appraisees - both key areas for effective appraisals. The course is briefly described. An initial pilot of the course led to some changes being made to the content, as it was discovered that the inclusion of training on Summary Form completion in the pilot made the course too dense and distracted from the experiential elements of the training. Two subsequent course deliveries were evaluated by conducting semi-structured interviews with over 40% of the participants from these two courses. The main purpose of the evaluation was to discover to what extent the participants felt they had benefited from the training and whether they felt the training had led to any changes in their practice as appraisers. Secondary aims were to discover the acceptability of the use of video recording as a training technique and views regarding a reaccreditation process for appraisers. Key findings were that almost all participants found this training beneficial and judged it to have led to positive changes in their practice, as well as reassuring them that their skills were up to scratch. Some appraisers felt that it had encouraged a slightly more challenging approach. The acceptability of the use of video recording during the training, as a means of allowing participants to gain further benefit from the work undertaken after the course was finished, was tested. This technique received a mixed response, with opinion divided as to whether it added value. Opinion amongst the participants was also mixed regarding whether this training should contribute towards their reaccreditation as appraisers. However, a number of participants felt this could be a valid element in an appraiser reaccreditation process.

Verifying appraisal evidence using feedback from trained peers: views and experiences of Scottish GP appraisers.

BACKGROUND: GP appraisal is currently considered inadequate because it lacks robustness. Objective assessment of appraisal evidence is needed to enable judgements on professional performance to be made. AIM: To determine GP appraisers' views of the acceptability, feasibility, and educational impact of external peer feedback received on three core appraisal activities undertaken as part of this study. DESIGN OF STUDY: Independent peer review and cross-sectional postal questionnaire study. SETTING: NHS Scotland. METHOD: One of three core appraisal activities (criterion audit, significant event analysis, or video of consultations) was undertaken by GP appraisers and subjected to peer review by trained colleagues. A follow-up postal questionnaire elicited participants' views on the potential acceptability, feasibility, and educational impact of this approach. RESULTS: Of 164 appraisers, 80 agreed to participate; 67/80 (84%) submitted one of three appraisal materials for peer review and returned completed questionnaires. For significant event analyses (n = 44), most responders believed the peer feedback method was feasible (100%) and fair (92.5%) and would add value to appraisal (95.5%). Peer feedback on criterion audits (n = 15) was believed to be acceptable and fair (93.3%) and it was thought it would be a useful educational tool (100%). Completing a consultation video (n = 8) was perceived to be feasible as part of normal general practice (n = 5). It was unanimously agreed that assessment of videos by peers has educational impact and would help improve appraisal. CONCLUSION: This group of GP appraisers strongly supported the role of external and independent feedback by trained peers as one approach to strengthening the existing appraisal process.

GP peer appraisal in Scotland: an ongoing and developing exercise in quality.

GP appraisal in Scotland is a strong and vibrant process. Not least this is due to NES's commitment to the ongoing development of its appraisers. Research suggests that the key to effective appraisal is the skill of the appraiser. Training in Scotland has developed using feedback from a wide range of sources. The move from information-based courses towards skills-based courses enhances appraiser development and allows the effective evaluation of appraiser competencies. With appraisal becoming a key element in the revalidation process the quality assurance of appraisal and appraisers becomes increasingly important.

What do doctors really think about the relevance and impact of GP appraisal 3 years on? A survey of Scottish GPs.

BACKGROUND: The aim of appraisal is to provide an opportunity for individuals to reflect on their work to facilitate learning and development. Appraisal for GPs has been a contractual requirement since 2004 in Scotland, and is seen as an integral part of revalidation. AIM: To investigate the outcomes of GP appraisal in terms of whether it has prompted change in medical practice, education and learning, career development, attitudes to health and probity, how GPs organise their work, and their perception of the overall value of the process. DESIGN OF STUDY: A cross-sectional postal questionnaire. SETTING: GP performers in Scotland who had undertaken appraisal. METHOD: The questionnaire was based on the seven principles outlined in Good Medical Practice, a literature review, and previous local research. The survey was conducted on a strictly anonymous basis with a random, representative sample of GPs. RESULTS: Fifty-three per cent (671/1278) responded. Forty-seven per cent (308/661) thought that appraisal had altered their educational activity, 33% (217/660) reported undertaking further education or training as a result of appraisal, and 13% (89/660) felt that appraisal had influenced their career development. Opinion was evenly split on the overall value of appraisal. CONCLUSION: Appraisal can have a significant impact on all aspects of a GP's professional life, and those who value the process report continuing benefit in how they manage their education and professional development. However, many perceive limited or no benefit. The renewed emphasis on appraisal requires examination of these findings and discussion of how appraisal can become more relevant.