Peer evaluation and feedback for invasive medical procedures: a systematic review.

BACKGROUND: There is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures. METHODS: A systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available. RESULTS: Of 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review. CONCLUSIONS: Peer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.

Toxicologic Pathology Forum*: Opinion on Considerations for the Use of Whole Slide Images in GLP Pathology Peer Review.

Whole slide imaging (WSI) technology has advanced to a point where it has replaced the glass slide as the primary means of pathology evaluation within many areas of medical pathology. The deployment of WSI in the field of toxicologic pathology has been delayed by a lack of clarity around the degree of validation required for its use on Good Laboratory Practice (GLP) studies. The current opinion piece attempts to provide a high-level overview of WSI technology to include basic methodology, advantages and disadvantages over a conventional microscope, validation status of WSI scanners, and perceived concerns over regulatory acceptance for the use of WSI for (GLP) peer review in the field of toxicologic pathology. Observations are based on the extensive use by AstraZeneca of WSI for the peer review of non-GLP studies conducted at Charles River facilities and represent the experiences of the authors. Note: This is an opinion article submitted to the Toxicologic Pathology Forum. It represents the views of the author(s). It does not constitute an official position of the Society of Toxicologic Pathology, British Society of Toxicological Pathology, or European Society of Toxicologic Pathology, and the views expressed might not reflect the best practices recommended by these Societies. This article should not be construed to represent the policies, positions, or opinions of their respective organizations, employers, or regulatory agencies.

Some assembly required: tracing the interpretative work of Clinical Competency Committees.

OBJECTIVES: This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. METHODS: Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi-structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. RESULTS: Members of the CCCs held an assumption that they would be presented with high-quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed 'problematic evidence' (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with 'problematic evidence', they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. CONCLUSIONS: Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement.

Daily peer review of abnormal cervical smears in the assessment of individual practice as an additional method of internal quality control.

OBJECTIVE: An important internal quality control system used in the Cancer Prevention and Research Institute cytopathology laboratory in Florence is the peer review procedure, based on the review of all abnormal cytological smears which routinely emerge. Peer review is an important training opportunity for all cytologists, especially for those with less experience. This article shows the results of the peer review procedure. METHODS: Of the 63 754 Papanicolaou (Pap) smears screened in 2011, 1086 were considered to be abnormal [at least atypical squamous cells of undetermined significance (ASC-US+)] on primary screening (selected by a single cytologist) and were subjected to the peer review procedure. The overall performance of the laboratory's cytologists was evaluated using a multiple rater analysis and the comparison of each cytologist with the final diagnosis. Further, the agreement was assessed by means of Cohen's kappa and weighted kappa statistics. RESULTS: In general, a moderate/substantial level of agreement between the ten cytologists and the final diagnoses was evident. Kappa values for each reader compared with the final diagnosis ranged from 0.54 to 0.69. The overall kappa value was 0.62 [95% confidence interval (CI), 0.58-0.66] and overall weighted kappa value was 0.76 (95% CI, 0.74-0.79). The category-specific agreement showed the lowest values for atypical squamous cells, cannot exclude high-grade squamous intraepithelial lesion (ASC-H). CONCLUSION: In summary, peer review represents an important internal quality control in the evaluation and improvement of inter-observer agreement and of the functioning of the laboratory as a whole. Multi-head microscope sessions may improve particularly the reproducibility of borderline diagnoses and, above all, can be an important training contribution for cytologists.

[Quality Assurance in Sociomedical Evaluation by Peer Review: A Pilot Project of the German Statutory Pension Insurance]. / Qualitätssicherung in der Sozialmedizinischen Begutachtung durch Peer Review - ein Pilotprojekt der Deutschen Rentenversicherung.

BACKGROUND: The sociomedical evaluation by the German Pension Insurance serves the purpose of determining entitlement to disability pensions. A quality assurance concept for the sociomedical evaluation was developed, which is based on a peer Review process. Peer review is an established process of external quality assurance in health care. The review is based on a hierarchically constructed manual that was evaluated in this pilot project. METHODS: The database consists of 260 medical reports for disability pension of 12 pension insurance agencies. 771 reviews from 19 peers were included in the evaluation of the inter-rater reliability. Kendall's coefficient of concordance W for more than 2 raters is used as primary measure of inter-rater reliability. RESULTS: Reliability appeared to be heterogeneous. Kendalls W varies for the particular criteria from 0.09 to 0.88 and reached for primary criterion reproducibility a value of 0.37. CONCLUSION: The reliability of the manual seemed acceptable in the context of existing research data and is in line with existing peer review research outcomes. Nevertheless, the concordance is limited and requires optimisation. Starting points for improvement can be seen in a systematic training and regular user meetings of the peers involved.

[Systemic error analysis as a key element of clinical risk management]. / Die systemische Fehleranalyse als zentrales Instrument des klinischen Risikomanagements.

Systemic error analysis plays a key role in clinical risk management. This includes all clinical and administrative activities which identify, assess and reduce the risks of damage to patients and to the organization. The clinical risk management is an integral part of quality management. This is also the policy of the Federal Joint Committee (Gemeinsamer Bundesausschuss, G-BA) on the fundamental requirements of an internal quality management. The goal of all activities is to improve the quality of medical treatment and patient safety. Primarily this is done by a systemic analysis of incidents and errors. A results-oriented systemic error analysis needs an open and unprejudiced corporate culture. Errors have to be transparent and measures to improve processes have to be taken. Disciplinary action on staff must not be part of the process. If these targets are met, errors and incidents can be analyzed and the process can create added value to the organization. There are some proven instruments to achieve that. This paper discusses in detail the error and risk analysis (ERA), which is frequently used in German healthcare organizations. The ERA goes far beyond the detection of problems due to faulty procedures. It focuses on the analysis of the following contributory factors: patient factors, task and process factors, individual factors, team factors, occupational and environmental factors, psychological factors, organizational and management factors and institutional context. Organizations can only learn from mistakes by analyzing these factors systemically and developing appropriate corrective actions. This article describes the fundamentals and implementation of the method at the University Medical Center Hamburg-Eppendorf.

A comparison of orthodontic treatment outcomes using the Objective Grading System (OGS) and the Peer Assessment Rating (PAR) index.

INTRODUCTION: The use of objective criteria is essential to uniformly quantify and measure the severity of malocclusions and the efficacy of different treatment modalities. The Peer Assessment Rating (PAR) index and, more recently, the American Board of Orthodontics Objective Grading System (OGS) were developed to fulfill this need. AIM: The aim of this retrospective study was to assess and compare treatment outcomes using the UK and US weighted PAR and the OGS. MATERIALS AND METHODS: The sample consisted of randomly selected records of 50 patients treated by residents in one postgraduate orthodontic clinic. UK and US weightings for the PAR index were applied and compared with OGS. RESULTS: There was no statistically significant association between the OGS and the PAR index grading systems. Neither the UK nor the US PAR weightings showed statistically significant correlation with the OGS. All cases were 'greatly improved' or 'improved' according to the PAR index, while most cases (62%) failed according to OGS. There was a statistically significant correlation between the unweighted PAR index and the OGS (r = -0.32, p = 0.024). The US and the UK weightings for the PAR were highly correlated (r = 0.90, p < 0.001). Both weighting systems were also highly correlated with the unweighted PAR (p < 0.001). There were no gender differences found in any of the scoring systems. CONCLUSIONS: The current PAR index cannot replace the OGS for evaluating treatment outcomes. The current OGS cannot detect the improvement achieved in a treated case.

Emergency department patient safety incident characterization: an observational analysis of the findings of a standardized peer review process.

BACKGROUND: Emergency Department (ED) care has been reported to be prone to patient safety incidents (PSIs). Improving our understanding of PSIs is essential to prevent them. A standardized, peer review process was implemented to identify and analyze ED PSIs. The primary objective of this investigation was to characterize ED PSIs identified by the peer review process. A secondary objective was to characterize PSIs that led to patient harm. In addition, we sought to provide a detailed description of the peer review process for others to consider as they conduct their own quality improvement initiatives. METHODS: An observational study was conducted in a large, urban, tertiary-care ED. Over a two-year period, all ED incident reports were investigated via a standardized, peer review process. PSIs were identified and analyzed for contributing factors including systems failures and practitioner-based errors. The classification system for factors contributing to PSIs was developed based on systems previously reported in the emergency medicine literature as well as the investigators' experience in quality improvement and peer review. All cases in which a PSI was discovered were further adjudicated to determine if patient harm resulted. RESULTS: In 24 months, 469 cases were investigated, identifying 152 PSIs. In total, 188 systems failures and 96 practitioner-based errors were found to have contributed to the PSIs. In twelve cases, patient harm was determined to have resulted from PSIs. Systems failures were identified in eleven of the twelve cases in which a PSI resulted in patient harm. CONCLUSION: Systems failures were almost twice as likely as practitioner-based errors to contribute to PSIs, and systems failures were present in the majority of cases resulting in patient harm. To effectively reduce PSIs, ED quality improvement initiatives should focus on systems failure reduction.

[Reliability of peer review-like dialogue in the German statutory quality assurance program]. / Reliabilität des Strukturierten Dialogs in der gesetzlich verpflichtenden Qualitätssicherung.

BACKGROUND: Quality measurement in the German statutory program for quality in health care follows a two-step process. For selected areas of health care, quality is measured via performance indicators (first step). Providers failing to achieve benchmarks in these indicators subsequently enter into a peer review process (second step) and are asked by the respective regional authority to provide a written statement regarding their indicator results. The statements are then evaluated by peers, with the goal to assess the provider's quality of care. In the past, similar peer review-based approaches to the measurement of health care quality in other countries have shown a tendency to lack reliability. So far, the reliability of this component of the German statutory program for quality in health care has not been investigated. METHOD: Using logistic regression models, the influence of the respective regional authority on the peer review component of health care quality measurement in Germany was investigated using three exemplary indicators and data from 2016. RESULTS: Both the probability that providers are asked to provide a statement as well as the results produced by the peer review process significantly depend on the regional authority in charge. This dependence cannot be fully explained by differences in the indicator results or by differences in case volume. CONCLUSIONS: The present results are in accordance with earlier findings, which show low reliability for peer review-based approaches to quality measurement. Thus, different results produced by the peer review component of the quality measurement process may in part be due to differences in the way the review process is conducted. This heterogeneity among the regional authorities limits the reliability of this process. In order to increase reliability, the peer review process should be standardized to a higher degree, with clear review criteria, and the peers should undergo comprehensive training for the review process. Alternatively, the future peer review component could be adapted to focus rather on identification of improvement strategies than on reliable provider comparisons.

Using peers to assess handoffs: a pilot study.

BACKGROUND: Handoffs among post-graduate year 1 (PGY1) trainees occur with high frequency. Peer assessment of handoff competence would add a new perspective on how well the handoff information helped them to provide optimal patient care. OBJECTIVE: The goals of this study were to test the feasibility of the approach of an instrument for peer assessment of handoffs by meeting criteria of being able to use technology to capture evaluations in real time, exhibiting strong psychometric properties, and having high PGY1 satisfaction scores. DESIGN: An iPad® application was built for a seven-item handoff instrument. Over a two-month period, post-call PGY1s completed assessments of three co-PGY1s from whom they received handoffs the prior evening. PARTICIPANTS: Internal Medicine PGY1s at the University of Pennsylvania. MAIN MEASURES: ANOVA was used to explore interperson score differences (validity). Generalizability analyses provided estimates of score precision (reproducibility). PGY1s completed satisfaction surveys about the process. KEY RESULTS: Sixty-two PGY1s (100 %) participated in the study. 59 % of the targeted evaluations were completed. The major limitations were network connectivity and inability to find the post-call trainee. PGY1 scores on the single item of "overall competency" ranged from 4 to 9 with a mean of 7.31 (SD 1.09). Generalizability coefficients approached 0.60 for 10 evaluations per PGY1 for a single rotation and 12 evaluations per PGY1 across multiple rotations. The majority of PGY1s believed that they could adequately assess handoff competence and that the peer assessment process was valuable (70 and 77 %, respectively). CONCLUSION: Psychometric properties of an instrument for peer assessment of handoffs are encouraging. Obtaining 10 or 12 evaluations per PGY1 allowed for reliable assessment of handoff skills. Peer evaluations of handoffs using mobile technology were feasible, and were well received by PGY1s.

Ten-year results of quality assurance in radiotherapy chart round.

BACKGROUND: The Royal Australian and New Zealand College of Radiologists (RANZCR) initiated a unique instrument to audit the quality of patient notes and radiotherapy prescriptions. We present our experience collected over ten years from the use of the RANZCR audit instrument. METHODS: In this study, the results of data collected prospectively from January 1999 to June 2009 through the audit instrument were assessed. Radiotherapy chart rounds were held weekly in the uro-oncology tumour stream and real time feedback was provided. Electronic medical records were retrospectively assessed in September 2009 to see if any omissions were subsequently corrected. RESULTS: In total 2597 patients were audited. One hundred and thirty seven (5%) patients had one hundred and ninety nine omissions in documentation or radiotherapy prescription. In 79% of chart rounds no omissions were found at all, in 12% of chart rounds one omission was found and in 9% of chart rounds two or more omissions were found. Out of 199 omissions, 95% were of record keeping and 2% were omissions in the treatment prescription. Of omissions, 152 (76%) were unfiled investigation results of which 77 (51%) were subsequently corrected. CONCLUSIONS: Real-time audit with feedback is an effective tool in assessing the standards of radiotherapy documentation in our department, and also probably contributed to the high level of attentiveness. A large proportion of omissions were investigation results, which highlights the need for an improved system of retrieval of investigation results in the radiation oncology department.

Optimizing peer review: A year of experience after instituting a real-time comment-enhanced program at a children's hospital.

OBJECTIVE: Peer review has become an essential component of a comprehensive radiology department quality assurance program. Multiple commercial programs, such as RADPEER, are available to fill this need but may be limited by low radiologist compliance and delayed or limited feedback. Consequently, these peer review programs may not achieve the greater goal of improving diagnostic quality. This article presents data from a peer review system implemented in an academic radiology group at a large urban multidisciplinary children's hospital. The peer review system offered instantaneous feedback with an enhanced comment feature for peer radiologists. MATERIALS AND METHODS: Peer review data were collected on 5278 radiologic studies over a 12-month period including 15 radiologists. The data were analyzed for compliance rate, discrepancy rate, and comment usage. RESULTS: The compliance rate for peer review averaged 52% for the 12-month period. The compliance rate trended upward over the course of the year, with a final month's compliance rate of 76%. The discrepancy rate between original interpretation and peer review was 3.6%. Comments were voluntarily included in 7.3% of nondiscrepant peer review scores. CONCLUSION: Our peer review process was enhanced by real-time comment-enriched feedback on both discrepant and nondiscrepant peer reviews. We show improved radiologist compliance over the course of a year in a peer review program with no incentives or penalties for performing reviews. To our knowledge, no compliance rates exist in current literature for comparison.

Progress towards a National Cardiac Procedure Database--development of the Australasian Society of Cardiac and Thoracic Surgeons (ASCTS) and Melbourne Interventional Group (MIG) registries.

Since the call for a National Cardiac Procedures Database in 2001, much work has been accomplished in both cardiac surgery and interventional cardiology in an attempt to establish a unified, systematic approach to data collection, defining a common minimum dataset pertinent to the Australian context, and instituting quality control measures to ensure integrity and privacy of data. In this paper we outline the aims of the Australasian Society of Cardiac and Thoracic Surgeons (ASCTS) and the Melbourne Interventional Group (MIG) registries, and propose a comprehensive set of standardised data elements and their definitions to facilitate transparency in data collection, consistency between these and other data sets, and encourage ongoing peer-review. The aims are to improve outcomes for patients by determining key performance indicators and standards of performance for hospital units, to allow estimation of procedural risks and likelihood of outcomes for patients, and to report outcomes to relevant stake-holders and the public.

Peer reviews: taking on new meanings.

Peer reviews in nursing are historically used to gauge performance within an individual's scope of practice or as a tool to evaluate a sentinel or adverse event. Quality of care measures, clinical pertinence, and evaluating standards of care have begun as parallel strategies to replace the former uses in assuring the right care at the right time in the right setting.

Credentialing, Certification, and Peer Review Essentials for the Neurosurgeon.

Credentialing and certification are essential processes during hiring to ensure that the physician is competent and possesses the qualifications and skill sets claimed. Peer review ensures the continuing evolution of these skills to meet a standard of care. We have provided an overview and discussion of these processes in the United States. Credentialing is the process by which a physician is determined to be competent and able to practice, used to ensure that medical staff meets specific standards, and to grant operative privileges at an institution. Certification is a standardized affirmation of a physician's competence on a nationwide basis. Although not legally required to practice in the United States, many institutions emphasize certification for full privileges on an ongoing basis at a hospital. In the United States, peer review of adverse events is a mandatory prerequisite for accreditation. The initial lack of standardization led to the development of the Health Care Quality Improvement Act, which protects those involved in the peer review process from litigation, and the National Provider Databank, which was established as a national database to track misconduct. A focus on quality improvement in the peer review process can lead to improved performance and patient outcomes. A thorough understanding of the processes of credentialing, certification, and peer review in the United States will benefit neurosurgeons by allowing them to know what institutions are looking for as well and their rights and responsibilities in any given situation. It could also be useful to compare these policies and practices in the United States to those in other countries.

Generalisability in unbalanced, uncrossed and fully nested studies.

OBJECTIVES: There is growing interest in multi-source, multi-level feedback for measuring the performance of health care professionals. However, data are often unbalanced (e.g. there are different numbers of raters for each doctor), uncrossed (e.g. raters rate the doctor on only one occasion) and fully nested (e.g. raters for a doctor are unique to that doctor). Estimating the true score variance among doctors under these circumstances is proving a challenge. METHODS: Extensions to reliability and generalisability (G) formulae are introduced to handle unbalanced, uncrossed and fully nested data to produce coefficients that take into account variances among raters, ratees and questionnaire items at different levels of analysis. Decision (D) formulae are developed to handle predictions of minimum numbers of raters for unbalanced studies. An artificial dataset and two real-world datasets consisting of colleague and patient evaluations of doctors are analysed to demonstrate the feasibility and relevance of the formulae. Another independent dataset is used for validating D predictions of G coefficients for varying numbers of raters against actual G coefficients. A combined G coefficient formula is introduced for estimating multi-sourced reliability. RESULTS: The results from the formulae indicate that it is possible to estimate reliability and generalisability in unbalanced, fully nested and uncrossed studies, and to identify extraneous variance that can be removed to estimate true score variance among doctors. The validation results show that it is possible to predict the minimum numbers of raters even if the study is unbalanced. DISCUSSION: Calculating G and D coefficients for psychometric data based on feedback on doctor performance is possible even when the data are unbalanced, uncrossed and fully nested, provided that: (i) variances are separated at the rater and ratee levels, and (ii) the average number of raters per ratee is used in calculations for deriving these coefficients.

Blueprinting for clinical performance deficiencies--lessons and principles from the General Medical Council's fitness to practise procedures.

BACKGROUND: The UK General Medical Council (GMC) in its regulatory capacity conducts formal tests of competence (TOCs) on doctors whose performance is of concern. TOCs are individually tailored to each doctor's specialty and grade. AIMS: To describe the development and implementation of an electronic blueprinting system that supports the delivery of TOCs. METHOD: A case study that describes the evolution of the GMC electronic blueprint including the derivation of its content and its functionality. RESULTS: A question bank has been created with all items classified according to the competencies defined by Good Medical Practice. This database aids test assembly and ensures that each assessment maps across the breadth of the blueprint. CONCLUSIONS: The blueprint described was easy to construct and is easy to use. It reflects the knowledge, skills and behaviours (learning outcomes) to be assessed. It guides commissioning of test material and enables the systematic and faithful sampling of common and important problems. The principles described have potential for wider application to blueprinting in undergraduate or clinical training programmes. Such a blueprint can provide the essential link between a curriculum and its assessment system and ensure that assessment content is stable over time.

The use of peer review in nursing education and clinical practice.

Peer review is an efficient tool for staff educators to use in promoting professional staff and student development. The information gathered from peer review can be used by staff educators to develop individualized performance improvement plans to support staff development and increase student learning. Peer review provides a form of feedback that can be used to identify staff clinical learning needs with the goal of increasing safety and quality care for patients.

Peer assessment of professionalism: a five-year experience in medical clerkship.

INTRODUCTION: Faculty assessment of students' professionalism is often based upon sporadic exposure to students. Peers are in a unique position to provide valid judgments of these behaviors. AIMS: (1) To learn if peer assessments of professional conduct correlate with traditional performance measures; (2) to determine if peer assessments of professionalism influence the designation of honors, and (3) to explore student and faculty opinions regarding peer assessment. SETTING: Internal Medicine Clerkship at Southern Illinois University. PROGRAM DESCRIPTION: Since 2001 anonymous student peer assessments of professionalism have been used in assigning clerkship grades. PROGRAM EVALUATION: Peer assessments of professionalism had weak, though significant, correlations with faculty ratings (r = 0.29), performance on the NBME subject test (r = 0.28), and performance on a cumulative performance assessment (r = 0.30), and did not change the total number of honors awarded. A majority of students (71%) felt comfortable evaluating their peers, and 77% would keep the peer evaluation procedure in place. A majority of faculty (83%) indicated that peer assessments added valuable information. DISCUSSION: Peer assessments of professional conduct have little correlation with other performance measures, are more likely to have a positive influence on final clerkship grades, and have little impact on awarding honors.