Risk Adjustment of the Modified Low Back Pain Disability Questionnaire and Neck Disability Index to Benchmark Physical Therapist Performance: Analysis From an Outcomes Registry.

BACKGROUND: Patient-reported outcomes (PROs) have been touted as the ultimate assessment of quality medical care and have been proposed as performance measures after appropriate risk adjustment. Although spine conditions represent the most common orthopedic disorders, the most used PROs for disabilities related to the back and neck-the Modified Low Back Pain Disability Questionnaire (MDQ) and the Neck Disability Index (NDI)-have not been evaluated as performance measures. OBJECTIVE: The objective of this study was to benchmark physical therapists' performance in the management of spine conditions not involving surgery through the use of risk-adjusted MDQ and NDI outcomes. DESIGN: This was a retrospective observational study. METHODS: Data were accessed for patients seeking physical therapy with no history of related surgery for back or neck pain (315,274 treatment episodes) between January 2015 and June 2018. Patients with complete data, including initial and matched final MDQ or NDI, were considered for analysis (182,276 patients; 2799 physical therapists). Linear models controlling for baseline PRO and patient characteristics predicted PRO change for each patient. An aggregated performance ratio of actual PRO change to predicted PRO change was calculated for each physical therapist, and then empirical bootstrapping was used to develop the median performance ratio and its confidence intervals. Physical therapists who met a 40-patient threshold for either cohort (MDQ or NDI) were classified as "outperforming," "meeting expectations," or "underperforming" relative to predicted values using these 95% confidence intervals. RESULTS: Performance ratios indicated that 10% and 11% of physical therapists outperformed, 79% and 78% met expectations, and 11% and 11% underperformed relative to the risk-adjusted predicted change in the MDQ (1240 therapists; 97,908 patients) and NDI (461 therapists; 26,123 patients), respectively. To demonstrate the clinical importance of risk adjustment, clinical performance was evaluated in the seemingly homogeneous subset of 208 physical therapists within 0.5 SD of the median baseline MDQ and the median actual change in the MDQ. Following risk adjustment, 2 physical therapists were classified in each of the outperforming and underperforming cohorts. LIMITATIONS: The secondarily obtained observational data used were not collected for research purposes. Additionally, the analyses were limited by missing baseline information and follow-up PROs. CONCLUSIONS: The risk-adjusted performance ratios for the MDQ and NDI resulted in disparate conclusions regarding the quality of care compared with the raw, unadjusted change scores. According to the baseline and unadjusted change in the MDQ, even physical therapists in the most homogeneous sample were differentiated following appropriate risk adjustment. Clinically important improvements in actual PROs were observed in the outperforming but not in the underperforming physical therapists. Clinically meaningful differences in the performance ratio are unknown and are a limitation to clinical application and an opportunity for future research.

A Practical Guide to Performance Improvement: Data Collection and Analysis.

This article discusses performance improvement (PI) data collection processes and the various tools that can be used to analyze and display data through the duration of a PI project. It describes the importance of data, how to determine the required amount of data, how to collect and analyze data, and in what format data should be presented. Personnel involved with PI projects may need to use various data collection methods and tools to ensure an effective project with a successful outcome. This article includes examples of how PI project team members can implement various data collection and analysis tools. After reviewing this article, the reader should have a better understanding of this part of the PI process.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room.

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Development of RAD-Score: A Tool to Assess the Procedural Competence of Diagnostic Radiology Residents.

OBJECTIVE: The purpose of this study is to develop a tool to assess the procedural competence of radiology trainees, with sources of evidence gathered from five categories to support the construct validity of tool: content, response process, internal structure, relations to other variables, and consequences. SUBJECTS AND METHODS: A pilot form for assessing procedural competence among radiology residents, known as the RAD-Score tool, was developed by evaluating published literature and using a modified Delphi procedure involving a group of local content experts. The pilot version of the tool was tested by seven radiology department faculty members who evaluated procedures performed by 25 residents at one institution between October 2014 and June 2015. Residents were evaluated while performing multiple procedures in both clinical and simulation settings. The main outcome measure was the percentage of residents who were considered ready to perform procedures independently, with testing conducted to determine differences between levels of training. RESULTS: A total of 105 forms (for 52 procedures performed in a clinical setting and 53 procedures performed in a simulation setting) were collected for a variety of procedures (eight vascular or interventional, 42 body, 12 musculoskeletal, 23 chest, and 20 breast procedures). A statistically significant difference was noted in the percentage of trainees who were rated as being ready to perform a procedure independently (in postgraduate year [PGY] 2, 12% of residents; in PGY3, 61%; in PGY4, 85%; and in PGY5, 88%; p < 0.05); this difference persisted in the clinical and simulation settings. User feedback and psychometric analysis were used to create a final version of the form. CONCLUSION: This prospective study describes the successful development of a tool for assessing the procedural competence of radiology trainees with high levels of construct validity in multiple domains. Implementation of the tool in the radiology residency curriculum is planned and can play an instrumental role in the transition to competency-based radiology training.

Measuring performance in the interpretation of chest radiographs: a pilot study.

AIM: To develop a system to assess the image interpretation performance of radiologists in identifying signs of malignancy on chest radiographs. MATERIALS AND METHODS: A test set of 30 chest radiographs was chosen by an experienced radiologist consisting of 11 normal and 19 abnormal cases. The malignant cases all had biopsy-proven pathology; the normal and benign cases all had at least 2 years of imaging follow-up. Fourteen radiologists with a range of experiences were recruited. Participants individually read the test set displayed on a standard reporting workstation, with their findings entered directly into a laptop running specially designed reporting software. For each case, relevant clinical information was given and the reader was asked to mark any perceived abnormality and rate their level of suspicion on a five-point scale (normal, benign, indeterminate, suspicious, or malignant). On completion, participants were given instant feedback with performance parameters including sensitivity and specificity automatically calculated. An opportunity was then given to review the cases together with an expert opinion and pathology. The time each participant took to complete the test was recorded. RESULTS: Six consultant radiologists who took part showed significantly better performance as determined by receiver operating characteristic (ROC) analysis compared to eight specialist registrars (area under the ROC curve [AUC]=0.9297 and 0.7648 respectively, p=0.003). There was a significant correlation with years of experience in the interpretation of chest radiographs and performance on the test set (r=0.573, p=0.032). Consultant radiologists completed the test significantly more quickly that the specialist registrars: mean time 19.65 minutes compared to 26.51 minutes (p=0.033). CONCLUSION: It is possible to use a test set to measure individual differences in the interpretation of chest radiographs. This has the potential to be a useful tool in performance testing.

The Impact of Structured Incentives on the Adoption of a Serious Game for Hand Hygiene Training in a Hospital Setting.

Hand hygiene is recognized by the CDC as the most effective method of preventing Hospital Acquired Infections (HAIs) which cost the US healthcare system $14 Billion. However, training and promotion of hand hygiene in healthcare settings is an on-going challenge. This paper describes a hand hygiene improvement campaign in Edinburgh Royal Infirmary (Scotland, UK) using the SureWash gesture recognition system (SureWash, IRL). The campaign consisted of two phases of three-months each; the first phase involved technology evaluation and familiarization in a variety of settings within the hospital. The second phase involved rotation between two units with specific changes to the incentives for completing the training. There were 2,010 individual training sessions with over 30% outside of office hours. Individuals completed an average of 2.72 training sessions each and 90% of staff passed the assessment. Senior staff noted a change in hand hygiene culture following the campaign and the good-natured competition between staff to demonstrate hand hygiene competence using the SureWash serious game. While the new technology did facilitate the culture change its successful implementation was dependent on a set of incentives for staff and a structured implementation plan.

Using computerised interactive response technology to assess electrocardiographers and for aggregating diagnoses.

The 12-lead electrocardiogram (ECG) is a crucial diagnostic tool. However, the ideal method to assess competency in ECG interpretation remains unclear. We sought to evaluate whether keypad response technology provides a rapid, interactive way to assess ECG knowledge. 75 participants were enrolled [32 (43%) Primary Care Physicians, 24 (32%) Hospital Medical Staff and 19 (25%) Nurse Practitioners]. Nineteen ECGs with 4 possible answers were interpreted. Out of 1425 possible decisions 1054 (73.9%) responses were made. Only 570/1425 (40%) of the responses were correct. Diagnostic accuracy varied (0% to 78%, mean 42%±21%) across the entire cohort. Participation was high, (median 83%, IQR 50%-100%). Hospital Medical Staff had significantly higher diagnostic accuracy than nurse practitioners (50±20% vs. 38±19%, p=0.04) and Primary Care Physicians (50±20% vs. 40±21%, p=0.07) although not significant. Interactive voting systems can be rapidly and successfully used to assess ECG interpretation. Further education is necessary to improve diagnostic accuracy.

Objective classification of performance in the use of a piercing saw in jewellery making.

Data from 15 jewellery students, in their 1st and 3rd years of training, were analysed to show how data collected from work settings can be used to objectively evaluate performance in the use of tools. Participants were asked to use a piercing saw to cut 5 lines in a piece of metal. Performance was categorised in terms of functional dynamics. Data from strain gauges and a tri-axial accelerometer (built into the handle of the saw) were recorded and thirteen metrics derived from these data. The key question for this paper is which metrics could be used to distinguish levels of ability. Principal Components Analysis identified five components: sawing action; grasp of handle; task completion time; lateral deviation of strokes; and quality of lines cut. Using representative metrics for these components, participants could be ranked in terms of performance (low, medium, high) and statistical analysis showed significant differences between participants on key metrics.