Use of Machine Learning to Assess Cataract Surgery Skill Level With Tool Detection.

Purpose: To develop a method for objective analysis of the reproducible steps in routine cataract surgery. Design: Prospective study; machine learning. Participants: Deidentified faculty and trainee surgical videos. Methods: Consecutive cataract surgeries performed by a faculty or trainee surgeon in an ophthalmology residency program over 6 months were collected and labeled according to degrees of difficulty. An existing image classification network, ResNet 152, was fine-tuned for tool detection in cataract surgery to allow for automatic identification of each unique surgical instrument. Individual microscope video frame windows were subsequently encoded as a vector. The relation between vector encodings and perceived skill using k-fold user-out cross-validation was examined. Algorithms were evaluated using area under the receiver operating characteristic curve (AUC) and the classification accuracy. Main Outcome Measures: Accuracy of tool detection and skill assessment. Results: In total, 391 consecutive cataract procedures with 209 routine cases were used. Our model achieved an AUC ranging from 0.933 to 0.998 for tool detection. For skill classification, AUC was 0.550 (95% confidence interval [CI], 0.547-0.553) with an accuracy of 54.3% (95% CI, 53.9%-54.7%) for a single snippet, AUC was 0.570 (0.565-0.575) with an accuracy of 57.8% (56.8%-58.7%) for a single surgery, and AUC was 0.692 (0.659-0.758) with an accuracy of 63.3% (56.8%-69.8%) for a single user given all their trials. Conclusions: Our research shows that machine learning can accurately and independently identify distinct cataract surgery tools in videos, which is crucial for comparing the use of the tool in a step. However, it is more challenging for machine learning to accurately differentiate overall and specific step skill to assess the level of training or expertise. Financial Disclosures: The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Health Resource and Cost Savings Achieved in a Multidisciplinary Lung Cancer Clinic.

Background: Lung cancer (LC) care is resource and cost intensive. We launched a Multidisciplinary LC Clinic (MDC), where patients with a new LC diagnosis received concurrent oncology consultation, resulting in improved time to LC assessment and treatment. Here, we evaluate the impact of MDC on health resource utilization, patient and caregiver costs, and secondary patient benefits. Methods: We retrospectively analyzed patients in a rapid assessment clinic with a new LC diagnosis pre-MDC (September 2016-February 2017) and post-MDC implementation (February 2017-December 2018). Data are reported as means; unpaired t-tests and ANOVA were used to assess for significance. We also conducted a cost analysis. Resource utilization, out-of-pocket costs, procedure-related costs, and indirect costs were evaluated from the societal perspective and presented in 2019 Canadian dollars (CAD); multi-way worst/best case and threshold sensitivity analyses were conducted. Results: We reviewed 428 patients (78 traditional model, 350 MDC). Patients in the MDC model required significantly fewer oncology visits from LC diagnosis to first LC treatment (1.62 vs. 2.68, p < 0.001), which was significant for patients with stage 1, 3, and 4 disease. Compared with the traditional model, there was no change in mean biopsies/patient (1.32 traditional vs. 1.17 MDC, p = 0.18) or staging investigations/patient (2.24 traditional vs. 2.02 MDC, p = 0.20). Post-MDC, there was an increase in invasive mediastinal staging for patients with stage 2/3 LC (15.0% vs. 60.0%, p < 0.001). Over 22 months, MDC resulted in savings of CAD 48,389 including CAD 24,167 CAD in direct patient out-of-pocket expenses. For the threshold analyses, MDC was estimated to cost CAD 25,708 per quality-adjusted life year (QALY), considered to be below current willingness to pay thresholds (at CAD 80,000 per QALY). MDC also facilitated oncology assessment for 29 non-LC patients. Conclusions: An MDC led to a reduction in patient visits and direct patient and caregiver costs.

Evaluation of ophthalmology residents' self-assessments and peer assessments in simulated surgery.

OBJECTIVE: To evaluate the accuracy of ophthalmology residents' self-assessment and peer assessment of surgical skills in a simulation setting. DESIGN: Simulation laboratory assessment. PARTICIPANTS: Ophthalmology residents novice to cataract surgery. METHODS: A modified International Council of Ophthalmology's Ophthalmology Surgical Competency Assessment Rubric: Phacoemulsification structured assessment tool for simulated cataract surgery was established by conventional Delphi method. Residents completed 10 independent simulated surgeries that were video-recorded. Two experts graded the videos using the assessment tool. Participants performed self-assessment of their own 10 videos, and peer assessment of 10 of their peers' videos. RESULTS: Nine cataract surgery experts provided feedback and modifications for the assessment tool. Agreement for the first round of the Delphi method ranged from 55.56% to 100%. Second round agreement was 80% or greater for all answers. The final assessment tool comprised (i) 4 procedural items scored from 0 (not performed) to 7 (competent), and (ii) a global rating scale (GRS) requiring yes/no answers to 4 performance-related questions. Eight residents participated in the study. There was excellent expert inter-rater reliability intraclass correlation ((ICC) = 0.844, 0.875, 0.809, 0.844) and fair to excellent inter-rater reliability between expert and peer scores (ICC = 0.702, 0.831, 0.521, 0.423), but systematic disagreement (ICC = -0.428, -0.038) or poor inter-rater reliability (ICC = 0.298, 0.362) between expert and self-scores. There was poor agreement for all GRS questions (κ statistic < 0.40) except 2 comparisons. CONCLUSIONS: In the simulation setting, experts were able to reliably assess trainees' performance using the assessment tool. Participants demonstrated inconsistency in assessing their own skills; however, they were adequate at assessing their peers' overall performance.

Continuing Professional Development for Primary Care Providers in Palliative and End-of-Life Care: A Systematic Review.

Background and Objective: This review updates and expands on previous reviews of educational interventions for primary care providers (PCPs) involved in palliative and end-of-life care (PEoLC) and is the first to include early studies related to medical assistance in dying (MAiD). Methods: A comprehensive search strategy was conducted across five electronic databases to locate published interventional studies related to ongoing PEoLC and/or MAiD education for primary care professionals. A descriptive summary of results and a narrative discussion of common themes and comparisons are provided. Results: Thirty-seven studies met the inclusion criteria. The researchers found a myriad of interventions, including courses based, practical experience, mentoring, and workshops. The researchers categorized results by four domains: attitude, confidence, knowledge, and skills. Across domains, seven educational topics emerged: general care, interprofessional collaboration, nutrition, pain and symptom management, patient communication, and professional coping. Overall, studies employed various methodologies, but often relied on cross-sectionally measured self-assessment. Two articles were found that measured the impact of MAiD education. Conclusion: These findings suggest that PEoLC education can improve PCPs' perceived attitudes, confidence, knowledge, and skills across multiple areas of palliative care practice. While PCPs across studies valued educational interventions, the findings relating to the impact of PEoLC education on PCP's provision of effective PEoLC were unclear. However, most interventions resulted in enhanced confidence and knowledge. To date, there are only two studies that have examined MAiD educational programs. There is a need for studies of higher rigor with more emphasis on follow-up to clarify the impact training has on those involved in PEoLC and MAiD.

Validation of a novel strabismus surgery 3D-printed silicone eye model for simulation training.

PURPOSE: To demonstrate the validity of a new 3D-printed silicone model for practicing strabismus surgery, compared with the rabbit head, in terms of simulator fidelity. METHODS: In this multicenter study, a validated questionnaire was developed to assess fidelity of the model and rabbit head. Participants were asked to rate overall globe, conjunctiva, muscle, and scleral fidelity using a 5-point scale. The survey instrument was disseminated at three strabismus instruction courses: at two meetings, participants practiced on the model and rabbit head prior to completing the questionnaire; at the third, instructors demonstrated advanced surgical skills using only the model and then completed the questionnaire. Repeated measures analysis of variance compared ratings. Pearson's or Spearman's correlation evaluated correlation between years of experience to participants' responses. Qualitative data were coded into themes. RESULTS: A total of 47 participants completed the questionnaire. The model rated 18% higher than rabbit head for anatomical accuracy (mean difference, 0.667; P = 0.001) and 25% higher for position of eyes within the head (mean difference, 0.867; P = 0.006). More experienced participants were more likely to strongly agree that the silicone conjunctiva effectively mimics real conjunctiva (ρ = 0.337; P = 0.036) and that scleral tissue effectively mimics real sclera (ρ = 0.298, P = 0.042). Qualitative data supported the model. CONCLUSIONS: This study demonstrated the validity of the surgical model in terms of fidelity compared to the rabbit head.

Exploring How the New Entrustable Professional Activity Assessment Tools Affect the Quality of Feedback Given to Medical Oncology Residents.

The post-graduate medical programs at Queen's University transitioned to a competency-based medical education framework on July 1, 2017. In advance of this transition, the Medical Oncology program participated in a pilot of six Entrustable Professional Activities (EPAs) focused workplace-based assessment (WBA) tools with faculty and residents. The purpose of this sequential explanatory mixed method study was to determine the extent to which these WBAs provided quality feedback for residents. The WBAs were introduced into daily clinical practice and, once completed, were collected by the research team. A resident focus group (n = 4) and faculty interviews (n = 5) were also conducted. Focus group and interview data were analyzed using an emergent thematic analysis. Data from the completed assessment tools were analyzed using both descriptive statistics and a literature-informed framework developed to assess the quality of feedback. Six main findings emerged: Verbal feedback is preferred over written; providing both written and verbal feedback is important; effective feedback was seen as timely, specific, and actionable; the process was conceptualized as coaching rather than high stakes; there were logistical concerns about the WBAs, and additional clarification about the WBA tools is needed. This study provides insight into faculty and resident perceptions of quality feedback and the potential for WBA tools to assist in providing effective feedback to residents as we shift to competency-based medical education in Canada. Our results suggest the need for additional faculty development around the use of the tools, and their intended role, and the elements of quality feedback.

Aligning Requirements of Training and Assessment in Radiation Treatment Planning in the Era of Competency-Based Medical Education.

PURPOSE: Radiation treatment planning (RTP) is a unique skill that requires interdisciplinary collaboration among radiation oncologists (ROs), dosimetrists, and medical physicists (MP) to train and assess residents. With the adoption of competency-based medical education (CBME) in Canada, it is essential residency program curricula focuses on developing competencies in RTP to facilitate entrustment. Our study investigates how radiation oncology team members' perspectives on RTP education align with requirements of the CBME approach, and its implications for improving residency training. METHODS AND MATERIALS: This qualitative research study took place in the Department of Oncology at a midsize academic institution. Through convenience sampling, focus groups were conducted with radiation oncologists (n = 11), dosimetrists (n = 7), medical physicists (n = 7), and residents (n = 7). Thematic design was adopted to analyze the transcripts through open coding resulting in 3 overarching themes. RESULTS: The results identified existing strengths and weaknesses of the residency program and future opportunities to redesign the curriculum and assessment process within a CBME model. Three overarching themes emerged from the analysis: (1) the strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for change. Stakeholders were optimistic CBME will help enrich resident learning with the increased frequency and quality of competency-based assessments. Participants suggested building a library of cases and developing computer-based learning resources to provide a safe environment to develop skills in contouring, dosimetry, and plan evaluation, in accordance with CBME training. CONCLUSIONS: This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies, including case libraries, computer-based learning, and quality assessments, were highlighted in designing an innovative RTP planning curriculum.

Involving ophthalmology departmental stakeholders in developing workplace-based assessment tools.

OBJECTIVE: As competency-based medical education (CBME) rolls out across Canada, the assessment process is changing. Our purpose was to involve stakeholders in the selection and modification of workplace-based assessment (WBA) tools for use in an ophthalmology residency program. DESIGN: This is a qualitative case study conducted in one ophthalmology department at a midsized teaching hospital in eastern Ontario. PARTICIPANTS: Ophthalmology faculty and residents within the Emergency Eye Clinic participated in the study. METHODS: Phase 1 consisted of faculty piloting the tools and providing written feedback. Phase 2 consisted of 2 focus groups, 1 for residents (n = 9) and 1 for faculty (n = 6), to explore their experiences with the 4 piloted adapted WBA tools. RESULTS: Residents and faculty discussed ongoing issues with buy-in and formalizing feedback through the new assessment process. Residents also reiterated the need for more constructive feedback delivered in a timely and sensitive manner. Generally, residents did not value numerical scales on the tools and preferred written comments. Both residents and faculty valued oral over written feedback given the interactive nature of oral feedback. CONCLUSIONS: This study provides insight into faculty and resident perspectives about WBA tools within one ophthalmology department. These results informed the development of WBA tools within the department and highlighted the importance of shifting the assessment culture to accommodate programmatic approaches to assessment in CBME. Involving key stakeholders in the change process has been a valuable strategy. Future research should examine whether or not these perspectives change after CBME implementation.

Knowledge retention after focused cardiac ultrasound training: a prospective cohort pilot study / Retenção do conhecimento após treinamento de ultrassonografia cardíaca focada: estudo-piloto prospectivo de coorte

Abstract Background and objectives: Focused Cardiac Ultrasound (FoCUS) has proven instrumental in guiding anesthesiologists' clinical decision-making process. Training residents to perform and interpret FoCUS is both feasible and effective. However, the degree of knowledge retention after FoCUS training remains a subject of debate. We sought to provide a description of our 4-week FoCUS curriculum, and to assess the knowledge retention among anesthesia residents at 6 months after FoCUS rotation. Methods: A prospective analysis involving eleven senior anesthesia residents was carried out. At end of FoCUS Rotation (EOR) participants completed a questionnaire (evaluating the number of scans completed and residents' self-rated knowledge and comfort level with FoCUS), and a multiple-choice FoCUS exam comprised of written- and video-based questions. Six months later, participants completed a follow-up questionnaire and a similar exam. Self-rated knowledge and exam scores were compared at EOR and after 6 months. Spearman correlations were conducted to test the relationship between number of scans completed and exam scores, perceived knowledge and exam scores, and number of scans and perceived knowledge. Results: Mean exam scores (out of 50) were 44.1 at EOR and 43 at the 6-month follow-up. Residents had significantly higher perceived knowledge (out of 10) at EOR (8.0) than at the 6-month follow-up (5.5), p = 0.003. At the EOR, all trainees felt comfortable using FoCUS, and at 6 months 10/11 still felt comfortable. All the trainees had used FoCUS in their clinical practice after EOR, and the most cited reason for not using FoCUS more frequently was the lack of perceived clinical need. A strong and statistically significant (rho = 0.804, p = 0.005) correlation between number of scans completed during the FoCUS rotation and 6-month follow-up perceived knowledge was observed. Conclusion: Four weeks of intensive FoCUS training results in adequate knowledge acquisition and 6-month knowledge retention.

Resumo Justificativa e objetivos: A ultrassonografia cardíaca no local de atendimento (USCLA) provou ser importante para orientar o processo de tomada de decisão clínica dos anestesiologistas. Treinar os residentes para fazer e interpretar uma USCLA é viável e eficaz. No entanto, o grau de retenção do conhecimento após o treinamento permanece um assunto de debate. Procuramos fornecer uma descrição do currículo de quatro semanas do treinamento de USCLA e avaliar a retenção do conhecimento entre os residentes de anestesia seis meses após a rotação em USCLA. Métodos: Uma análise prospectiva foi realizada com 11 residentes seniores de anestesia. Ao final da rotação em USCLA, os participantes preencheram um questionário (avaliando o número de exames ultrassonográficos concluídos, o conhecimento adquirido e o nível de conforto dos residentes com a USCLA) e fizeram um exame de múltipla escolha para USCLA, composto por perguntas escritas e baseadas em vídeo. Seis meses depois, os participantes preencheram um questionário de acompanhamento e um exame similar. A autoavaliação do conhecimento e os escores do exame foram comparados no final da rotação e após seis meses. Correlações de Spearman foram usadas para testar a relação entre o número de exames concluídos e os escores dos exames, o conhecimento percebido, os escores dos exames, o número de exames e o conhecimento percebido. Resultados: Os escores médios dos exames (50) foram: 44,1 no final da rotação e 43 após seis meses. Os residentes tiveram conhecimento percebido significativamente maior (10) no final da rotação (8,0) que após seis meses (5,5), p = 0,003. No final da rotação, todos os residentes se sentiram confortáveis usando o aparelho de USCLA e, aos seis meses, 10/11 ainda se sentiam confortáveis. Todos os residentes haviam usado o USCLA em sua prática clínica após o final da rotação e a razão mais citada para não usar o USCLA com mais frequência foi a falta de necessidade clínica percebida. Uma correlação forte e estatisticamente significativa (rho = 0,804, p = 0,005) foi observada entre o número de exames realizados durante a rotação em USCLA e o conhecimento percebido em seis meses de seguimento. Conclusão: Quatro semanas de treinamento intensivo de USCLA resultaram em aquisição e retenção adequadas do conhecimento por seis meses.