RESUMO
BACKGROUND: Institutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive. AIM: To determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns. SETTING AND PARTICIPANTS: Single institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018. PROGRAM DESCRIPTION: The performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary. PROGRAM EVALUATION: The Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37. DISCUSSION: NLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.
RESUMO
Programmatic assessment supports the evolution from assessment of learning to fostering assessment for learning and as learning practices. A well-designed programmatic assessment system aligns educational objectives, learning opportunities, and assessments with the goals of supporting student learning, making decisions about student competence and promotion decisions, and supporting curriculum evaluation. We present evidence-based guidance for implementing assessment for and as learning practices in the pre-clinical knowledge assessment system to help students learn, synthesize, master and retain content for the long-term so that they can apply knowledge to patient care. Practical tips are in the domains of culture and motivation of assessment, including how an honour code and competency-based grading system can support an assessment system to develop student self-regulated learning and professional identity, curricular assessment structure, such as how and when to utilize low-stakes and cumulative assessment to drive learning, exam and question structure, including what authentic question and exam types can best facilitate learning, and assessment follow-up and review considerations, such exam retake processes to support learning, and academic success structures. A culture change is likely necessary for administrators, faculty members, and students to embrace assessment as most importantly a learning tool for students and programs.
Assuntos
Currículo , Aprendizagem , Competência Clínica , Avaliação Educacional , Docentes , Humanos , EstudantesRESUMO
Building upon the disruption to lecture-based methods triggered by the introduction of problem-based learning, approaches to promote collaborative learning are becoming increasingly diverse, widespread and generally well accepted within medical education. Examples of relatively new, structured collaborative learning methods include team-based learning and just-in-time teaching. Examples of less structured approaches include think-pair share, case discussions, and the flipped classroom. It is now common practice in medical education to employ a range of instructional approaches to support collaborative learning. We believe that the adoption of such approaches is entering a new and challenging era. We define collaborate learning by drawing on the broader literature, including Chi's ICAP framework that emphasizes the importance of sustained, interactive explanation and elaboration by learners. We distinguish collaborate learning from constructive, active, and passive learning and provide preliminary evidence documenting the growth of methods that support collaborative learning. We argue that the rate of adoption of collaborative learning methods will accelerate due to a growing emphasis on the development of team competencies and the increasing availability of digital media. At the same time, the adoption collaborative learning strategies face persistent challenges, stemming from an overdependence on comparative-effectiveness research and a lack of useful guidelines about how best to adapt collaborative learning methods to given learning contexts. The medical education community has struggled to consistently demonstrate superior outcomes when using collaborative learning methods and strategies. Despite this, support for their use will continue to expand. To select approaches with the greatest utility, instructors must carefully align conditions of the learning context with the learning approaches under consideration. Further, it is critical that modifications are made with caution and that instructors verify that modifications do not impede the desired cognitive activities needed to support meaningful collaborative learning.
Assuntos
Educação Médica/tendências , Aprendizagem Baseada em Problemas/tendências , Ensino/métodos , Competência Clínica , Comportamento Cooperativo , Currículo/tendências , Educação Médica/normas , Processos Grupais , Humanos , Aprendizagem Baseada em Problemas/normas , Ensino/normasRESUMO
Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.