Crowdsourcing Morphology Assessments in Oculoplastic Surgery: Reliability and Validity of Lay People Relative to Professional Image Analysts and Experts.

PURPOSE: To determine if crowdsourced ratings of oculoplastic surgical outcomes provide reliable information compared to professional graders and oculoplastic experts. METHODS: In this prospective psychometric evaluation, a scale for the rating of postoperative eyelid swelling was constructed using randomly selected images and topic experts. This scale was presented adjacent to 205 test images, including 10% duplicates. Graders were instructed to match the test image to the reference image it most closely resembles. Three sets of graders were solicited: crowdsourced lay people from Amazon Mechanical Turk marketplace, professional graders from the Doheny Image Reading Center (DIRC), and American Society of Ophthalmic Plastic and Reconstructive Surgery surgeons. Performance was assessed by classical correlational analysis and generalizability theory. RESULTS: The correlation between scores on the first rating and the second rating for the 19 repeated occurrences was 0.60 for lay observers, 0.80 for DIRC graders and 0.84 for oculoplastic experts. In terms of inter-group rating reliability for all photos, the scores provided by lay observers were correlated with DIRC graders at a level of r = 0.88 and to experts at r = 0.79. The pictures themselves accounted for the greatest amount of variation among all groups. The amount of variation in the scores due to the rater was highest in the lay group at 25%, and was 20% and 21% for DIRC graders and experts, respectively. CONCLUSIONS: Crowdsourced observers are insufficiently precise to replicate the results of experts in grading postoperative eyelid swelling. DIRC graders performed similarly to experts and present a less resource-intensive option.

Screen-Based Simulation for Training and Automated Assessment of Teamwork Skills: Comparing 2 Modes With Different Interactivity.

INTRODUCTION: The need for teamwork training is well documented; however, teaching these skills is challenging given the logistics of assembling individual team members together to train in person. We designed 2 modes of screen-based simulation for training teamwork skills to assess whether interactivity with nonplayer characters was necessary for in-game performance gains or for player satisfaction with the experience. METHODS: Mixed, randomized, repeated measures study with licensed healthcare providers block-stratified and randomized to evaluation-participant observes and evaluates the team player in 3 scenarios-and game play-participant is immersed as the leader in the same 3 scenarios. Teamwork construct scores (leadership, communication, situation monitoring, mutual support) from an ontology-based, Bayesian network assessment model were analyzed using mixed randomized repeated measures analyses of variance to compare performance, across scenarios and modes. Learning was measured by pretest and posttest quiz scores. User experience was evaluated using χ2 analyses. RESULTS: Among 166 recruited and randomized participants, 120 enrolled in the study and 109 had complete data for analysis. Mean composite teamwork Bayesian network scores improved for successive scenarios in both modes, with evaluation scores statistically higher than game play for every teamwork construct and scenario (r = 0.73, P = 0.000). Quiz scores improved from pretest to posttest (P = 0.004), but differences between modes were not significant. CONCLUSIONS: For training teamwork skills using screen-based simulation, interactivity of the player with the nonplayer characters is not necessary for in-game performance gains or for player satisfaction with the experience.

The teacher's role in promoting collaborative dialogue in the classroom.

BACKGROUND: Research on student-led small-group learning in schools going back nearly four decades has documented many types of student participation that promote learning. Less is known about how the teacher can foster effective groupwork behaviours. AIMS: This paper reviews research that explores the role of the teacher in promoting learning in small groups. The focus is on how students can learn from their peers during small-group work, how teachers can prepare students for collaborative group work, and the role of teacher discourse and classroom norms in small-group dialogue. METHOD: Studies selected for review focused on student-led small-group contexts for learning in which students were expected to collaborate, reported data from systematic observations of group work, and linked observational data to teacher practices and student learning outcomes. RESULTS AND CONCLUSIONS: This review uncovered multiple dimensions of the teacher's role in fostering beneficial group dialogue, including preparing students for collaborative work, forming groups, structuring the group-work task, and influencing student interaction through teachers' discourse with small groups and with the class. Common threads through the research are the importance of students explaining their thinking, and teacher strategies and practices that may promote student elaboration of ideas.