Analytics of self-regulated learning scaffolding: effects on learning processes.

Self-regulated learning (SRL) is the ability to regulate cognitive, metacognitive, motivational, and emotional states while learning and is posited to be a strong predictor of academic success. It is therefore important to provide learners with effective instructions to promote more meaningful and effective SRL processes. One way to implement SRL instructions is through providing real-time SRL scaffolding while learners engage with a task. However, previous studies have tended to focus on fixed scaffolding rather than adaptive scaffolding that is tailored to student actions. Studies that have investigated adaptive scaffolding have not adequately distinguished between the effects of adaptive and fixed scaffolding compared to a control condition. Moreover, previous studies have tended to investigate the effects of scaffolding at the task level rather than shorter time segments-obscuring the impact of individual scaffolds on SRL processes. To address these gaps, we (a) collected trace data about student activities while working on a multi-source writing task and (b) analyzed these data using a cutting-edge learning analytic technique- ordered network analysis (ONA)-to model, visualize, and explain how learners' SRL processes changed in relation to the scaffolds. At the task level, our results suggest that learners who received adaptive scaffolding have significantly different patterns of SRL processes compared to the fixed scaffolding and control conditions. While not significantly different, our results at the task segment level suggest that adaptive scaffolding is associated with earlier engagement in SRL processes. At both the task level and task segment level, those who received adaptive scaffolding, compared to the other conditions, exhibited more task-guided learning processes such as referring to task instructions and rubrics in relation to their reading and writing. This study not only deepens our understanding of the effects of scaffolding at different levels of analysis but also demonstrates the use of a contemporary learning analytic technique for evaluating the effects of different kinds of scaffolding on learners' SRL processes.

A feasibility study for a unified, multimodal analysis of online information foraging in health-related topics.

Background: Digital health literacy (DHL) is the ability to find, understand, and appraise online health-related information, as well as apply it to health behavior. It has become a core competence for navigating online information and health service environments. DHL involves solving ill-structured problems, where the problem and its solution are not clearcut and may have no single answer, such as in the process of sensemaking. We employ and expand on information foraging theory to address how experts and novices in information retrieval perform a search task. Our overarching aim is to pinpoint best practices and pitfalls in understanding and appraising health-related information online to develop a digital intervention to increase DHL and critical thinking. Methods: In this feasibility study, we recruited a total of twenty participants for our expert and novice subsamples. We collected sociodemographic data with a self-developed survey, video data through an observation protocol of a 10-minute search task, as well as audio-video data via a retrospective think-aloud. The three, multimodal data streams were transcribed and aligned. Codes were developed inductively in several iterations, then applied deductively to the entire dataset. Tabularized, coded and segmented qualitative data were used to create various quantitative models, which demonstrate viability for the qualitative and statistical comparison of our two subsamples. Results: Data were visualized with Epistemic Network Analysis to analyze code co-occurrences in the three aligned data streams, and with Qualitative/Unified Exploration of State Transitions to examine the order in which participants in our two subsamples encountered online content. Conclusions: This paper describes our methods and planned analyses elaborated with mock figures. Quantifying qualitative data, aligning data streams, and representing all information in a tabularized dataset allows us to group data according to various participant attributes and employ data visualization techniques to pinpoint patterns therein.

Diagnostic Activities and Diagnostic Practices in Medical Education and Teacher Education: An Interdisciplinary Comparison.

In this article, we investigate diagnostic activities and diagnostic practices in medical education and teacher education. Previous studies have tended to focus on comparing knowledge between disciplines, but such an approach is complicated due to the content specificity of knowledge. We compared 142 learners from medical education and 122 learners from teacher education who were asked to (a) diagnose eight simulated cases from their respective discipline in a simulation-based learning environment and (b) write a justificatory report for each simulated case. We coded all justificatory reports regarding four diagnostic activities: generating hypotheses, generating evidence, evaluating evidence, and drawing conclusions. Moreover, using the method of Epistemic Network Analysis, we operationalized diagnostic practices as the relative frequencies of co-occurring diagnostic activities. We found significant differences between learners from medical education and teacher education with respect to both their diagnostic activities and diagnostic practices. Learners from medical education put relatively more emphasis on generating hypotheses and drawing conclusions, therefore applying a more hypothesis-driven approach. By contrast, learners in teacher education had a stronger focus on generating and evaluating evidence, indicating a more data-driven approach. The results may be explained by different epistemic ideals and standards taught in higher education. Further research on the issue of epistemic ideals and standards in diagnosing is needed. Moreover, we recommend that educators think beyond individuals' knowledge and implement measures to systematically teach and increase the awareness of disciplinary standards.

Evaluating how residents talk and what it means for surgical performance in the simulation lab.

BACKGROUND: This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate skills and knowledge through discourse. METHODS: Senior residents (N = 11) were recorded while performing a simulated laparoscopic ventral hernia (LVH) repair. Audio transcripts were coded for five discourse elements related to knowledge, skills, and operative independence. Epistemic network analysis was used to model the ordered integration of the five discourse elements. RESULTS: Participants with poorer hernia repair outcomes had stronger connections between the discourse elements operative planning and asking for information or advice (Operative planning), while participants with better hernia repair outcomes had stronger connections between the discourse elements giving assistant instructions and identifying errors (Operative management): (p = .006; Cohen's d = 2.79). CONCLUSION: Participants with better hernia repair outcomes engaged in more operative management communication during the simulated procedure. This ability to integrate multiple operative steps and verbally communicate them significantly correlated with better operative outcomes.

Teaching practicing surgeons what not to do: An analysis of instruction fluidity during a simulation-based continuing medical education course.

BACKGROUND: Interest is growing in simulation-based continuing medical education courses for practicing surgeons. However, little research has explored the instruction employed during these courses. This study examines instruction practices used during an annual simulation-based continuing medical education course. METHODS: Audio-video data were collected from surgeon instructors (n = 12) who taught a simulated laparoscopic hernia repair continuing medical education course across 2 years. Surgeon learners (n = 58) were grouped by their self-reported laparoscopic and hernia repair experience. Instructors' transcribed dialogue was automatically coded for 5 types of responses to the following questions: anecdotes, confirming, correcting, guidance, and what not to do. Differences in these responses were measured against the progress of the simulations and across learners with different experience levels. Postcourse interviews with instructors were conducted for additional qualitative validation. RESULTS: Performing t tests of instructor responses revealed that they were significantly more likely to answer in forms coded as anecdotes when responding to relative experts and in forms coded as what not to do when responding to novices. Linear regressions of each code against normalized progressions of each simulation revealed a significant relationship between progression through a simulation and frequency of the what not to do code for less-experienced learners. Postcourse interviews revealed that instructors continuously assess participants throughout a session and modify their teaching strategies. CONCLUSION: Instructors significantly modified the focus of their teaching as a function both of their learners' self-reported experience levels, their assessment of learner needs, and learner progression through the training sessions.

Quantifying the Qualitative with Epistemic Network Analysis: A Human Factors Case Study of Task-Allocation Communication in a Primary Care Team.

Health care is fundamentally about people, and therefore, engineering approaches for studying healthcare systems must consider the perspective, concepts and methods offered by the human factors and ergonomics (HFE) discipline. HFE analysis is often qualitative to provide in-depth description of work systems and processes. To deepen our understanding of care processes, we propose the next level of analysis, i.e. quantification of qualitative data. Here, we describe epistemic network analysis (ENA) as a novel method to quantify qualitative data and present a case study applying ENA to assess communication in a primary care team. One high-performing primary care team consisting of a physician, nurse, medical assistant and unit clerk was observed for 15 hours. We analyzed task-allocation communications and identified the sender, receiver, synchronicity and acceptance. We used logistic regression and ENA to evaluate sender, receiver and synchronicity impact on task acceptance. The physician and unit clerk were most successful allocating tasks. Future work should consider the role of synchronous, interruptive communication as potentially useful in time-critical tasks and further investigate the role of the unit clerk. HFE researchers should consider ENA as a tool to expand and deepen their understanding of care processes by quantifying qualitative data.

Using epistemic network analysis to identify targets for educational interventions in trauma team communication.

BACKGROUND: Epistemic Network Analysis (ENA) is a technique for modeling and comparing the structure of connections between elements in coded data. We hypothesized that connections among team discourse elements as modeled by ENA would predict the quality of team performance in trauma simulation. METHODS: The Modified Non-technical Skills Scale for Trauma (T-NOTECHS) was used to score a simulation-based trauma team resuscitation. Sixteen teams of 5 trainees participated. Dialogue was coded using Verbal Response Modes (VRM), a speech classification system. ENA was used to model the connections between VRM codes. ENA models of teams with lesser T-NOTECHS scores (n = 9, mean = 16.98, standard deviation [SD] = 1.45) were compared with models of teams with greater T-NOTECHS scores (n = 7, mean = 21.02, SD = 1.09). RESULTS: Teams had different patterns of connections among VRM speech form codes with regard to connections among questions and edifications (meanHIGH = 0.115, meanLOW = -0.089; t = 2.21; P = .046, Cohen d = 1.021). Greater-scoring groups had stronger connections between stating information and providing acknowledgments, confirmation, or advising. Lesser-scoring groups had a stronger connection between asking questions and stating information. Discourse data suggest that this pattern reflected increased uncertainty. Lesser-scoring groups also had stronger connections from edifications to disclosures (revealing thoughts, feelings, and intentions) and interpretations (explaining, judging, and evaluating the behavior of others). CONCLUSION: ENA is a novel and valid method to assess communication among trauma teams. Differences in communication among higher- and lower-performing teams appear to result from the ways teams use questions. ENA allowed us to identify targets for improvement related to the use of questions and stating information by team members.

Why saying what you mean matters: An analysis of trauma team communication.

BACKGROUND: We hypothesized that team communication with unmatched grammatical form and communicative intent (mixed mode communication) would correlate with worse trauma teamwork. METHODS: Interdisciplinary trauma simulations were conducted. Team performance was rated using the TEAM tool. Team communication was coded for grammatical form and communicative intent. The rate of mixed mode communication (MMC) was calculated. MMC rates were compared to overall TEAM scores. Statements with advisement intent (attempts to guide behavior) and edification intent (objective information) were specifically examined. The rates of MMC with advisement intent (aMMC) and edification intent (eMMC) were also compared to TEAM scores. RESULTS: TEAM scores did not correlate with MMC or eMMC. However, aMMC rates negatively correlated with total TEAM scores (r = -0.556, p = 0.025) and with the TEAM task management component scores (r = -0.513, p = 0.042). CONCLUSIONS: Trauma teams with lower rates of mixed mode communication with advisement intent had better non-technical skills as measured by TEAM.

The hands and head of a surgeon: Modeling operative competency with multimodal epistemic network analysis.

BACKGROUND: This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors. METHODS: Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills. RESULTS: There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents. CONCLUSION: These results suggest that error checklists and surgeons' speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.

Intakes conference: understanding the impact of resident autonomy on a morning report conference.

BACKGROUND: Morning report (MR) is one of the most common conferences in medical residency training. Literature has suggested that an imposed structure is necessary for success. However, studies have suggested that rigid structure may decrease resident motivation. Three goals of a successful MR include large breadth of presented cases, high resident participation, and well-formatted presentations. PURPOSE: In this article, we present a qualitative case analysis of an unstructured resident- led MR and analyze breadth of cases, resident participation, and presentation format. METHOD: We collected data that included case presentation tracking, video recording conferences, and interviewing residents and faculty. RESULTS: Case presentations covered a broad range of topics with little duplication (2.2%). Residents preformed most (94%) of presentations; most residents (92.3%) presented cases. Case presentations were structured in 1 of 3 formats that fulfilled suggested guidelines from MR literature. CONCLUSION: In this unstructured MR conference, structural aims were met with the relatively autonomous decisions of motivated resident participants.

What is digital medicine?

Changes in health care are a fundamental part of social and intellectual evolution. The modern practice of scientific medicine depends on the existence of the written and printed word to store medical information. Because computers can transform information as well as store it, new digital tools cannot only record clinical data, they can also generate medical knowledge. In doing so, they make it possible to develop "digital medicine" that is potentially more precise, more effective, more experimental, more widely distributed, and more egalitarian than current medical practice. Critical steps in the creation of digital medicine are careful analysis of the impact of new technologies and coordinated efforts to direct technological development towards creating a new paradigm of medical care.