An Application of Kane's Validity Framework to Evaluate Formative and Summative Assessment Instruments for Telesimulations in Clinical Lactation.

INTRODUCTION: Health professional learners have limited exposure to breastfeeding patients from diverse backgrounds in clinical rotations. Instead, simulation-based training is used for lactation skills training. There are no validated or standardized simulations and assessment rubrics for lactation. In this pilot, breastfeeding telesimulations with standardized patients (SPs) wearing a high-fidelity breast model matching their skin tone were developed. The validity of Formative and Summative Assessment Rubrics (FAR, SAR) were assessed following Kane's validity framework. The objective was to provide initial evidence for the validity of the FAR and SAR as constructs of competence in lactation support at the entry to practice or practice level. METHODS: Three breastfeeding case scenarios, FAR, and SAR were developed and evaluated with clinical lactation specialists (evaluators, n = 17) and SPs. The FAR was used in practice telesimulations where SPs' (n = 14) performance and telesimulation feasibility were assessed. The FAR was updated in preparation for a pilot study where medical students (n = 13) completed the 3 telesimulations. In the pilot, the updated FAR was used by SPs (n = 6) to assess medical students' performance of clinical skills. After the pilot, rubrics were updated after focus groups with SPs and discussions with evaluators. Evaluators (n = 3) graded students' posttelesimulation documentations using the SAR. Cronbach ɑ level and the intraclass correlation coefficient were assessed iteratively to collect evidence for the scoring, generalizability, and extrapolation of the FAR and SAR according to Kane's framework. RESULTS: The FAR and SAR were found to have acceptable internal consistency and moderate to high interrater reliability (intraclass correlation coefficient, 0.55-0.94), which provided evidence for scoring and generalizability of the instruments. Evaluators agreed that SPs' performances were realistic (5.6/6), and SPs' feedback was organized (5.5/6) and helpful (5.6/6), which provided evidence for extrapolation. CONCLUSIONS: Initial evidence for validity of scoring, generalization, and extrapolation FAR and SAR (according to Kane's framework) in assessing health professional learner's performance of clinical lactation skills has been presented. These results from a pilot study suggest that the FAR and SAR are reliable instruments for assessing learners' clinical performance in a breastfeeding-focused telesimulation where the SP wears a high-fidelity breast model matching their skin tone. Additional studies will be required to collect evidence according to all 4 categories of Kane's framework for the validity of the FAR and SAR.

The Efficacy of Hybrid Telesimulation with Standardized Patients in Teaching Medical Students Clinical Lactation Skills: A Pilot Study.

Introduction: Medical students lack competencies in clinical lactation. We determined the effect of hybrid telesimulation with a standardized patient (SP) on medical students' clinical performance in lactation support. We assessed students' engagement and satisfaction with the experience. Materials and Methods: Undergraduate medical students (n = 13) completed (1) preparatory case scenarios with multiple-choice questions and (2) three telesimulations with SPs wearing a high-fidelity breast model. Students had the option to complete the Encounter Documentation. SPs used the Formative Assessment Rubric (FAR) to evaluate students' interpersonal skills and clinical lactation experts used the Summative Assessment Rubric to evaluate documentation skills. Investigators collected satisfaction data from a focus group and written evaluation. Dunn's multiple comparison and Freidman tests were used to measure differences in FAR scores between cases and telesimulations. Qualitative data were analyzed using thematic analysis. Results: Most students (70%) attempted case questions multiple times and scores improved (p < 0.0001) between attempts. FAR scores suggest students were prepared for telesimulations (5.5/6-pt Likert) and interpersonal skills were appropriate (5.4/6), with no differences by case (p = 0.11). FAR scores increased between telesimulation 1-2 (+24.5/114, p = 0.002) and 2-3 (+17.5/114, p = 0.014). Students were satisfied with the experience and would recommend it to classmates (both 4.6/6). Thematic analysis revealed feedback regarding interpersonal skills was helpful. Conclusions: Medical students must develop skills to support breastfeeding in virtual settings. Telesimulation can be incorporated into existing curricula to support clinical lactation competencies.