Implementing evidence-based assertions of clinical actionability in the context of secondary findings: Updates from the ClinGen Actionability Working Group.

PURPOSE: The ClinGen Actionability Working Group (AWG) developed an evidence-based framework to generate actionability reports and scores of gene-condition pairs in the context of secondary findings from genome sequencing. Here we describe the expansion of the framework to include actionability assertions. METHODS: Initial development of the actionability rubric was based on previously scored adult gene-condition pairs and individual expert evaluation. Rubric refinement was iterative and based on evaluation, feedback, and discussion. The final rubric was pragmatically evaluated via integration into actionability assessments for 27 gene-condition pairs. RESULTS: The resulting rubric has a 4-point scale (limited, moderate, strong, and definitive) and uses the highest-scoring outcome-intervention pair of each gene-condition pair to generate a preliminary assertion. During AWG discussions, predefined criteria and factors guide discussion to produce a consensus assertion for a gene-condition pair, which may differ from the preliminary assertion. The AWG has retrospectively generated assertions for all previously scored gene-condition pairs and are prospectively asserting on gene-condition pairs under assessment, having completed over 170 adult and 188 pediatric gene-condition pairs. CONCLUSION: The AWG expanded its framework to provide actionability assertions to enhance the clinical value of their resources and increase their utility as decision aids regarding return of secondary findings.

An Assessment Rubric for a Resident Training Program in Surgery: A Single-Institution Experience.

Using a Collaborative Action Research model, our research team established a one-month clinical resident training program for first- and second-year clinical residents. We created and implemented an assessment rubric to assess the residents' progress toward independent practice in surgery, and thereby, to evaluate the program itself. The program included training in three areas: basic techniques and procedures in the operating room, surgical ward management, and academic activities. The rubric measured the residents' performance according to three achievement levels: Level 1 (demonstration), Level 2 (active help) and Level 3 (passive help). The program and rubric implementation began in June 2019 and continued until March 2020, when the program outcomes and shortcomings were analyzed. Among nineteen clinical residents, a total of nine clinical residents participated in the study. Most participants reached achievement Level 3 for their performance of basic techniques in the operating room. Finally, we discussed ideas for improvement and drafted plans for an improved rubric to complete the action research cycle. Our research team found the rubric to be a useful tool in evaluating the status of the new clinical resident training program.

Development and psychometric testing of a clinical reasoning rubric based on the nursing process.

BACKGROUND: To facilitate the development of clinical reasoning skills in nursing students, educators must possess the ability to teach and evaluate them. This study aimed to describe the development and validation process of an analytic rubric of clinical reasoning skills based on the nursing process in undergraduate nursing students. METHODS: A seven-step method was used for rubric development. The initial validation process of the rubric of clinical reasoning was performed with the participation of key stakeholders to assess its face and content validity as well as applicability in the classroom and bedside. An initial pilot test was performed based on scenario-based examinations in the nursing process training course so that convergent validity was used to show how closely the new scale is related to the previous measure for evaluating students' tasks. Internal consistency and inter-rater correlation coefficient measurement for reliability were assessed. RESULTS: The rubric to assess clinical reasoning skills was developed into eight categories according to the five stages of the nursing process. Content and face validity of the rubric were done qualitatively and resulted in a clear, simple rubric relevant to clinical reasoning skills assessment. The convergent validity was confirmed by the conventional method. The reliability was approved by a high inter-rater correlation coefficient based on the assessment by two random independent raters. CONCLUSION: The clinical reasoning meta-rubric developed in this study meets the purpose of the study. This analytical rubric can be applied to guide teaching and learning as well as evaluate clinical reasoning based on the findings. Testing the applicability confirmed its validity and reliability for assessing clinical reasoning skills in nursing process education during the undergraduate nursing program.

Measuring deliberate reflection in residents: validation and psychometric properties of a measurement tool.

PURPOSE: Reflective capacity is "the ability to understand critical analysis of knowledge and experience to achieve deeper meaning." In medicine, there is little provision for post-graduate medical education to teach deliberate reflection. The feasibility, scoring characteristics, reliability, validation, and adaptability of a modified previously validated instrument was examined for its usefulness assessing reflective capacity in residents as a step toward developing interventions for improvement. METHODS: Third-year residents and fellows from four anesthesia training programs were administered a slightly modified version of the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) in a prospective, observational study at the end of the 2019 academic year. Six written vignettes of imperfect anesthesia situations were created. Subjects recorded their perspectives on two randomly assigned vignettes. Responses were scored using a 5-element rubric; average scores were analyzed for psychometric properties. An independent self-report assessment method, the Cognitive Behavior Survey: Residency Level (rCBS) was used to examine construct validity. Internal consistency (ICR, Cronbach's alpha) and interrater reliability (weighted kappa) were examined. Pearson correlations were used between the two measures of reflective capacity. RESULTS: 46/136 invited subjects completed 2/6 randomly assigned vignettes. Interrater agreement was high (k = 0.85). The overall average REFLECT score was 1.8 (1-4 scale) with good distribution across the range of scores. ICR for both the REFLECT score (mean 1.8, sd 0.5; α = 0.92) and the reflection scale of the rCBS (mean 4.5, sd 1.1; α = 0.94) were excellent. There was a significant correlation between REFLECT score and the rCBS reflection scale (r = .44, p < 0.01). CONCLUSIONS: This study demonstrates feasibility, reliability, and sufficiently robust psychometric properties of a modified REFLECT rubric to assess graduate medical trainees' reflective capacity and established construct/convergent validity to an independent measure. The instrument has the potential to assess the effectiveness of interventions intended to improve reflective capacity.

Applying a validated scoring rubric to pre-clerkship medical students' standardized patient notes: a pilot study.

BACKGROUND: With the elimination in 2021 of the United States Medical Licensing Examination Step 2 Clinical Skills test, it is incumbent upon U.S. medical schools to develop local validated assessments of clinical reasoning. While much attention has been paid to summative exams for graduating students, formative exams for pre-clerkship students have not been well studied. METHODS: We applied the University of Illinois at Chicago College of Medicine (UIC-COM) Patient Note (PN) Scoring Rubric to templated PNs written by 103 pre-clerkship students for two cases in an objective structured clinical examination (OSCE) at the Yale School of Medicine. The rubric consists of four section scores (Documentation, Differential Diagnosis, Justification, and Workup, each scored 1 to 4) and a composite score (scaled 23 to 100). We calculated item discrimination for each section score and Cronbach's alpha for each case. We surveyed students about their experience writing the templated PN. RESULTS: Mean Documentation, Differential Diagnosis, Justification, Workup, and composite scores for case A were 2.16, 1.80, 1.65, 2.29, and 47.67, respectively. For case B, the scores were 2.13, 1.21, 1.60, 1.67, and 40.54, respectively. Item discrimination ranged from 0.41 to 0.80. Cronbach's alpha for cases A and B was 0.48 and 0.25, respectively. A majority of the students felt that the exercise was useful and appropriate to their level of training. CONCLUSIONS: Despite performing poorly, pre-clerkship students found the note-writing task beneficial. Reliability of the scoring rubric was suboptimal, and modifications are needed to make this exercise a suitable measure of clinical reasoning.

Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students.

BACKGROUND: Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determine the agreement among four evaluators and compare the overall scores awarded when assessing portfolios of endodontic preclinical treatments performed by dental students by using an analytic rubric and a numeric rating scale. METHODS: A random sample of 42 portfolios performed by fourth-year dental students at preclinical endodontic practices were blindly assessed by four evaluators using two different evaluation methods: an analytic rubric specifically designed and a numeric rating scale. Six categories were analyzed: radiographic assessment, access preparation, shaping procedure, obturation, content of the portfolio, and presentation of the portfolio. The maximum global score was 10 points. The overall scores obtained with both methods from each evaluator were compared by Student's t, while agreement among evaluators was measured by Intraclass correlation coefficients (ICC). The influence of the difficulty of the endodontic treatment on the evaluators´ scores was analyzed by one-way ANOVA. Statistical tests were performed at a pre-set alpha of 0.05 using Stata 16. RESULTS: Difficulty of canal treatment did not influence the scores of evaluators, irrespective of the evaluation method used. When the analytic rubric was used, inter-evaluator agreement was substantial for radiographic assessment, access preparation, shaping procedure, obturation, and overall scores. Inter-evaluator agreement ranged from moderate to fair with the numeric rating scale. Mean higher overall scores were achieved when numeric rating scale was used. Presentation and content of the portfolio showed slight and fair agreement, respectively, among evaluators, regardless the evaluation method applied. CONCLUSIONS: Assessment guided by an analytic rubric allowed evaluators to reach higher levels of agreement than those obtained when using a numeric rating scale. However, the rubric negatively affected overall scores.

The implementation of Tanner's Clinical Judgment Model and the Indonesian version of the Lasater Clinical Judgment Rubric in the clinical setting.

A critical component for nurses to provide safe and quality care with strong patient outcomes is clinical judgment. This study aimed to identify the difference in the clinical judgment score in newly graduated nurses by implementing Tanner's Clinical Judgment Model. This study followed an experimental design with a pretest-posttest and involved thirty-two newly graduated nurses from three private hospitals in Indonesia. The intervention group had a scheduled post-conference with their preceptors using Tanner's Clinical Judgment Model's guidelines with high-level, open-ended questions for two consecutive weeks. Independent samples t-test was used to compare pre-and post-test scores utilizing the valid and reliable Indonesia version of the Lasater Clinical Judgment Rubric. There was no significant difference between the pre-test results of the intervention and control groups, with a significance value of 0.647. The results indicated a significant difference in the mean clinical judgment scores between the intervention and control groups after two weeks of intervention, where the intervention group's scores were higher than the control group's with a significance value of <0.001 with a mean difference of 6.75 and CI of 95 % (4.18-9.31). Tanner's Clinical Judgment Model showed a significant impact in increasing clinical judgment scores. The Indonesia version of the Lasater Clinical Judgment Rubric is a valuable tool for assessing and evaluating the development of clinical judgment and provides detailed targeted areas needing improvement. Preceptors and nursing leaders in the clinical setting should continue to use this model and tools to assist newly graduated nurses in developing clinical judgment skills during their transition period.

Facilitator competency rubric in nursing simulations: transcultural adaptation and validation of the German version.

BACKGROUND: Simulations are part of nursing education. To obtain good results, simulation facilitators need to be competent in simulation pedagogy. Part of this study was the transcultural adaptation and validation of the Facilitator Competency Rubric into German (FCRG) and the evaluation of the factors associated with higher competencies. METHOD: A written-standardized cross-sectional survey was conducted. N = 100 facilitators (mean age: 41.0 (9.8), female: 75.3%) participated. Test-re-test, confirmatory factor analysis (CFA), and ANOVAs were conducted to evaluate the reliability and validity of, and the factors associated with, FCRG. Intraclass correlation coefficient (ICC) values > .9 indicate excellent reliability. RESULTS: The FCRG achieved good intra-rater reliability (all ICC > .934). A moderate correlation (Spearman-rho .335, p < .001) with motivation indicates convergent validity. The CFA showed sufficient to good model fits (CFI = .983 and SRMR = .016). Basic simulation pedagogy training is associated with higher competencies (p = .036, b = 17.766). CONCLUSION: The FCRG is a suitable self-assessment tool for evaluating a facilitator's competence in nursing simulation.

The impact of tutoring on nursing students' clinical judgment: A quasi-experimental study.

BACKGROUND: Nurses' lack of clinical judgment often leads to adverse patient outcomes due to failure to recognize clinical deterioration, intervene, and manage complications. Teaching clinical judgment through a nursing process can help nursing students provide safe and competent patient care with improved health outcomes and to pass the National Council Licensure Examination for Registered Nurses (NCLEX-RN). AIMS: The aim of this study was to examine the effect of tutoring on clinical judgment of undergraduate nursing students utilizing Lasater's Clinical Judgment Rubric (LCJR). This study also compared the clinical judgment of male and female nursing students and students from different semester levels. METHODS: This quasi-experimental study utilized a single group pretest, posttest design. A convenience sample of n = 40 undergraduate nursing students from the Los Angeles County College of Nursing and Allied Health participated in the study. The participants underwent a pretest simulation, four sessions of the Clinical Judgment Model (CJM)-based tutoring, and a posttest simulation. RESULTS: The posttest clinical judgment scores (35.70 ± 3.6) were significantly different from the pretest scores (25.78 ± 5.20). The tutoring had a significant effect on the clinical judgment of nursing students t(39) = -11.64, n = 40, p < .001, at 95% CI of the mean difference. LINKING EVIDENCE TO ACTION: Enhancing nursing students' clinical judgment is crucial to provide high-quality, safe patient care with improved health outcomes. The CJM-based tutoring is an effective strategy for developing clinical judgment in nursing students. This new teaching approach can train students to critically think, develop clinical judgment, and prepare for the complex healthcare environment. Therefore, nurse educators should focus on integrating clinical judgment into the prelicensure nursing program curriculum as a priority.

The SPEER: An interprofessional team behavior rubric to optimize geriatric clinical care.

Geriatric patients with complex health care needs can benefit from interprofessional (IP) care; however, a major gap in health professional education is determining how to prepare future providers for IP collaboration. Effective IP team behavior assessment tools are needed to teach, implement, and evaluate IP practice skills. After review of IP evaluation tools, the Standardized Patient Encounter Evaluation Rubric (SPEER) was created to evaluate team dynamics in IP practice sites.Independent sample t-tests between faculty and learner SPEER scores showed learners scored themselves 15 points higher than their faculty scores (p < .001). Cronbach's α showed high internal consistency (α = 0.91). Paired t-tests found that learners identified improvements in the team's ability to address the patient's education needs and to allow the patients to voice their expectations. Faculty identified improvements in the teams' ability to make recommendations. Faculty evaluations of learner teams showed improvements in raw ratings on all but two items. Qualitative data analysis for emergent themes showed learners desired team functioning feedback and how teamwork could improve to provide optimal IP care.In conclusion, the SPEER can help faculty and learners identify growth in their teams' ability to perform key IP skills in clinical sites.

Inconceivable: assessment of teaching at the university level.

Although most colleges and universities use a scholarly, multifaceted, and data-rich methodology to evaluate a faculty member's research, by comparison they are quite lacking in the way they evaluate teaching. We share advice from education experts for guidance at designing an ideal process "to evaluate teaching productivity as seriously as we do research" as well as an outline of a teaching rubric used to evaluate university faculty members.

Reflective narratives on unfolding ethics case vignettes: integration into the physiology course of first-year medical students.

Reflective narrative writing on unfolding ethics case vignettes has the potential to promote values like empathy among medical students, which will prove useful in their future. The objective of this study was to explore the scope of guided reflective narratives on unfolding ethics case vignettes integrated into physiology teaching to promote empathy among first-year medical students. First-year medical students (n = 150) were presented with unfolding case vignettes exploring ethical issues, as part of their physiology curriculum, with questions and prompts for a guided reflective narrative. Feedback about the program was obtained through a semistructured questionnaire. The contents of the students' reflective narratives and feedback were analyzed. The narratives revealed that students empathized with the various individuals involved in medical treatment and research, which was also reflected in student feedback (n = 116; 79%) and high Likert scale scores (74%). The program was considered effective by students in broadening their perspective and promoting critical thinking (n = 129; 88%). Through analysis of reflections, a rubric that would enable the evaluation of students' empathy on a scale of increasing ability to look at a situation from another's point of view emerged. This rubric could be used to quantitatively assess the impact of similar programs. Although divided in their individual opinions, the study population was able to relate to all individuals involved in medical practice and research, recognizing the need for fair and ethical practices. The content analysis led to the development of a rubric to grade empathy.NEW & NOTEWORTHY Unfolding case vignettes exploring ethical issues were introduced into the physiology curriculum of first-year medical students. The effectiveness of guided reflections following the case vignettes was explored. The analysis of the reflections and feedback on the program revealed that students became more empathetic. The reflections broadened their perspective and encouraged critical thinking. The analysis led to development of a rubric to grade the level of empathy, which can be used for quantitative assessment of empathy.

Students' perceptions, engagement and satisfaction with the use of an e-rubric for the assessment of manual skills in physiotherapy.

INTRODUCTION: In recent years, formative assessment has gained importance in health care education to facilitate and enhance learning throughout the training period. Within the frame of active methodologies, rubrics have become an essential instrument for formative assessment. Most rubric-based assessment procedures focus on measuring the effects of rubrics on teachers. However, few studies focus their attention on the perception that students have of the evaluation process through rubrics. METHODS: A cross-sectional survey study was carried out with 134 students enrolled in the pre-graduate Physiotherapy education. Assessment of manual skills during a practical examination was performed using an e-rubric tool. Peer-assessment, self-assessment and teacher´s assessment were registered. After completion of the examination process, students' perceptions, satisfaction and engagement were collected. RESULTS: Quantitative results related to students' opinion about e-rubric based assessment, students' engagement, perceived benefits and drawbacks of the e-rubric as well as the overall assessment of the learning experience were obtained. 86.6% of the students agreed upon the fact that "the rubric allowed one to know what it is expected from examination" and 83.6% of the students agreed upon the fact that "the rubric allowed one to verify the level of competence acquired". A high rate of agreement (87.3%) was also reached among students concerning feedback. CONCLUSIONS: E-rubrics seemed to have the potential to promote learning by making criteria and expectations explicit, facilitating feedback, self-assessment and peer-assessment. The importance of students in their own learning process required their participation in the assessment task, a fact that was globally appreciated by the students. Learning experience was considered interesting, motivating, it promoted participation, cooperative work and peer-assessment. The use of e-rubrics increased engagement levels when attention was focused on their guidance and reflection role.

Radiography education with VR using head mounted display: proficiency evaluation by rubric method.

BACKGROUND: The use of head mounted display (HMD)-based immersive virtual reality (VR) coaching systems (HMD-VRC) is expected to be effective for skill acquisition in radiography. The usefulness of HMD-VRC has been reported in many previous studies. However, previous studies have evaluated the effectiveness of HMD-VRC only through questionnaires. HMD-VRC has difficulties in palpation and patient interaction compared to real-world training. It is expected that these issues will have an impact on proficiency. The purpose of this study is to determine the impact of VR constraints in HMD-VRC, especially palpation and patient interaction, on radiographic skills proficiency in a real-world setting. METHODS: First-year students (n = 30) at a training school for radiology technologists in Japan were randomly divided into two groups, one using HMD-VRC (HMD-VRC group) and the other practicing with conventional physical equipment (RP group) and trained for approximately one hour. The teachers then evaluated the students for proficiency using a rubric method. RESULTS: In this study, it was found that some skills in the HMD-VRC group were equivalent to those of the RP group and some were significantly lower than those of the RP group. There was a significant decrease in proficiency in skills related to palpation and patient interaction. CONCLUSIONS: This study suggests that HMD-VRC can be less effective than real-world training in radiographic techniques, which require palpation and patient interaction. For effective training, it is important to objectively evaluate proficiency in the real world, even for HMD-VRC with new technologies, such as haptic presentation and VR patient interaction. TRIAL REGISTRATION: The study was conducted with the approval of the Ethics Committee of International University of Health and Welfare (Approval No.21-Im-035, Registration date: September 28, 2021).

Development of a performance rubric for transanal endoscopic rectal purse-string sutures.

BACKGROUND: Placing a transanal endoscopic rectal purse-string suture (taEPS) is the crucial first component of transanal total mesorectal excision (taTME). However, no structured training is available to improve the procedure-specific skills for taEPS. The aim of this study was to create a performance rubric to improve taEPS skills and provide preliminary evidence for its validity. METHODS: A performance rubric was created based on technical considerations for taEPS, identified by consulting with taTME surgical and performance assessment experts. Ten independent, blinded raters assessed 10 videotaped taEPS procedures of consecutive taTME cases, at National Cancer Center Hospital East (NCCHE), Chiba, Japan, in January 2018-March 2019 using the rubric and the Global Operative Assessment of Laparoscopic Skills (GOALS). Internal consistency and inter-rater reliabilities were calculated. Videotaped taEPS procedures were timed and assessed by the rubric. Correlation between rubric scores and suturing times were analyzed. RESULTS: The rubric consists of four items: loading the needle (LN), atraumatic needle passage (AP), planned suture path (PS), and overall performance (OA). Videotaped performances were graded on a 3-point Likert scale; scores were calculated as sums of the points. Cronbach's α for internal consistency was 0.713. Inter-rater reliabilities were LN: 0.73, AP: 0.76, PS: 0.71, and OA: 0.70. Rubric and GOALS scores were strongly correlated (r = 0.964, p < 0.001). In 112 consecutive taEPS performances, rubric scores were strongly correlated with suturing time (r = - 0.69, p < 0.001). Surgeons' experience with taTME was associated with rubric scores and suturing time. CONCLUSIONS: This study provides preliminary validation for the taEPS skill performance rubric. The rubric's structured training may facilitate skill acquisition by providing trainees with critical clinical considerations.

Focus on nursing point-of-care tools: application of a new evaluation rubric.

Objective: Point-of-care tools (PoCTs) provide evidence-based information on patient care and procedures at the time of need. Registered nurses have unique practice needs, and many PoCTs are marketed to support their practice. However, there is little reported evidence in the literature about evaluating nursing-focused PoCTs. Case Presentation: The investigators developed a rubric containing evaluation criteria based on content, coverage of nursing topics, transparency of the evidence, user perception, and customization of PoCTs for supporting nursing practice. The investigators selected five PoCTs cited in the literature and of interest to local nursing leadership: ClinicalKey for Nursing, DynaMed, Lippincott's Advisor and Procedures, Nursing Reference Center Plus, and UpToDate. Application of the rubric found Lippincott had the highest coverage of diagnoses, while ClinicalKey for Nursing had strong content focused on interventions and outcomes. Nursing Reference Center Plus provided the most well-rounded coverage of nursing terminology and topics. DynaMed and UpToDate were more transparent with indicating conflict of interest, but both had lower coverage of nursing terminology, content, and care processes. Conclusion: None of the five PoCTs strongly met all of the evaluated criteria. The rubric developed for this study highlights each PoCT's strengths and weaknesses that can then be used to inform the decision-making process based on priorities and budget. The investigators recommend licensing a nursing PoCT and a PoCT like DynaMed or UpToDate to provide comprehensive, evidence-based, patient care coverage and to meet the diverse information needs of nurses.

Psychometric evaluation of the Ethical Caring Competency Scale in nursing.

BACKGROUND: An ethical competence list for nurses could guide educators and managers in the field of health care to both support the development of ethical conduct and improve the assessment of ethical competence in health care. AIM: This study aimed to verify the reliability and validity of the Ethical Caring Competency Scale (ECCS) and to obtain suggestions for its use as an evaluation form in rubric format among a sample of Japanese nurses. RESEARCH DESIGN: This research employed a descriptive and cross-sectional design. PARTICIPANTS AND RESEARCH CONTEXT: A self-administered questionnaire was distributed to 1157 nurses working in two hospitals in Japan. The contents of the survey were demographic data, a draft of the ECCS consisting of 22 competencies from four core competencies, questions regarding experience in learning about medical/nursing ethics, and the Work Motivation Measurement Scale for Nurses. Three levels of difficulty for the 22 items were established using relative comparisons of the mean scores within the four core competencies. Three groups, namely, an expert group, a middle group, and a beginner group, were categorized according to the quartiles of the total ECCS score. ETHICAL CONSIDERATIONS: This study was approved by the Clinical Research Ethics Committee of Hamamatsu University School of Medicine (Decision no. 18-267). The ethical principles of voluntary participation, anonymity, and confidentiality were considered. FINDINGS: A total of 962 valid responses were analyzed. The ECCS scores for the three levels of difficulty were significantly different from each other. Stability was confirmed by the test-retest of the total ECCS scores (r = .900, p < .0001). The total ECCS scores for the three groups showed significant differences in all pairs. The Cronbach's α coefficient ranged from .72 to .89 for each core competency, and internal consistency was confirmed. CONCLUSION: The reliability and validity of the ECCS as a scale were statistically verified, and we were able to obtain suggestions for its application as a form of evaluation in rubric format.

Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables.

In this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts' assessment. Specifically, we improved and validated its scores' performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity criteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.

Comparison between students' self-evaluation and faculty members' evaluation in a clinical endodontic course at King Saud University.

INTRODUCTION: The objective of this study was to compare faculty member evaluations with student self-evaluations in a clinical endodontic course in the dental school at King Saud University and to evaluate the reliability of the students' self-assessment scores after using a rubric with well-defined criteria. MATERIALS AND METHODS: Evaluated and self-evaluated endodontic cases that were clinically treated by the fourth-year undergraduate dental students at the College of Dentistry, Girls University Campus, at King Saud University over 2 years (2017-2018) were included. Cases included anterior teeth, premolars and molars. The evaluation form was divided into six sections with well-defined criteria to cover all aspects of nonsurgical root canal treatment with a maximum grade of 10 points can be scored for each student per case. The students evaluated themselves for each section and then were evaluated by two faculty members. Student and faculty assessment agreement and the reliability of the students' self-assessment scores were measured. A p ≤ .05 was considered significant. RESULTS: A total of 363 cases were included: 26.7% anterior teeth, 38.84% premolars and 34.43% molars. The students evaluated themselves with higher grades compared to the evaluators' grading in all steps and in the overall grading in all teeth types. The students' self-assessment scores showed good and moderate reliability in all steps and in the overall grading. CONCLUSION: The students tend to overrate their performance, and their assessments have moderate to good reliability, which reflects the reliability of the rubric used as an accurate measurement tool that helps the evaluator and the student objectively assess their performance.

Development of a Formative Assessment Rubric for Peer Evaluation of Teaching (FARPET) and Pilot Use in Veterinary Online Teaching.

Peer evaluation of teaching (PET) serves an important role as a component of faculty development in the medical education field. With the emergence of COVID-19, the authors recognized the need for a flexible tool that could be used for a variety of lecture formats, including virtual instruction, and that could provide a framework for consistent and meaningful PET feedback. This teaching tip describes the creation and pilot use of a PET rubric, which includes six fixed core items (lesson structure, content organization, audiovisual facilitation, concept development, enthusiasm, and relevance) and items to be assessed separately for asynchronous lectures (cognitive engagement-asynchronous) and synchronous lectures (cognitive engagement-synchronous, discourse quality, collaborative learning, and check for understanding). The instrument packet comprises the rubric, instructions for use, definitions, and examples of each item, plus three training videos for users to compare with authors' consensus training scores; these serve as frame-of-reference training. The instrument was piloted among veterinary educators, and feedback was sought in a focus group setting. The instrument was well received, and training and use required a minimum time commitment. Inter-rater reliability within 1 Likert scale point (adjacent agreement) was assessed for each of the training videos, and consistency of scoring was demonstrated between focus group members using percent agreement (0.82, 0.85, 0.88) and between focus members and the authors' consensus training scores (all videos: 0.91). This instrument may serve as a helpful resource for institutions looking for a framework for PET. We intend to continually adjust the instrument in response to feedback from wider use.