Formative behavioral assessment across eight constructs: Dependability of direct behavior ratings and formative behavior rating measures.

Due to their promise as a feasible tool for evaluating the effects of school-based interventions, Direct Behavior Ratings (DBR) have received much research attention over the past 2 decades. Although DBR methodology has demonstrated much promise, favorable psychometric characteristics only have been demonstrated for tools measuring a small number of constructs. Likewise, although a variety of methods of DBR have been proposed, most extant studies have focused on the use of single-item methods. The present study examined the dependability of four methods of formative behavioral assessment (i.e., single-item and multi-item ratings administered either daily [DBR] or weekly [formative behavior rating measures or FBRM]) across eight psychological constructs (i.e., interpersonal skills, academic engagement, organizational skills, disruptive behavior, oppositional behavior, interpersonal conflict, anxious depressed, and social withdrawal). School-based professionals (N = 91; i.e., teachers, paraprofessionals, and intervention specialists) each rated one student across all eight constructs after being assigned to one of the four assessment conditions. Dependability estimates varied substantially across methods and constructs (range = 0.75-0.96), although findings of the present study support the use of the broad set of formative assessment tools evaluated.

Implications of bias in automated writing quality scores for fair and equitable assessment decisions.

Recent advances in automated writing evaluation have enabled educators to use automated writing quality scores to improve assessment feasibility. However, there has been limited investigation of bias for automated writing quality scores with students from diverse racial or ethnic backgrounds. The use of biased scores could contribute to implementing unfair practices with negative consequences on student learning. The goal of this study was to investigate score bias of writeAlizer, a free and open-source automated writing evaluation program. For 421 students in Grades 4 and 7 who completed a state writing exam that included composition and multiple choice revising and editing questions, writeAlizer was used to generate automated writing quality scores for the composition section. Then, we used multiple regression models to investigate whether writeAlizer scores demonstrated differential predictions of the composition and overall scores on the state-mandated writing exam for students from different racial or ethnic groups. No evidence of bias for automated scores was observed. However, after controlling for automated scores in Grade 4, we found statistically significant group differences in regression models predicting overall state test scores 3 years later but not the essay composition scores. We hypothesize that the multiple choice revising and editing sections, rather than the scoring approach used for the essay portion, introduced construct-irrelevant variance and might lead to differential performance among groups. Implications for assessment development and score use are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Cost analysis and cost-effectiveness of hand-scored and automated approaches to writing screening.

Although researchers have investigated technical adequacy and usability of written-expression curriculum-based measures (WE-CBM), the economic implications of different scoring approaches have largely been ignored. The absence of such knowledge can undermine the effective allocation of resources and lead to the adoption of suboptimal measures for the identification of students at risk for poor writing outcomes. Therefore, we used the Ingredients Method to compare implementation costs and cost-effectiveness of hand-calculated and automated scoring approaches. Data analyses were conducted on secondary data from a study that evaluated predictive validity and diagnostic accuracy of quantitative approaches for scoring WE-CBM samples. Findings showed that automated approaches offered more economic solutions than hand-calculated methods; for automated scores, the effects were stronger when the free writeAlizer R package was employed, whereas for hand-calculated scores, simpler WE-CBM metrics were less costly than more complex metrics. Sensitivity analyses confirmed the relative advantage of automated scores when the number of classrooms, students, and assessment occasions per school year increased; again, writeAlizer was less sensitive to the changes in the ingredients than the other approaches. Finally, the visualization of the cost-effectiveness ratio illustrated that writeAlizer offered the optimal balance between implementation costs and diagnostic accuracy, followed by complex hand-calculated metrics and a proprietary automated program. Implications for the use of hand-calculated and automated scores for the universal screening of written expression with elementary students are discussed.

Diagnosed and Nondisclosed Sport-Related Concussion: An Exploratory Comparison Study by ADHD Status in Collegiate Athletes.

OBJECTIVE: To determine if ADHD status in collegiate athletes was associated with differences in the number of diagnosed and nondisclosed sport-related concussions, and reasons why suspected concussive injuries were not reported. METHOD: A cross-sectional survey was completed by 858 collegiate athletes (65 with ADHD; 793 without ADHD). The survey included self-report items regarding ADHD status and histories of diagnosed and nondisclosed sport-related concussions. Groups were matched for gender, age, and sport with a ratio of one participant with ADHD to four without ADHD. RESULTS: Collegiate athletes with ADHD reported significantly more diagnosed sport-related concussions and were more likely to have a history of diagnosis compared to those without ADHD. There were no significant differences between groups regarding sport-related concussion nondisclosure history. CONCLUSION: Collegiate athletes with ADHD have an increased prevalence rate of diagnosed sport-related concussions compared to those without ADHD; however, this developmental disorder may not influence injury nondisclosure decisions.

Five direct behavior rating multi-item scales: Sensitivity to the effects of classroom interventions.

Direct Behavior Rating (DBR) is a tool designed for the assessment of behavioral changes over time. Unlike methods for summative evaluations, the development of progress monitoring tools requires evaluation of sensitivity to change. The present study aimed to evaluate this psychometric feature of five newly developed DBR Multi-Item Scales (DBR-MIS). Teachers identified students with behaviors interfering with their learning or the learning of others and implemented a Daily Report Card (DRC) intervention in the classroom settings for two months. The analyses were performed on 31 AB single case studies. Change metrics were calculated at an individual level by using Tau-UA vs. B + trend B and Hedges' g and at a scale-level by using Mixed Effect Meta-Analysis, Hierarchical Linear Models (HLMs), and Between-Case Standardized Mean Difference (BC-SMD). HLMs were estimated considering both fixed and random effects of intervention and linear trend within the intervention phase. The results supported sensitivity to change for three DBR-MIS (i.e., Academic Engagement, Organizational Skills, and Disruptive Behavior), and the relative magnitudes were consistent across the metrics. Sensitivity to change of DBR-MIS Interpersonal Skills received moderate support. Conversely, empirical evidence was not provided for sensitivity to change of DBR-MIS Oppositional Behavior. Particular emphasis was placed on the intervention trend in that responses to behavioral interventions might occur gradually or require consistency over time in order to be observed by raters. Implications for the use of the new DBR-MIS in the context of progress monitoring of social-emotional behaviors are discussed.

An Online Tool to Assess Sentence Comprehension in Teenagers at Risk for School Exclusion: Evidence From L2 Italian Students.

This study presents a web-based sentence comprehension test aimed at identifying high school students who are at risk for a language delay. By assessing linguistic skills on a sample of high school students with Italian as an L2 and their monolingual peers, attending a vocational school, we were able to identify a subgroup of L2 students with consistent difficulties in sentence comprehension, though their reading skills were within the average range. The same subgroup revealed to experience a lack of support within the school context, suggesting that poor L2 skills might be a critical variable to consider in order to identify students at risk for school exclusion. Regarding the test, accuracy to the on-line sentence comprehension task was significantly predicted by reading abilities and vocabulary skills, thus indicating that this test might represent a rapid but efficient way to assess linguistic abilities at school. We recommend that establishing a valid and practical procedure for the evaluation of linguistic skills in bilingual students who struggle with their L2 is the first step toward promoting social inclusion in the multilingual classroom, in order to increase their ability to actively participate in school and social activities.

Cognitive Profile of Intellectually Gifted Adults: Analyzing the Wechsler Adult Intelligence Scale.

The Wechsler Adult Intelligence Scale (WAIS) has been used extensively to study intellectual abilities of special groups. Here, we report the results of an intellectually gifted group on the WAIS-IV. Gifted individuals are people who obtained scores equal to or greater than 2 standard deviations above the mean on an intelligence test. Hence, the current study aims first, to examine mean group performance data of gifted individuals on the WAIS-IV; second, to revalidate the pattern of performance identified in this special group in previous studies (i.e., verbal skills higher than all other abilities); third, to compare scatter measures across intellectual domains with a matched comparison group. A total of 130 gifted individuals (79 males) were administered the full battery and their performance was compared with a matched comparison group. Analyses revealed that gifted group displayed higher scores in all intellectual domains. Contrary to expectations, they showed the highest scores in perceptual reasoning tasks. A multivariate approach revealed that this ability was statistically different from all other domains within the gifted group. Moreover, gifted individuals showed higher discrepancies across intellectual domains than average-intelligence people. Findings have important practical implications to detect intellectual giftedness in adulthood.