Formative behavioral assessment across eight constructs: Dependability of direct behavior ratings and formative behavior rating measures.

Due to their promise as a feasible tool for evaluating the effects of school-based interventions, Direct Behavior Ratings (DBR) have received much research attention over the past 2 decades. Although DBR methodology has demonstrated much promise, favorable psychometric characteristics only have been demonstrated for tools measuring a small number of constructs. Likewise, although a variety of methods of DBR have been proposed, most extant studies have focused on the use of single-item methods. The present study examined the dependability of four methods of formative behavioral assessment (i.e., single-item and multi-item ratings administered either daily [DBR] or weekly [formative behavior rating measures or FBRM]) across eight psychological constructs (i.e., interpersonal skills, academic engagement, organizational skills, disruptive behavior, oppositional behavior, interpersonal conflict, anxious depressed, and social withdrawal). School-based professionals (N = 91; i.e., teachers, paraprofessionals, and intervention specialists) each rated one student across all eight constructs after being assigned to one of the four assessment conditions. Dependability estimates varied substantially across methods and constructs (range = 0.75-0.96), although findings of the present study support the use of the broad set of formative assessment tools evaluated.

Meta-analysis of skill-based and therapeutic interventions to address math anxiety.

The purpose of this meta-analysis was to examine the impact of school-based therapeutic and math skill interventions on math anxiety symptoms and math achievement among K-12 students. Potential moderators included treatment type and study quality. A systematic search yielded 17 included studies representing 1786 primary and secondary students. The results suggested that therapeutic interventions reduced math anxiety symptoms (gav = -0.51) better than math skill interventions (gav = -0.32) and math skill interventions improved math achievement (gav = 0.76) more than therapeutic interventions (gav = 0.12). Moderator analysis indicated that when accounting for study quality, the differences between intervention type were not significant for either of the outcome measures (i.e., achievement and math anxiety). Theoretical and practical implications of these findings are discussed.

Measurement Efficiency of a Teacher Rating Scale to Screen for Students at Risk for Social, Emotional, and Behavioral Problems.

Teacher rating scales are broadly used for psycho-educational assessment in schools. In particular, they play an important role in screening students for social, emotional, and behavioral problems. In order to optimize the efficiency of these measures, it is important to minimize the number of items comprising them while maintaining sound psychometric characteristics. This study examines the measurement efficiency of a teacher rating scale for student social, emotional, and behavioral risk. The goal was to shorten an existing behavior screening tool. A total of 139 classroom teachers and 2566 students from Grades 1-6 (Mage = 8.96 years, SD = 1.61) participated in the study. In sum, 35 items assessing internalizing and externalizing behavior problems were analyzed applying the item response theory (generalized partial credit model). The results show that social, emotional, and behavioral risks can be captured with a total of 12 items. This reduction of almost 66% of the initial item pool would take teachers about 90 s to fill out for one student. Thus, the rating scale can be used by teachers in an efficient yet psychometrically sound manner.

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating-multi-item scales.

Although originally conceived of as a marriage of direct behavioral observation and indirect behavior rating scales, recent research has indicated that Direct Behavior Ratings (DBRs) are affected by rater idiosyncrasies (rater effects) similar to other indirect forms of behavioral assessment. Most of this research has been conducted using generalizability theory (GT), yet another approach, many-facet Rasch measurement (MFRM), has recently been utilized to illuminate the previously opaque nature of these rater idiosyncrasies. The purpose of this study was to utilize both approaches (GT and MFRM) to consider rater effects with 126 second- through fifth-grade students who were rated on two DBR-Multi-Item Scales by four raters (22 of these ratings were fully crossed). Results indicated the presence of rater effects and revealed nuances about their nature, including showing differences across construct domains, identifying items that are potentially more susceptible to rater effects than others, and isolating specific raters who appear to have been more susceptible to rater effects than other raters. These findings further indicate the indirect nature of DBRs and offer potential avenues for addressing and ameliorating rater effects in research and practice. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The science of school psychologists: Developing a standard definition.

This article offers a summary, synthesis, and highlights of processes, surveys, summits, and discussions among members of the Society for the Study of School Psychology exploring current and historical perspectives as to a potential definition of school psychology research. After multiple revisions, the final definition is as follows: School psychology research is the systematic investigation of phenomena concerning the educational, emotional, behavioral, and social outcomes, as well as the school, family, and societal systems in which these outcomes are imbedded. It is principally concerned with socially responsive implementation and translating research into equitable practices that can lead to improved functioning of students, families, schools, and community systems that support educational and psychological services. It includes basic and applied research and theory development that focuses on a wide array of topics including, for example, prevention, intervention, assessment, diagnosis, diversity, equity and inclusion, measurement, methodology, statistics, and professional issues. This wide range of topics is informed by diverse theories arising from a broad array of disciplines and investigated by researchers from a variety of educational and scientific backgrounds. Common to all of the areas of research inquiry is a focus on enhancing student, family, and community outcomes-particularly educational and social outcomes of all students-and in building the capacity of systems (schools, communities, and other systems) to support those outcomes with a commitment to incorporating social responsiveness in the research process. The aim of this definition is to embrace the breadth of scholarship informing and advancing the science, practice, and policy relevant to the field of school psychology.

Five direct behavior rating multi-item scales: Sensitivity to the effects of classroom interventions.

Direct Behavior Rating (DBR) is a tool designed for the assessment of behavioral changes over time. Unlike methods for summative evaluations, the development of progress monitoring tools requires evaluation of sensitivity to change. The present study aimed to evaluate this psychometric feature of five newly developed DBR Multi-Item Scales (DBR-MIS). Teachers identified students with behaviors interfering with their learning or the learning of others and implemented a Daily Report Card (DRC) intervention in the classroom settings for two months. The analyses were performed on 31 AB single case studies. Change metrics were calculated at an individual level by using Tau-UA vs. B + trend B and Hedges' g and at a scale-level by using Mixed Effect Meta-Analysis, Hierarchical Linear Models (HLMs), and Between-Case Standardized Mean Difference (BC-SMD). HLMs were estimated considering both fixed and random effects of intervention and linear trend within the intervention phase. The results supported sensitivity to change for three DBR-MIS (i.e., Academic Engagement, Organizational Skills, and Disruptive Behavior), and the relative magnitudes were consistent across the metrics. Sensitivity to change of DBR-MIS Interpersonal Skills received moderate support. Conversely, empirical evidence was not provided for sensitivity to change of DBR-MIS Oppositional Behavior. Particular emphasis was placed on the intervention trend in that responses to behavioral interventions might occur gradually or require consistency over time in order to be observed by raters. Implications for the use of the new DBR-MIS in the context of progress monitoring of social-emotional behaviors are discussed.

The Effectiveness of Daily Behavior Report Cards for Children With ADHD: A Meta-Analysis.

Objective: This meta-analysis examined group-design studies investigating the effectiveness of Daily Behavior Report Cards (DBRC) as a school-based intervention to manage the classroom behavior of students with ADHD. Methods: A search of three article databases (PsycINFO, ERIC and Medline) identified seven group design evaluations of DBRC interventions. This meta-analysis included a total of 272 participants, with an average age of 7.9 years old. Three of the studies compared a control group to a treatment group with randomized group assignment, one study compared a control group to three treatment groups, two studies compared pre-and post-treatment scores in the same group, and one study compared pre-and post-treatment results of two intervention groups without random assignment. Dependent measures for these studies were teacher ratings (n = 5) and systematic direct observation of student academic and social behaviour (n = 2). Standardized mean differences (Hedge's g) were calculated to obtain a pooled effect size using fixed effects. Results: DBRCs were associated with reductions teacher-rated ADHD symptoms, with a Hedge's g of 0.36 (95% CI: 0.12-0.60, z=2.93, p ≤ .005) with low heterogeneity (Q-value: 2.40, I2 = 0.00). This result excluded two studies that used observational coding instead of standardized tests to evaluate the effects of the intervention. A moderator analysis indicated that the effect size for systematic direct observation was large (Hedge's G = 1.05[95% CI: 0.66-1.44, z=5.25, p ≤ .00]), with very high heterogeneity (Q-value: 46.34, I2: 93.53). A second moderator analysis found differences in the effects of DBRCs for comorbid externalizing symptoms with an overall effect size of 0.34 (95%CI: -0.04-0.72, z=1.76 p =0.08) with high heterogeneity (Q-value: 3.98, I2: 74.85). Conclusions: DBRCs effectively reduce the frequency and severity of ADHD symptoms in classroom settings. Additionally, they have a significant effect on co-occuring externalizing behaviors. It appears that systematic direct observation may be a more sensitive measure of treatment effects compared to teacher ratings of ADHD symptoms.

A universal behavioral screener linked to personalized classroom interventions: Psychometric characteristics in a large sample of German schoolchildren.

The current study represents the first psychometric evaluation of an American English into German translation of a school-based universal screening measure designed to assess academic and disruptive behavior problems. This initial study examines the factor structure and diagnostic accuracy of the newly translated measure in a large sample of 1009 German schoolchildren attending grades 1-6 in Western Germany. Confirmatory factor analysis supported a two-factor model for both male- and female- students. Configural invariance was supported between male- and female-samples. However scalar invariance was not supported, with higher thresholds for ratings of female students. Results of receiver operating characteristic (ROC) curve analyses were indicative of good to excellent diagnostic accuracy with areas under the curve ranging from 0.89 to 0.93. Optimal cut-off scores were 10, 5, and 13 for the Academic Productivity/Disorganization, Oppositional/Disruptive, and the Total Problems Composite scores of the Integrated System Teacher Rating Form respectively. This initial study of the newly translated measure supports further investigations into its utility for universal screening in German speaking schools.

Classification accuracy and acceptability of the Integrated Screening and Intervention System Teacher Rating Form.

This study examines the classification accuracy and teacher acceptability of a problem-focused screener for academic and disruptive behavior problems, which is directly linked to evidence-based intervention. Participants included 39 classroom teachers from 2 public school districts in the Northeastern United States. Teacher ratings were obtained for 390 students in Grades K-6. Data from the screening instrument demonstrate favorable classification accuracy, and teacher ratings of feasibility and acceptability support the use of the measure for universal screening in elementary school settings. Results indicate the novel measure should facilitate classroom intervention for problem behaviors by identifying at-risk students and informing targets for daily behavior report card interventions. (PsycINFO Database Record

Parent-administered computer-assisted tutoring targeting letter-sound knowledge: Evaluation via multiple-baseline across three preschool students.

Knowledge of letters sounds has been identified as a primary objective of preschool instruction and intervention. Despite this designation, large disparities exist in the number of letter sounds children know at school entry. Enhancing caregivers' ability to teach their preschool-aged children letter sounds may represent an effective practice for reducing this variability and ensuring that more children are prepared to experience early school success. This study used a non-concurrent multiple-baseline-across-participants design to evaluate the effectiveness of caregivers (N=3) delivering a computer-assisted tutoring program (Tutoring Buddy) targeting letter sound knowledge to their preschool-aged children. Visual analyses and effect size estimates derived from Percentage of All Non-Overlapping Data (PAND) statistics indicated consistent results for letter sound acquisition, as 6weeks of intervention yielded large effects for letter sound knowledge (LSK) across all three children. Large effect sizes were also found for letter sound fluency (LSF) and nonsense word fluency (NWF) for two children. All three caregivers rated the intervention as highly usable and were able to administer it with high levels of fidelity. Taken together, the results of the present study found Tutoring Buddy to be an effective, simple, and usable way for the caregivers to support their children's literacy development.

Multi-item direct behavior ratings: Dependability of two levels of assessment specificity.

Direct Behavior Rating-Multi-Item Scales (DBR-MIS) have been developed as formative measures of behavioral assessment for use in school-based problem-solving models. Initial research has examined the dependability of composite scores generated by summing all items comprising the scales. However, it has been argued that DBR-MIS may offer assessment of 2 levels of behavioral specificity (i.e., item-level, global composite-level). Further, it has been argued that scales can be individualized for each student to improve efficiency without sacrificing technical characteristics. The current study examines the dependability of 5 items comprising a DBR-MIS designed to measure classroom disruptive behavior. A series of generalizability theory and decision studies were conducted to examine the dependability of each item (calls out, noisy, clowns around, talks to classmates and out of seat), as well as a 3-item composite that was individualized for each student. Seven graduate students rated the behavior of 9 middle-school students on each item over 3 occasions. Ratings were based on 10-min video clips of students during mathematics instruction. Separate generalizability and decision studies were conducted for each item and for a 3-item composite that was individualized for each student based on the highest rated items on the first rating occasion. Findings indicate favorable dependability estimates for 3 of the 5 items and exceptional dependability estimates for the individualized composite.

An evaluation of observational methods for measuring response to classwide intervention.

Although there is much research to support the effectiveness of classwide interventions aimed at improving student engagement, there is also a great deal of variability in terms of how response to group-level intervention has been measured. The unfortunate consequence of this procedural variability is that it is difficult to determine whether differences in obtained results across studies are attributable to the way in which behavior was measured or actual intervention effectiveness. The purpose of this study was to comparatively evaluate the most commonly used observational methods for monitoring the effects of classwide interventions in terms of the degree to which obtained data represented actual behavior. The 5 most common sampling methods were identified and evaluated against a criterion generated by averaging across observations conducted on 14 students in one seventh-grade classroom. Results suggested that the best approximation of mean student engagement was obtained by observing a different student during each consecutive 15-s interval whereas observing an entire group of students during each interval underestimated the mean level of behavior within a phase and the degree of behavior change across phases. In contrast, when observations were restricted to the 3 students with the lowest levels of engagement, data revealed greater variability in engagement across baseline sessions and suggested a more notable change in student behavior subsequent to intervention implementation.

Development of a problem-focused behavioral screener linked to evidence-based intervention.

This study examines the factor structure, reliability and validity of a novel school-based screening instrument for academic and disruptive behavior problems commonly experienced by children and adolescents with attention deficit hyperactivity disorder (ADHD). Participants included 39 classroom teachers from two public school districts in the northeastern United States. Teacher ratings were obtained for 390 students in grades K-6. Exploratory factor analysis supports a two-factor structure (oppositional/disruptive and academic productivity/disorganization). Data from the screening instrument demonstrate favorable internal consistency, temporal stability and convergent validity. The novel measure should facilitate classroom intervention for problem behaviors associated with ADHD by identifying at-risk students and determining specific targets for daily behavior report card interventions.

The influence of student characteristics on the dependability of behavioral observation data.

Although generalizability theory has been used increasingly in recent years to investigate the dependability of behavioral estimates, many of these studies have relied on use of general education populations as opposed to those students who are most likely to be referred for assessment due to problematic classroom behavior (e.g., inattention, disruption). The current study investigated the degree to which differences exist in terms of the magnitude of both variance component estimates and dependability coefficients between students nominated by their teachers for Tier 2 interventions due to classroom behavior problems and a general classroom sample (i.e., including both nominated and non-nominated students). The academic engagement levels of 16 (8 nominated, 8 non-nominated) middle school students were measured by 4 trained observers using momentary time-sampling procedures. A series of G and D studies were then conducted to determine whether the 2 groups were similar in terms of the (a) distribution of rating variance and (b) number of observations needed to achieve an adequate level of dependability. Results suggested that the behavior of students in the teacher-nominated group fluctuated more across time and that roughly twice as many observations would therefore be required to yield similar levels of dependability compared with the combined group. These findings highlight the importance of constructing samples of students that are comparable to those students with whom the measurement method is likely to be applied when conducting psychometric investigations of behavioral assessment tools.

The influence of observation length on the dependability of data.

Although direct observation is one of the most frequently used assessment methods by school psychologists, studies have shown that the number of observations needed to obtain a dependable estimate of student behavior may be impractical. Because direct observation may be used to inform important decisions about students, it is crucial that data be reliable. Preliminary research has suggested that dependability may be improved by extending the length of individual observations. The purpose of the current study was, therefore, to examine how changes in observational duration affect the dependability of student engagement data. Twenty seventh grade students were each observed for 30-min across 2 days during math instruction. Generalizability theory was then used to calculate reliability-like coefficients for the purposes of intraindividual decision making. Across days, acceptable levels of dependability for progress monitoring (i.e., .70) were achieved through two 30-min observations, three 15-min observations, or four to five 10-min observations. Acceptable levels of dependability for higher stakes decisions (i.e., .80) required over an hour of cumulative observation time. Within a given day, a 15 minute observation was found to be adequate for making low-stakes decisions whereas an hour long observation was necessary for high-stakes decision making. Limitations of the current study and implications for research and practice are discussed.

The efficiency of behavior rating scales to assess inattentive-overactive and oppositional-defiant behaviors: applying generalizability theory to streamline assessment.

Although the efficiency with which a wide range of behavioral data can be obtained makes behavior rating scales particularly attractive tools for the purposes of screening and evaluation, feasibility concerns arise in the context of formative assessment. Specifically, informant load, or the amount of time informants are asked to contribute to the assessment process, likely has a negative impact on the quality of data over time and the informant's willingness to participate. Two important determinants of informant load in progress monitoring are the length of the rating scale (i.e., the number of items) and how frequently informants are asked to provide ratings (i.e., the number of occasions). The purpose of the current study was to investigate the dependability of the IOWA Conners Teacher Rating Scale (Loney & Milich, 1982), which is used to differentiate inattentive-overactive from oppositional-defiant behaviors. Specifically, the facets of items and occasions were examined to identify combinations of these sources of error necessary to reach an acceptable level of dependability for both absolute and relative decisions. Results from D studies elucidated a variety of possible item-occasion combinations reaching the criteria for adequate dependability. Recommendations for research and practice are discussed.

Retrospective ratings of ADHD symptoms made at young adulthood by clinic-referred boys with ADHD-related problems, their brothers without ADHD, and control participants.

Retrospective childhood attention-deficit/hyperactivity disorder (ADHD) symptoms are required to diagnosis adult ADHD, but the validity of self-rated symptoms across time is questionable. Here, boys with ADHD-related problems, their brothers without ADHD, and former schoolmates rated themselves during young adulthood for ages 9, 14, and 19. Brothers rated probands retrospectively at the same ages. The young adults referred as children for ADHD (a) acknowledged childhood symptoms; (b) described improvement over time; (c) did not differ from brothers or controls on most self-ratings of young adult symptoms; (d) rated themselves as more symptomatic at age 9, but less symptomatic at age 19, than their brothers rated them; and (e) agreed only to some degree with brothers' ratings of probands' aggression (median correlation = .22). Probands' ratings showed limited agreement with judges' symptom ratings (median correlation = .16) and young adult follow-up examiners' ratings (median correlation = .14). These findings are not accounted for solely by changes in informants, nor by the course of ADHD psychopathology. They suggest some stability but limited internal consistency and validity for retrospective ADHD ratings by probands and brothers.

Consultation-based academic interventions for children with ADHD: effects on reading and mathematics achievement.

The purpose of this investigation was to evaluate the relative efficacy of two consultation-based models for designing academic interventions to enhance the educational functioning of children with attention-deficit/hyperactivity disorder (ADHD). Children (N=167) meeting DSM-IV criteria for ADHD were randomly assigned to one of two consultation groups: Individualized Academic Intervention (IAI; interventions designed using a data-based decision-making model that involved ongoing feedback to teachers) and Generic Academic Intervention (GAI; interventions designed based on consultant-teacher collaboration, representing "consultation as usual"). Teachers implemented academic interventions over 15 months. Academic outcomes (e.g., standardized achievement test, and teacher ratings of academic skills) were assessed on four occasions (baseline, 3 months, 12 months, 15 months). Hierarchical linear modeling analyses indicated significant positive growth for 8 of the 14 dependent variables; however, trajectories did not differ significantly across consultation groups. Interventions in the IAI group were delivered with significantly greater integrity; however, groups did not differ with respect to teacher ratings of treatment acceptability. The results of this study provide partial support for the effectiveness of consultation-based academic interventions in enhancing educational functioning in children with ADHD; however, the relative advantages of an individualized model over "consultation as usual" have yet to be established.

Validation of three dimensions of childhood psychopathology in young clinic-referred boys.

Short measures of child inattention-overactivity (IO), aggression-defiance (AG), and anxiety-depression or emotionality (EM) derived through a double validation procedure are administered to mothers of 243 clinic-referred suburban New York boys between 6 and 10 years of age. Mother-rated IO is uniquely related to poor performance on cognitive and achievement tests; observed inattentive, hyperactive, and impulsive behaviors in a restricted academic setting; less father education and lower family income; and most mother-reported impairments and treatment use. Mothers of high-IO boys describe themselves and their sons as having similar childhood symptoms. AG is uniquely related to child-reported disruptive behavior and sensation seeking, many measures of family conflict and negative parenting styles, and mother-reported symptom pervasiveness and number of treatments. EM is uniquely related only to poorer cognitive and achievement test performance, living with one parent, parents who considered themselves too busy, and fewer friends. Each dimension also is associated with parallel teacher-rated factors.

A DSM-IV-referenced screening instrument for preschool children: the Early Childhood Inventory-4.

OBJECTIVE: To examine the validity of the Early Childhood Inventory-4 (ECI-4), a parent and teacher rating scale designed to screen for DSM-IV emotional and behavioral disorders. METHOD: The convergent, divergent, and discriminant validity and clinical utility of the ECI-4 was studied in a sample of 224 consecutive referrals (3-6-year-old children) to a child psychiatric outpatient clinic. RESULTS: The ECI-4 demonstrated adequate criterion validity for the most common disorders (attention-deficit/hyperactivity disorder, oppositional defiant disorder, pervasive developmental disorder) when compared with data-based psychiatric diagnoses and correlated well with relevant scales of the Child Behavior Checklist, Teacher's Report Form, and Inattention/Overactivity With Aggression (IOWA) Conners. CONCLUSION: The ECI-4 appears to be a useful screening measure for certain disorders in clinically referred children, but continued research is needed to determine its value in other settings (e.g., school and community), and its validity with other measurement methodologies.