Timing and frequency of screening in schools: A latent variable analysis of behavioral stability over time.

The growth of school-based initiatives incorporating multitiered systems of support (MTSS) for social, emotional, and behavioral domains has fueled interest in behavioral assessment. These assessments are foundational to determining risk for behavioral difficulties, yet research to date has been limited with regard to when and how often to administer them. The present study evaluated these questions within the framework of behavioral stability and examined the extent to which behavior is stable when measured by two school-based behavioral assessments: the Direct Behavior Rating-Single-Item Scales (DBR-SIS), and the Behavioral and Emotional Screening System (BESS). Participants included 451 students rated three times per year across 4 years, with the primary teacher from each year providing the within-year ratings. Latent variable models were employed to measure the constructs underlying the observed assessment scores. Models demonstrated that the DBR-SIS best captured changes within the year, whereas the BESS scores remained stable across time points within a year. Across years, scores from both assessments captured changes. The unique contributions of each assessment in the data-based decision-making process are discussed, and recommendations are given for their combined use within and across school years. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Examining the stability of social, emotional, and behavioral risk status: Implications for screening frequency.

Responsive service delivery frameworks rely on the use of screening approaches to identify students in need of support and to guide subsequent assessment and intervention efforts. However, limited empirical investigations have been directed to informing how often screening should occur for social, emotional, and behavioral difficulties in school settings. The purpose of the current study was to evaluate the stability of risk status on 3 different screening instruments across 3 administrations across the course of a school year. A total of 1,594 students had complete screening data across 3 time points, corresponding to a total of 187 teachers from 22 different public schools located within the northeastern and midwestern United States. Across measures, we examined patterns of risk across time points and investigated the utility of (a) different screening schedules (fall, winter, and spring) and (b) borderline screening when conducting multiple screenings per year. Results indicated that a large proportion of students exhibited stable risk patterns across time points and suggested that borderline screening may be a viable alternative for schools with limited resources. Implications for practice and future research are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Developing a direct rating behavior scale for depression in middle school students.

Research has supported the applied use of Direct Behavior Rating Single-Item Scale (DBR-SIS) targets of "academic engagement" and "disruptive behavior" for a range of purposes, including universal screening and progress monitoring. Though useful in evaluating social behavior and externalizing problems, these targets have limited utility in evaluating emotional behavior and internalizing problems. Thus, the primary purpose of this study was to support the initial development and validation of a novel DBR-SIS target of "unhappy," which was intended to tap into the specific construct of depression. A particular focus of this study was on the novel target's utility within universal screening. A secondary purpose was to further validate the aforementioned existing DBR-SIS targets. Within this study, 87 teachers rated 1,227 students across two measures (i.e., DBR-SIS and the Teacher Observation of Classroom Adaptation-Checklist [TOCA-C]) and time points (i.e., fall and spring). Correlational analyses supported the test-retest reliability of each DBR-SIS target, as well as its convergent and discriminant validity across concurrent and predictive comparisons. Receiver operating characteristic (ROC) curve analyses further supported (a) the overall diagnostic accuracy of each target (as indicated by the area under the curve [AUC] statistic), as well as (b) the selection of cut scores found to accurately differentiate at-risk and not at-risk students (as indicated by conditional probability statistics). A broader review of findings suggested that across the majority of analyses, the existing DBR-SIS targets outperformed the novel "unhappy" target. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Methods matter: A multi-trait multi-method analysis of student behavior.

Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating - Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3-8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed.

Test order in teacher-rated behavior assessments: Is counterbalancing necessary?

Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance. (PsycINFO Database Record

Assessing, understanding, and supporting students with autism at school: Contemporary science, practice, and policy.

Over the past 3 decades, there has been an unprecedented increase in students identified as eligible for special education as a result of students meeting criteria for autism spectrum disorder (ASD). The increasing number of students with ASD in the schools presents significant challenges to teachers, school psychologists, and other school professionals working with this population. Although there is considerable research addressing assessment, identification, and support services for children with ASD, there is a need for further research focused on these topics within the school context. Employing a diverse array of methodologies, the articles in this special topic section address several gaps in the literature, including (a) the application of evidence-based programs within the school context, (b) the social validity of well-established evidence-based practices with both parents and educators, (c) the assessment of social communication, (d) intervention and assessment of spatial and body awareness for children with ASD, (e) the use of peer-mediated discreet trial training, and (f) discrepancies across informants for both externalizing and internalizing symptoms associated with ASD. The results of these studies provide school psychologists and other education professionals with specific directions for advocacy and service delivery that aim to enhance school outcomes for students with ASD. (PsycINFO Database Record

Reliability of Direct Behavior Ratings - Social Competence (DBR-SC) data: How many ratings are necessary?

The purpose of this investigation was to evaluate the reliability of Direct Behavior Ratings-Social Competence (DBR-SC) ratings. Participants included 60 students identified as possessing deficits in social competence, as well as their 23 classroom teachers. Teachers used DBR-SC to complete ratings of 5 student behaviors within the general education setting on a daily basis across approximately 5 months. During this time, each student was assigned to 1 of 2 intervention conditions, including the Social Competence Intervention-Adolescent (SCI-A) and a business-as-usual (BAU) intervention. Ratings were collected across 3 intervention phases, including pre-, mid-, and postintervention. Results suggested DBR-SC ratings were highly consistent across time within each student, with reliability coefficients predominantly falling in the .80 and .90 ranges. Findings further indicated such levels of reliability could be achieved with only a small number of ratings, with estimates varying between 2 and 10 data points. Group comparison analyses further suggested the reliability of DBR-SC ratings increased over time, such that student behavior became more consistent throughout the intervention period. Furthermore, analyses revealed that for 2 of the 5 DBR-SC behavior targets, the increase in reliability over time was moderated by intervention grouping, with students receiving SCI-A demonstrating greater increases in reliability relative to those in the BAU group. Limitations of the investigation as well as directions for future research are discussed herein. (PsycINFO Database Record

A comparison of measures to screen for social, emotional, and behavioral risk.

The purpose of this study was to examine the relation between teacher-implemented screening measures used to identify social, emotional, and behavioral risk. To this end, 5 screening options were evaluated: (a) Direct Behavior Rating - Single Item Scales (DBR-SIS), (b) Social Skills Improvement System - Performance Screening Guide (SSiS), (c) Behavioral and Emotional Screening System - Teacher Form (BESS), (d) Office discipline referrals (ODRs), and (e) School nomination methods. The sample included 1974 students who were assessed tri-annually by their teachers (52% female, 93% non-Hispanic, 81% white). Findings indicated that teacher ratings using standardized rating measures (DBR-SIS, BESS, and SSiS) resulted in a larger proportion of students identified at-risk than ODRs or school nomination methods. Further, risk identification varied by screening option, such that a large percentage of students were inconsistently identified depending on the measure used. Results further indicated weak to strong correlations between screening options. The relation between broad behavioral indicators and mental health screening was also explored by examining classification accuracy indices. Teacher ratings using DBR-SIS and SSiS correctly identified between 81% and 91% of the sample as at-risk using the BESS as a criterion. As less conservative measures of risk, DBR-SIS and SSiS identified more students as at-risk relative to other options. Results highlight the importance of considering the aims of the assessment when selecting broad screening measures to identify students in need of additional support.

Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS).

The purpose of this investigation was to evaluate the models for interpretation and use that serve as the foundation of an interpretation/use argument for the Social and Academic Behavior Risk Screener (SABRS). The SABRS was completed by 34 teachers with regard to 488 students in a Midwestern high school during the winter portion of the academic year. Confirmatory factor analysis supported interpretation of SABRS data, suggesting the fit of a bifactor model specifying 1 broad factor (General Behavior) and 2 narrow factors (Social Behavior [SB] and Academic Behavior [AB]). The interpretive model was further supported by analyses indicative of the internal consistency and interrater reliability of scores from each factor. In addition, latent profile analyses indicated the adequate fit of the proposed 4-profile SABRS model for use. When cross-referenced with SABRS cut scores identified via previous work, results revealed students could be categorized as (a) not at-risk on both SB and AB, (b) at-risk on SB but not on AB, (c) at-risk on AB but not on SB, or (d) at-risk on both SB and AB. Taken together, results contribute to growing evidence supporting the SABRS within universal screening. Limitations, implications for practice, and future directions for research are discussed herein.

Direct behavior rating as a school-based behavior universal screener: replication across sites.

The purpose of this study was to evaluate the utility of Direct Behavior Rating Single Item Scale (DBR-SIS) targets of disruptive, engaged, and respectful behavior within school-based universal screening. Participants included 31 first-, 25 fourth-, and 23 seventh-grade teachers and their 1108 students, sampled from 13 schools across three geographic locations (northeast, southeast, and midwest). Each teacher rated approximately 15 of their students across three measures, including DBR-SIS, the Behavioral and Emotional Screening System (Kamphaus & Reynolds, 2007), and the Student Risk Screening Scale (Drummond, 1994). Moderate to high bivariate correlations and area under the curve statistics supported concurrent validity and diagnostic accuracy of DBR-SIS. Receiver operating characteristic curve analyses indicated that although respectful behavior cut scores recommended for screening remained constant across grade levels, cut scores varied for disruptive behavior and academic engaged behavior. Specific cut scores for first grade included 2 or less for disruptive behavior, 7 or greater for academically engaged behavior, and 9 or greater for respectful behavior. In fourth and seventh grades, cut scores changed to 1 or less for disruptive behavior and 8 or greater for academically engaged behavior, and remained the same for respectful behavior. Findings indicated that disruptive behavior was particularly appropriate for use in screening at first grade, whereas academically engaged behavior was most appropriate at both fourth and seventh grades. Each set of cut scores was associated with acceptable sensitivity (.79-.87), specificity (.71-.82), and negative predictive power (.94-.96), but low positive predictive power (.43-.44). DBR-SIS multiple gating procedures, through which students were only considered at risk overall if they exceeded cut scores on 2 or more DBR-SIS targets, were also determined acceptable in first and seventh grades, as the use of both disruptive behavior and academically engaged behavior in defining risk yielded acceptable conditional probability indices. Overall, the current findings are consistent with previous research, yielding further support for the DBR-SIS as a universal screener. Limitations, implications for practice, and directions for future research are discussed.

Direct Behavior Rating: an evaluation of time-series interpretations as consequential validity.

Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three "true" trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67% of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.

Direct behavior rating as a school-based behavior screener for elementary and middle grades.

The purpose of this study was to investigate how Direct Behavior Rating Single Item Scales (DBR-SIS) involving targets of academically engaged, disruptive, and respectful behaviors function in school-based screening assessment. Participants included 831 students in kindergarten through eighth grades who attended schools in the northeastern United States. Teachers provided behavior ratings for a sample of students in their classrooms on the DBR-SIS, the Behavioral and Emotional Screening System (Kamphaus & Reynolds, 2007), and the Student Risk Screening Scale (Drummond, 1994). Given variations in rating procedures to accommodate scheduling differences across grades, analysis was conducted separately for elementary school and middle school grade levels. Results suggested that the recommended cut scores, the combination of behavior targets, and the resulting conditional probability indices varied depending on grade level grouping (lower elementary, upper elementary, middle). For example, for the lower elementary grade level grouping, a combination of disruptive behavior (cut score=2) and academically engaged behavior (cut score=8) was considered to offer the best balance among indices of diagnostic accuracy, whereas a cut score of 1 for disruptive behavior and 8 for academically engaged behavior were recommended for the upper elementary school grade level grouping and cut scores of 1 and 9, respectively, were suggested for middle school grade level grouping. Generally, DBR-SIS cut scores considered optimal for screening using single or combined targets including academically engaged behavior and disruptive behavior by offering a reasonable balance of indices for sensitivity (.51-.90), specificity (.47-.83), negative predictive power (.94-.98), and positive predictive power (.14-.41). The single target of respectful behavior performed poorly across all grade level groups, and performance of DBR-SIS targets was relatively better in the elementary school than middle school grade level groups. Overall, results supported that disruptive behavior is highly important in evaluating risk status in lower grade levels and that academically engaged behavior becomes more pertinent as students reach higher grade levels. Limitations, future directions, and implications are discussed.

Stability of measures from children's interviews: the effects of time, sample length, and topic.

PURPOSE: The purpose of this study was to examine the reliability of, and sources of variability in, language measures from interviews collected from young school-age children. METHOD: Two 10-min interviews were collected from 20 at-risk kindergarten children by an examiner using a standardized set of questions. Test-retest reliability coefficients were calculated for 8 language measures. Generalizability theory (G-theory) analyses were completed to document the variability introduced into the measures from the child, session, sample length, and topic. RESULTS: Significant and strong reliability correlation coefficients were observed for most of the language sample measures. The G-theory analyses revealed that most of the variance in the language measures was attributed to the child. Session, sample length, and topic accounted for negligible amounts of variance in most of the language measures. CONCLUSION: Measures from interviews were reliable across sessions, and the sample length and topic did not have a substantial impact on the reliability of the language measures. Implications regarding the clinical feasibility of language sample analysis for assessment and progress monitoring are discussed.

Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades.

The purpose of the current investigation was to develop and provide initial validation of the Social and Academic Behavior Risk Screener (SABRS). Research was conducted in southeast elementary schools with 54 teacher and 243 student participants. An initial item pool was created through review of developmental research on the trajectory of behavior problems and competencies, as well as various models of social, emotional, and academic competence. A content validation process in addition to reliability and exploratory factor analyses resulted in development of a 12-item SABRS scale. Two factors emerged, with six items corresponding to "Social Behavior" and 6 items corresponding to "Academic Behavior." Subsequent correlational and receiver operating characteristic curve analyses revealed each scale, as well as an overall combined scale, to be a concurrently valid and diagnostically accurate predictor of the Social Skills Improvement System (SSIS) Teacher Rating Scales. The reader is provided a review of implications for practice and directions for future SABRS research.

Assessing influences on intervention implementation: revision of the usage rating profile-intervention.

Although treatment acceptability was originally proposed as a critical factor in determining the likelihood that a treatment will be used with integrity, more contemporary findings suggest that whether something is likely to be adopted into routine practice is dependent on the complex interplay among a number of different factors. The Usage Rating Profile-Intervention (URP-I; Chafouleas, Briesch, Riley-Tillman, & McCoach, 2009) was recently developed to assess these additional factors, conceptualized as potentially contributing to the quality of intervention use and maintenance over time. The purpose of the current study was to improve upon the URP-I by expanding and strengthening each of the original four subscales. Participants included 1005 elementary teachers who completed the instrument in response to a vignette depicting a common behavior intervention. Results of exploratory and confirmatory factor analyses, as well as reliability analyses, supported a measure containing 29 items and yielding 6 subscales: Acceptability, Understanding, Feasibility, Family-School Collaboration, System Climate, and System Support. Collectively, these items provide information about potential facilitators and barriers to usage that exist at the level of the individual, intervention, and environment. Information gleaned from the instrument is therefore likely to aid consultants in both the planning and evaluation of intervention efforts.

The association between sexual assault and suicidal activity in a national sample.

Sexual violence is a potential key risk factor for adolescent suicidal behavior but has not been studied extensively. Thus, the current study examined the extent to which sexual assault predicted suicide attempts among adolescent students in the national Youth Risk Behavior Surveillance System survey (2007 data). Gender differences in suicidal behavior overall and among sexual assault victims were examined. The results supported that students with sexual assault histories were significantly more likely (odds ratio [OR]=6.4) to have reported at least one suicide attempt in the past year than students who did not report sexual assault histories. Male students with a sexual assault history reported suicide attempts requiring medical attention more frequently than male attempters without sexual assault histories, as well as both groups of female suicide attempters. Implications of the findings for suicide prevention and intervention programs are discussed.

Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings.

This study examined the impact of various components of rater training on the accuracy of rating behavior using Direct Behavior Rating-Single Item Scales (DBR-SIS). Specifically, the addition of frame-of-reference and rater error training components to a standard package involving an overview and then modeling, practice, and feedback was investigated. In addition, amount of exposure to the direct training component (i.e., number of practice and feedback opportunities) was evaluated, and the rates at which behavior was displayed were carefully manipulated to control for and evaluate training impact by target and rate of behavior. The sample consisted of undergraduate students assigned to one of 6 possible conditions. Overall findings suggested that completion of a training package did result in enhanced accuracy when using DBR-SIS to rate academic engagement and disruption. However, results also supported that the most comprehensive package of DBR training may not always result in greater improvements over a standard package involving direct training. In general, a more intensive training package appeared beneficial at improving ratings for targets that had previously been difficult to rate accurately (e.g., medium rate disruptive behavior). Limitations and implications for future research are discussed.

Direct behavior rating scales as screeners: a preliminary investigation of diagnostic accuracy in elementary school.

This study presents an evaluation of the diagnostic accuracy and concurrent validity of Direct Behavior Rating Single Item Scales for use in school-based behavior screening of second-grade students. Results indicated that each behavior target was a moderately to highly accurate predictor of behavioral risk. Optimal universal screening cut scores were also identified for each scale, with results supporting reduced false positive rates through the simultaneous use of multiple scales.

An investigation of the generalizability and dependability of direct behavior rating single item scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students.

A total of 4 raters, including 2 teachers and 2 research assistants, used Direct Behavior Rating Single Item Scales (DBR-SIS) to measure the academic engagement and disruptive behavior of 7 middle school students across multiple occasions. Generalizability study results for the full model revealed modest to large magnitudes of variance associated with persons (students), occasions of measurement (day), and associated interactions. However, an unexpectedly low proportion of the variance in DBR data was attributable to the facet of rater, as well as a negligible variance component for the facet of rating occasion nested within day (10-min interval within a class period). Results of a reduced model and subsequent decision studies specific to individual rater and rater type (research assistant and teacher) suggested degree of reliability-like estimates differed substantially depending on rater. Overall, findings supported previous recommendations that in the absence of estimates of rater reliability and firm recommendations regarding rater training, ratings obtained from DBR-SIS, and subsequent analyses, be conducted within rater. Additionally, results suggested that when selecting a teacher rater, the person most likely to substantially interact with target students during the specified observation period may be the best choice.