Methods for Population Research on Substance Use and Consequences.

This chapter reviews the array of methods used in contemporary research on population-level research on substance use and its consequences. We argue that there are critical questions that can best - or in some cases, only - be addressed at the level of a population. We then describe the major categories of data collection methods used in population research, including surveys, ecological momentary assessment, administrative data, audit methods, and unobtrusive assessment of substance use. Two categories of measures are then discussed: measures of an individual's use of substances and related problems and measures of harm to others caused by one's use. We then review factors that may be considered causes or correlates of substance use and consequences, including both individual and environmental factors. We close with a few thoughts on the accumulation of knowledge and its translation to policy and practice.

Differences in observers' and teachers' fidelity assessments.

As evidence-based programs become disseminated, understanding the degree to which they are implemented with fidelity is crucial. This study tested the validity of fidelity ratings made by observers versus those made by teachers. We hypothesized that teachers' reports about fidelity would have a positivity bias when compared to observers' reports. Further, we hypothesized that there would generally be low correspondence between teachers' and observers' ratings of fidelity. We examined teachers' and observers' ratings as they were related to mediating variables targeted for change by the intervention. Finally, we examined the role that years of teaching experience played in achieving fidelity. Eighteen teachers and four research assistants participated in this project as raters. Teachers made video recordings of their implementation of All Stars and completed fidelity assessment forms. Trained observers independently completed parallel forms for 215 sampled classroom sessions. Both teachers and observers rated adherence, quality of delivery, attendance, and participant engagement. Teachers made more positive fidelity ratings than did observers. With the exception of ratings for attendance, teachers and observers failed to agree on fidelity ratings. Observers' ratings were significantly related to students' pretest assessments of targeted program mediators. That observers' ratings were related to students' pretest scores, suggests it is easier to teach well when students are predisposed to program success. Teachers' ratings were infrequently related to mediators, but when they were, the relationship was counterintuitive. Experienced teachers taught with greater fidelity than novice teachers. While possibly inflated and inaccurate, gathering fidelity assessments from teachers may sensitize them to issues of fidelity as a result of requiring form completion. Assessing fidelity through observers' ratings of video recordings has significant merit. As a longterm investment in improving prevention outcomes, policy makers should consider requiring both teacher and observer fidelity assessments as essential components of evaluation.

Styles of Adaptation: The Impact of Frequency and Valence of Adaptation on Preventing Substance Use.

Purpose: To be effective, evidence-based programs should be delivered as prescribed. This suggests that adaptations that deviate from intervention goals may limit a program's effectiveness. This study examines the impact that number and quality of adaptations have on substance use outcomes. Design: We examined 306 video recordings of teachers delivering 'All Stars', a middle school drug prevention program. Multiple observers coded each recording, noting the number and type of adaptation each teacher made. Each adaptation was given a valence rating. Adaptations that were deleterious to program goals received negative valence ratings; positive ratings were given for adaptations that were likely to facilitate achievement of program goals; neutral ratings were given to adaptations that were expected to have neither a positive nor negative impact on program goals. Findings: All teachers made adaptations. Teachers were consistent across time in the types of adaptations they made, suggesting each teacher has a personalized style of adapting. Those who made few adaptations, and whose average adaptation was rated as being positive had a higher percentage of students who remained non-drug users. In contrast, teachers who made many adaptations, whether their average valence rating was positive, neutral or negative, failed to have as many students remain non-drug users. Measures of fidelity, including quality of delivery and teacher understanding were related to valence of adaptations, with better performance related to making positive adaptations. Practical Implications: Through training and supervision, teachers should be guided and encouraged to follow programs directions, making few adaptations and ensuring that adaptations that are made advance the goals of intervention. Programs should define acceptable and unacceptable ways they may be adapted. Value: This study provides significant evidence about the challenges that face disseminated evidence-based programs.

Engaging Youth in Substance Misuse Prevention within State Prevention Systems: Provider Perspectives.

Objective: Engaging youth in planning, developing, and implementing substance misuse prevention efforts can improve those efforts. However, specific local policies and systems constrain youth engagement practice. This study examines how to engage youth in substance misuse prevention within state prevention systems. Methods: In the qualitative study, semi-structured interviews with 13 prevention providers in the North Carolina prevention system were conducted via video call and transcribed and analyzed via thematic and codebook approaches. Procedures are reported according to the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist. Results: Providers viewed youth engagement favorably yet mostly did not differentiate between youth-targeted prevention initiatives and youth-engaged initiatives. Findings reveal how state-level decisions, such as definitions and funding policies, support and challenge how providers implement youth engagement. Conclusions: To encourage youth engagement, state prevention systems should clarify for providers the distinction between prevention efforts targeted to youth and those that engage youth in planning and implementing and define which activities are desirable and what the role of youth should be in the activities. Finally, prevention providers need support for youth-engaged approaches to prevention, specifically training for youth and adult allies and opportunities for knowledge-sharing.

Changing course: supporting a shift to environmental strategies in a state prevention system.

BACKGROUND: This study examines how the North Carolina state prevention system responded to a policy shift from individual-level prevention strategies to environmental strategies from the perspective of the organizations implementing the policy shift. METHODS: We use two data sources. First, we conducted interviews to collect qualitative data from key informants. Second, we used prevention provider agency expenditure data from the year the shift was announced and the following year. RESULTS: The interviews allowed us to identify effective features of policy change implementation in complex systems, such as the need for clear communication and guidance about the policy changes. Our interview and expenditure analyses also underscore variation in the level of guidance and oversight provided by implementing agencies to prevention providers. CONCLUSIONS: Our analyses suggest that more active monitoring and oversight may have facilitated more consistent implementation of the policy shift toward greater use of environmental prevention strategies.

Three-year trajectory of teachers' fidelity to a drug prevention curriculum.

Little is known about the trajectories over time of classroom teachers' fidelity to drug prevention curricula. Using the "Concerns-Based Adoption Model" (C-BAM) as a theoretical framework, we hypothesized that teachers' fidelity would improve with repetition. Participants comprised 23 middle school teachers who videotaped their administration of three entire iterations of the All Stars curriculum. Investigators coded two key curriculum lessons, specifically assessing the proportion of activities of each lesson teachers attempted and whether they omitted, added, or changed prescribed content, or delivered it using new methods. Study findings provided only partial support for the C-BAM model. Considerable variability in teachers' performance over time was noted, suggesting that their progression over time may be nonlinear and dynamic, and quite possibly a function of their classroom and school contexts. There was also evidence that, by their third iteration of All Stars, teachers tended to regress toward the baseline mean. That is, the implementation quality of those that started out with high levels of fidelity tended to degrade, while those that started out with very low fidelity to the curriculum tended to improve. Study findings suggest the need for ongoing training and technical assistance, as well as "just in time" messages delivered electronically; but it is also possible that some prevention curricula may impose unrealistic expectations or burdens on teachers' abilities and classroom time.

Implementing evidence-based substance use prevention curricula in North Carolina public school districts.

The Safe and Drug-Free Schools and Communities Act (SDFSCA) provides funding for prevention education to nearly every school district in the nation. Recent federal policy requires SDFSCA recipients to implement evidence-based prevention programs. This paper reports the extent to which North Carolina public school districts implement evidence-based substance use prevention curricula. Results showed that while the majority of school districts use evidence-based prevention curricula, they are rarely the most commonly used curricula. Evidence-based curricula are much more likely to be used at the middle school level than at the elementary or high school levels. Urbanicity, coordinator time, and coordinator experience correlated with extensive use of evidence-based curricula in the bivariate analysis, but only time spent on prevention by the Safe and Drug-Free Schools (SDFS) coordinator significantly predicted extensive use in the multivariate analysis. Increasing district SDFSCA coordinator time is a necessary step for diffusing evidence-based curricula.

Measuring Fidelity and Adaptation: Reliability of a Instrument for School-Based Prevention Programs.

There is a need to standardize methods for assessing fidelity and adaptation. Such standardization would allow program implementation to be examined in a manner that will be useful for understanding the moderating role of fidelity in dissemination research. This article describes a method for collecting data about fidelity of implementation for school-based prevention programs, including measures of adherence, quality of delivery, dosage, participant engagement, and adaptation. We report about the reliability of these methods when applied by four observers who coded video recordings of teachers delivering All Stars, a middle school drug prevention program. Interrater agreement for scaled items was assessed for an instrument designed to evaluate program fidelity. Results indicated sound interrater reliability for items assessing adherence, dosage, quality of teaching, teacher understanding of concepts, and program adaptations. The interrater reliability for items assessing potential program effectiveness, classroom management, achievement of activity objectives, and adaptation valences was improved by dichotomizing the response options for these items. The item that assessed student engagement demonstrated only modest interrater reliability and was not improved through dichotomization. Several coder pairs were discordant on items that overall demonstrated good interrater reliability. Proposed modifications to the coding manual and protocol are discussed.

Promoting quality of program delivery via an internet message delivery system.

This article presents results from a study that evaluated an online message system designed to improve the delivery of prevention programs. We conducted a quasi-experimental study with 32 agencies and schools that implemented substance use prevention programs and examined differences between the comparison and intervention groups. We also examined the impact of dosage of the message system by comparing results among three groups of teachers: non-users, low users, and high users. Results for norm setting were marginally significant, such that teachers within the agencies assigned to the intervention condition scored higher on their understanding of norm setting at posttest compared to teachers within comparison agencies, after controlling for pretest knowledge scores and demographic items. In the model examining impact of dosage, high users of the intervention scored significantly higher on self-reported understanding of their program, quality of delivery, and program effectiveness compared to non-users. Low users of the intervention reported significantly higher quality of delivery compared to non-users.

The role of teacher communicator style in the delivery of a middle school substance use prevention program.

We examine whether teachers' communicator style relates to student engagement, teacher-student relationships, student perceptions of teacher immediacy, as well as observer ratings of delivery skills during the implementation of All Stars, a middle school-based substance use prevention program. Data from 48 teachers who taught All Stars up to 3 consecutive years and their respective seventh-grade students (n = 2,240) indicate that having an authoritative communication style is negatively related to student engagement with the curriculum and the quality of the student-teacher relationship, while having an expressive communicator style improves teachers' immediacy to student needs. Adaptations made by a subsample of teachers (n = 27) reveal that those who were more expressive asked students more questions, used more motivational techniques, and introduced more new concepts than authoritarian teachers.

Coaching to Enhance Quality of Implementation in Prevention.

PURPOSE: This study describes topics covered by coaches assisting teachers implementing a research-based drug prevention program and explores how coaching affected student outcomes. DESIGN: The All Stars drug prevention curriculum was implemented by 16 urban teachers who received four coaching sessions. Two coaches participated. Coaches were interviewed by investigators to assess topics covered. Students completed pretest-posttest measures of mediators and substance use behaviors. FINDINGS: The average teacher was coached on 11.7 different topics, out of a total of 23 topics. Coaching topics most heavily emphasized included: introduction and wrap up; time management; general classroom management; teacher's movement around the class; asking open-ended questions; using students' questions, comments and examples to make desired points; general preparation; engaging high-risk youth; reading from the curriculum; implementing activities correctly; focusing on objectives and goals; maintaining a focus on the task; and improving depth of understanding. Seven coaching topics were found to relate to changes in student mediators and behavior. RESEARCH LIMITATIONS/IMPLICATIONS: The current study was exploratory. Future research should explore how teachers develop the particular skills required by prevention programs and how coaches can assist them. PRACTICAL IMPLICATIONS: We postulate five levels of skill development which coaches may address: (1) fundamental teaching skills, (2) mechanics of program delivery, (3) development of an interactive teaching style, (4) effective response to student input, and (5) effective tailoring and adaptation. ORIGINALITY/VALUE: This represents one of a very few studies that explores how coaching impacts outcomes in substance abuse prevention.

Teachers' delivery skills and substance use prevention program outcomes: the moderating role of students' need for cognition and impulse decision making.

The purpose of this study was to extend the literature in both substance use implementation and persuasive health communication by examining the extent to which students' need for cognition and impulsive decision-making moderated the relationship between teachers' classroom communication behavior and program outcomes in an evidence-based middle school substance use prevention curriculum. Participants included 48 teachers and their respective 7th grade students who participated in a randomized trial testing the effectiveness of personal coaching as a means to improve the quality with which teachers implemented the All Stars curriculum. Need for cognition and impulse decision-making were both associated with positive changes in lifestyle incongruence and commitments to not use substances for students whose teachers displayed greater interactive teaching. Further, need for cognition was associated with lower alcohol use rates while impulse decision making related to lower rates of marijuana use in classes with interactive teaching.

The effects of students' curriculum engagement, attitudes toward their teachers, and perception of their teachers' skills on school-based prevention curriculum outcomes.

We examined the association between changes in the substances and mediating variables targeted by the All Stars drug prevention curriculum, and students' engagement in and enjoyment of the curriculum, their attitudes toward their teachers, and their perceptions of their teachers' skills. Forty-eight school staff administered at least one All Stars class, for up to three consecutive years, to their seventh grade students in 107 classes in a large Midwestern school district. A sample of 2428 students completed a linked pretest and post-test, for a response rate of 91%. We found that students' engagement in and enjoyment of the curriculum, their attitudes toward their teachers, and their perceptions of their teachers' skill were all associated with positive changes in the curriculum's five mediators, but not with changes in students' substance use per se. Study findings suggest the importance of these three attributes to the achievement of the objectives of prevention curricula.

The potential of coaching as a strategy to improve the effectiveness of school-based substance use prevention curricula.

Research-based substance use prevention curricula typically yield small effects when implemented by school teachers under real-world conditions. Using a randomized controlled trial, the authors examined whether expert coaching improves the effectiveness of the All Stars prevention curriculum. Although a positive effect on students' cigarette use was noted, this finding may be attributed to marked baseline differences on this variable across the intervention and control groups. No effects were found on students' alcohol or marijuana use or on any of several variables thought to mediate curriculum effects. The effects of coaching on teachers may not become evident until future years, when they have moved beyond an initial mechanical delivery of the curriculum.

Measuring quality of delivery in a substance use prevention program.

The purpose of this study was to develop and validate an observation measure designed to capture teachers' use of interactive teaching skills within the delivery of the All Stars substance use prevention program. Coders counted the number of times teachers praised and encouraged students, accepted and used students' ideas, asked questions, self-disclosed personal anecdotes, and corrected student misbehavior. These teacher behaviors loaded on three factors: classroom management, acknowledgment, and student-centered methods. Classroom management was negatively related to student engagement. Acknowledgment was negatively related to students' normative beliefs. Student-centered methods were positively related to student idealism and normative beliefs, and marginally predicted decreases in student marijuana use. Editors' Strategic Implications: The authors provide a promising approach to studying pedagogical prevention approaches, and they also link teaching processes to student outcomes. This study of program delivery should be of general interest (i.e., not limited to substance use prevention) to practitioners and researchers.

Does federal policy support the use of scientific evidence in school-based prevention programs?

Since 1998, federal policy has explicitly required the use of "evidence-based" prevention programs in schools. We review how this policy has been implemented through state recipients of the Safe and Drug Free Schools (SDFS) Program, and how other federal and private agencies have supported the policy by providing guidance about the scientific evidence for specific programs' effectiveness. We report data from a survey of SDFS state office directors, and we compare and contrast the most popular lists of effective programs. State offices supply the infrastructure for administering the SDFS Program, providing technical assistance to local school districts, monitoring the implementation of federal policy at the local level, and determining funding eligibility based on compliance. We found that states rely heavily on federal lists to determine whether school districts are meeting federal policy requirements, particularly the National Registry of Effective Programs and Practices (NREPP). Both SDFS and NREPP are changing, however, and the changes do not bode well for the transfer of prevention science to schools. Conclusions and recommendations are presented.

Implementation fidelity in a teacher-led alcohol use prevention curriculum.

There is now ample evidence that teachers tend to make substantial modifications to both the prescribed content and methods of the curricula they administer, and that such modifications are likely to attenuate curricula effects. We examine the fidelity with which teachers implement "Protecting You, Protecting Me," an underage alcohol use prevention curriculum. Findings suggest that while teachers attempted to implement most sections of a lesson, the lessons taught were consistently--and often extensively--adapted. We conclude that since teachers are likely to continue to modify lessons, curriculum developers and trainers should enhance their understanding of how prevention curricula are taught under real world conditions, help teachers to reinforce key curriculum concepts, and consider modifying those curricular sections that teachers are adapting with greatest frequency.