Effective Instruction Treatment Integrity

This book is compiled from the proceedings of the sixth summit entitled “Performance Feedback: Using Data to Improve Educator Performance.” The 2011 summit topic was selected to help answer the following question: What basic practice has the potential for the greatest impact on changing the behavior of students, teachers, and school administrative personnel?

This Guide seeks to provide assistance to educational practitioners in evaluating whether an educational intervention is backed by rigorous evidence of effectiveness, and in implementing evidence-based interventions in their schools or classrooms.

This policy brief lays out five components of a vision for the future and identifies opportunities to support teacher education reform. Examples of promising developments are also addressed that involve full-scale program redesign featuring collaboration across general and special education.

The purpose of this study was to examine the effectiveness of enhanced anchor-instruction and traditional problem instruction in improving problem-solving performance. 

Dane and Schneider propose treatment integrity as a multi-dimensional construct and describe five dimensions that constitute the construct.

The impact of an intervention is influenced by how well it fis into the context of a classroom.  This paper suggests a number of variables to consider and how they might be measured prior to the development of an intervention.

To produce better outcomes for students two things are necessary: (1) effective, scientifically supported interventions (2) those interventions implemented with high integrity.  Typically, much greater attention has been given to identifying effective practices.  This review focuses on features of high quality implementation.

Presentation by Wing Institute with goals: Make the case that treatment integrity monitoring is a necessary part of service delivery; describe dimensions of treatment integrity; suggest methods for increasing treatment integrity; place treatment integrity within systems framework . 

Over the last fifty years, there have been many educational reform efforts, most of which have had a relatively short lifespan and failed to produce the promised results. One possible reason for this is for the most part these innovations have been poorly implemented. In this chapter, the author proposes a data-based decision making approach to assuring high quality implementation.

Reform efforts tend to come and go very quickly in education. This paper makes the argument that the sustainability of programs is closely related to how well those programs are implemented.

In this conceptualization of treatment integrity, there are four dimensions relevant to practice: (a) exposure (dosage), (b) adherence, (c) quality of delivery, and (d) student responsiveness.  It is important to understand that these dimensions do not stand alone but rather interact to impact the ultimate effectiveness of an intervention. 

The usual approach to determining if an intervention is effective for a student is to review student outcome data; however, this is only part of the task. Student data can only be understood if we know something about how well the intervention was implemented. Student data without treatment integrity data are largely meaningless because without knowing how well an intervention has been implemented, no judgments can be made about the effectiveness of the intervention. Poor outcomes can be a function of an ineffective intervention or poor implementation of the intervention. Without treatment integrity data, there is a risk that an intervention will be judged as ineffective when, in fact, the quality of implementation was so inadequate that it would be unreasonable to expect positive outcomes.

For the best chance of producing positive educational outcomes for all children, two conditions must be met: (a) adopting effective empirically supported (evidence-based) practices and (b) implementing those practices with sufficient quality that they make a difference (treatment integrity)

This study evaluated the impact of public feedback in RtI team meetings on the quality of implementation.  Feedback improved poor implementation and maintained high level implementation.

The first purpose of this review is to assess the impact of implementation on program outcomes, and the second purpose is to identify factors affecting the implementation process.

This paper summarizes survey results about the acceptability of different methods for monitoring treatment integrity and performance feedback.

Technical issues (specification of treatment components, deviations from treatment protocols and amount of behavior change, and psychometric issues in assessing Treatment Integrity) involved in the measurement of Treatment Integrity are discussed.

This study evaluated the differences in estimates of treatment integrity be measuring different dimensions of it.

Used a direct observation-based approach to identify behavioral conditions in sending (i.e., special education) and in receiving (i.e., regular education) classrooms and to identify targets for intervention that might facilitate mainstreaming of behavior-disordered (BD) children.

This book provides research as well as case studies of successful professional development strategies and practices for educators.

The authors proposed a preliminary FI theory (FIT) and tested it with moderator analyses. The central assumption of FIT is that FIs change the locus of attention among 3 general and hierarchically organized levels of control: task learning, task motivation, and meta-tasks (including self-related) processes.

This book examines the use of video recording to to improve teacher performance. The book shows how every classroom can easily benefit from setting up a camera and hitting “record”.  

Peer assessment has become a popular education intervention. A review of the literature finds few studies on the impact of Peer Review on student outcomes. This meta-analysis examines the effect sizes found in 58 studies. 

The purpose of this chapter is to describe a critical component of the response-to-
intervention (RTI) process: monitoring student response to general education instruction.

This meta-analysis reports on the overall effectiveness of video analysis when used with special educators, as well as on moderator analyses related to participant and instructional characteristics.

This study compared the effects of discussing issues of implementation challenges and performance feedback on increasing the integrity of implementation. Performance feedback was more effective than discussion in increasing integrity.

This book looks at how new ideas spread via communication channels over time. Such innovations are initially perceived as uncertain and even risky. To overcome this uncertainty, most people seek out others like themselves who have already adopted the new idea. Thus the diffusion process typically takes months or years. But there are exceptions: use of the Internet in the 1990s, for example, may have spread more rapidly than any other innovation in the history of humankind. 

This paper reviews options for treatment integrity measurement emphasizing how direct behavior rating technology might be incorporated within a multi-tiered model of intervention delivery.

The authors reviewed all intervention studies published in the Journal of Positive Behavior Interventions between 1999-2009 to determine the percent of those studies that reported a measure of treatment integrity. Slightly more than 40% reported a measure of treatment integrity.

The authors reviewed four school psychology journals between 1995-2008 to estimate the percent of intervention studies that reported some measure of treatment integrity. About 50% reported a measure of treatment integrity.

A multiple baseline across items design was used to evaluate the effects of a cover, copy, and compare (CCC) intervention on students' accuracy in identifying states on a map of the United States. The results showed the CCC intervention was effective in increasing the class mean accuracy levels in locating states.

This book is written for school administrators, staff developers, behavior specialists, and instructional coaches to offer guidance in implementing research-based practices that establish effective classroom management in schools. The book provides administrators with practical strategies to maximize the impact of professional development. 

This paper examines a range of education failures: common mistakes in how new practices are selected, implemented, and monitored. The goal is not a comprehensive listing of all education failures but rather to provide education stakeholders with an understanding of the importance of vigilance when implementing new practices.

Inattention to treatment integrity is a primary factor of failure during implementation. Treatment integrity is defined as the extent to which an intervention is executed as designed, and the accuracy and consistency with which the intervention is implemented 

A summary of the available studies accumulated over the past 40 years on a key education driver, teacher competencies offers practical strategies, practices, and rules to guide teachers in ways to improve instruction that improves student performance and the quality of the work experience.

The present study was conducted to investigate the relationship between training procedures and treatment integrity. 

A soon to be published meta-analysis of Direct Instruction (DI) curricula that reviews research on DI curricula between 1966-2016 reports that DI curricula produced moderate to large effect sizes across the curriculum areas reading, math, language, and spelling.  The review is notable because it reviews a much larger body of DI research than has occurred in the past and covers a wide range of experimental designs (from single subject to randomized trials).  328 studies were reviewed and almost 4,000 effects were considered.  Given the variability in research designs and the breadth of the effects considered, it suggests that DI curricula produce robust results.  There was very little decline during maintenance phases of the study and greater exposure to the curricula resulted in greater effects.

This study evaluated the effects of performance feedback on increasing the quality of implementation of interventions by teachers in a public school setting.

More than two-thirds of students living in U.S. low-income urban areas have not demonstrated basic levels of math achievement. Teachers are confronted with a difficult task of meeting the needs of an increasingly academically diverse population of urban students. There is a well-confirmed knowledge base on effective instruction, but teachers need massive amounts of information for effective, sustainable improvement and data-driven decision making.