Treatment Integrity

Treatment integrity is a core component of data-based decision making (Detrich, 2013). The usual approach is to consider student data when making decisions about an intervention; however, if there are no data about how well the intervention was implemented, then meaningful judgments cannot be made about effectiveness.

This book is compiled from the proceedings of the sixth summit entitled “Performance Feedback: Using Data to Improve Educator Performance.” The 2011 summit topic was selected to help answer the following question: What basic practice has the potential for the greatest impact on changing the behavior of students, teachers, and school administrative personnel?

This study evaluated the effects of performance feedback to increase the implementation of skills taught during in-service training.

Significant dollars are spent each school year on professional development programs to improve teachers’ effectiveness. This study assessed the integrity with which pre-service teachers used a differential reinforcement of alternate behavior (DRA) strategy taught to them during their student teaching experience.

This study evaulates the effects of performance feedback as part of proffessional development across three studies.

One of the primary goals of implementation science is to insure that programs are implemented with integrity.  This paper presents an integrated model of implementation that emphasizes treatment integrity.

We reviewed parametric analyses of treatment integrity levels published in 10 behavior analytic journals through 2017. We discuss the general findings from the identified literature as well as directions for future research.

Treatment fidelity data (descriptive and statistical) are critical to interpreting and generalizing outcomes of intervention research. Despite recommendations for treatment fidelity reporting from funding agencies and researchers, past syntheses have found treatment fidelity is frequently unreported in educational interventions and fidelity data are seldom used to analyze its relation to student outcomes.

Incidental teaching is often a component of early childhood intervention programs.  This study evaluated the use of grahical feedback to increase the use of incidental teaching.

One of the challenges for increasing treatment integrity is finding effective methods for doing so.  This study evaluated the use of checklist-based training to increase treatment integrity.

This study evaluated the impact of performance feedback on how well problem-solving teams implemeted a structured decision-making protocal.  Teams performed better when feedback was provided.

This study investigated the effects of performance feedback to increase treatment integrity.

This study evaluated the effects of performance feedback in increasing treatment integrity.  It also evaluated the possible reactivitiy effects of being observed.

Reviews of treatment outcome literature indicate treatment integrity is not regularly assessed. In consultation, two levels of treatment integrity (i.e., consultant procedural integrity [CPI] and intervention treatment integrity [ITI]) provide relevant implementation data.

Collecting treatment integrity data is critical for (a) strengthening internal validity within a research study, (b) determining the impact of an intervention on student outcomes, and (c) assessing the need for implementation supports. Although researchers have noted the increased inclusion of treatment integrity data in published articles, there has been limited attention to how treatment integrity is assessed.

This study examines obstacles encountered by 33 educators along with suggested interventions to overcome impediments to effective delivery of classroom management interventions or behavior support plans. Having the right classroom management plan isn’t enough if you can’t deliver the strategies to the students in the classroom.

The study compared basic and elaborated corrections within the context of otherwise identical computer-assisted instruction (CAI) programs that taught reasoning skills. Twelve learning disabled and 16 remedial high school students were randomly assigned to either the basic-corrections or elaborated-corrections treatment. Criterion-referenced test scores were significantly higher for the elaborated-corrections treatment on both the post and maintenance tests and on the transfer test. Time to complete the program did not differ significantly for the two groups.

This study evaluated the effects of video modeling on staff implementation of a problem solving intervention.

Dane and Schneider propose treatment integrity as a multi-dimensional construct and describe five dimensions that constitute the construct.

The authors examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. 

This study evaluated the effects of allowing teachers to “test drive” interventions and then select the intervention they most preferred.  The result was an increase in treatment integrity.

To prevent academic failure and promote long-term success, response-to-intervention (RtI) is designed to systematically increase the intensity of delivering research-based interventions. Interventions within an RtI framework must not only be effective but also be implemented with treatment fidelity and delivered with the appropriate level of treatment intensity to improve student mathematics achievement.

The impact of an intervention is influenced by how well it fis into the context of a classroom.  This paper suggests a number of variables to consider and how they might be measured prior to the development of an intervention.

To produce better outcomes for students two things are necessary: (1) effective, scientifically supported interventions (2) those interventions implemented with high integrity.  Typically, much greater attention has been given to identifying effective practices.  This review focuses on features of high quality implementation.

Presentation by Wing Institute with goals: Make the case that treatment integrity monitoring is a necessary part of service delivery; describe dimensions of treatment integrity; suggest methods for increasing treatment integrity; place treatment integrity within systems framework . 

Over the last fifty years, there have been many educational reform efforts, most of which have had a relatively short lifespan and failed to produce the promised results. One possible reason for this is for the most part these innovations have been poorly implemented. In this chapter, the author proposes a data-based decision making approach to assuring high quality implementation.

Reform efforts tend to come and go very quickly in education. This paper makes the argument that the sustainability of programs is closely related to how well those programs are implemented.

In this conceptualization of treatment integrity, there are four dimensions relevant to practice: (a) exposure (dosage), (b) adherence, (c) quality of delivery, and (d) student responsiveness.  It is important to understand that these dimensions do not stand alone but rather interact to impact the ultimate effectiveness of an intervention. 

The usual approach to determining if an intervention is effective for a student is to review student outcome data; however, this is only part of the task. Student data can only be understood if we know something about how well the intervention was implemented. Student data without treatment integrity data are largely meaningless because without knowing how well an intervention has been implemented, no judgments can be made about the effectiveness of the intervention. Poor outcomes can be a function of an ineffective intervention or poor implementation of the intervention. Without treatment integrity data, there is a risk that an intervention will be judged as ineffective when, in fact, the quality of implementation was so inadequate that it would be unreasonable to expect positive outcomes.

For the best chance of producing positive educational outcomes for all children, two conditions must be met: (a) adopting effective empirically supported (evidence-based) practices and (b) implementing those practices with sufficient quality that they make a difference (treatment integrity)

This study comared the effects of goal setting about student performance and feedback about student performance with daily written feedback about student performance, feedback about accuracy of implementation, and cancelling meetings if integrity criterion was met. 

This study evaluated the impact of allowing  teachers to miss coaching meetings if their treatment integrity scores met or exceeded criterion.

This study evaluated the effects of video modeling on how well teachers implemented interventions.  There was an increase in integrity but it remained variable.  More stable patterns of implementation were observed when teachers were given feedback about their peroformance.

This study evaluated the impact of public feedback in RtI team meetings on the quality of implementation.  Feedback improved poor implementation and maintained high level implementation.

The first purpose of this review is to assess the impact of implementation on program outcomes, and the second purpose is to identify factors affecting the implementation process.

This paper summarizes survey results about the acceptability of different methods for monitoring treatment integrity and performance feedback.

Presents 4 case studies demonstrating an innovative approach for studying and promoting treatment integrity in a manner acceptable to consultees and related to treatment success.

A growing number of evidence-based psychotherapies hold the promise of substantial benefits for children, families, and society.  For the benefits of evidence-based programs to be realized on a scale sufficient to be useful to individuals and society, evidence-based psychotherapies need to be put into practice outside of controlled clinical trials. 

This paper discusses common elements of successfully sustaining effective practices across a variety of disciplines.

This is a comprehensive literature review of the topic of Implementation examining all stages beginning with adoption and ending with sustainability.

Quality indicators of prereferral interventions (i.e., behavioral definition, direct measure, step-by-step plan, treatment integrity, graphing of results, and direct comparison to baseline) were investigated as predictors of prereferral intervention outcomes with a sample of regular education teachers and related services personnel on the same 312 students. 

This paper describes the scaling up and dissemination of a partent training program in Norway while maintaining fidelity of implementation.

This study examined general education teachers’ implementation of a peer tutoring intervention for five elementary students referred for consultation and intervention due to academic concerns. Treatment integrity was assessed via permanent products produced by the intervention.

Organizations house many individuals.  Many of them are responsible implementing the same practice.  If organizations are to meet their goal it is important for the organization have systems for assuring high levels of treatment integrity.

Many students with autism spectrum disorder (ASD) receive behavioral interventions to improve academic and prosocial functioning and remediate current skill deficits. Sufficient treatment integrity is necessary for these interventions to be successful.

Technical issues (specification of treatment components, deviations from treatment protocols and amount of behavior change, and psychometric issues in assessing Treatment Integrity) involved in the measurement of Treatment Integrity are discussed.

The concept of treatment fidelity (integrity) is important across a diversity of fields that are involved with providing treatments or interventions to individuals. Despite variations in terminology across these diverse fields, the concern that treatments or interventions are delivered as prescribed or intended is of paramount importance to document that changes in individuals’ functioning (medical, nutritional, psychological, or behavioral) are due to treatments and not from uncontrolled, extraneous variables.

The concept of treatment integrity is an essential component to data-based decision making within a response-to-intervention model. Although treatment integrity is a topic receiving increased attention in the school-based intervention literature, relatively few studies have been conducted regarding the technical adequacy of treatment integrity assessment methods.

This study reviewed all intervention studies published between 1980-1990 in Journal of Applied Behavior Analysis in which children were the subjects of the study. The authors found that treatment integrity was reported in only 16% of the studies.

The authors reviewed three learning disabilities journals between 1995-1999 to determine what percent of the intervention studies reported measures of treatment integrity. Only 18.5% reported treatment integrity measures.

This review examines the treatment integrity data of literacy interventions for students with emotional and/or behavioral disorders (EBD). Findings indicate that studies focusing on literacy interventions for students with EBD included clear operational definitions and data on treatment integrity to a higher degree than have been found in other disciplines.

This study evaluated the differences in estimates of treatment integrity be measuring different dimensions of it.

The paper describes the Implemenation Planning protocol as an approach for increasing treament integrity.

This study evaluated the relative benefits of verbal feedback and verbal plus grahic feedback as a means for increasing treatment integrity.  The verbal plus graphic feedback was more effective than verbal feedback alone.

Determining whether or not the treatment or innovation under study is actually in use, and if so, how it is being used, is essential to the interpretation of any study. The concept of Levels of Use of the Innovation (LoU) permits an operational, cost-feasible description and documentation of whether or not an innovation or treatment is being implemented.

This study examines adoption and implementation of the US Department of Education's new policy, the `Principles of Effectiveness', from a diffusion of innovations theoretical framework. In this report, we evaluate adoption in relation to Principle 3: the requirement to select research-based programs.

For decades, schools have taught children the strategies of struggling readers, using a theory about reading that cognitive scientists have repeatedly debunked. And many teachers and parents don't know there's anything wrong with it.

This paper suggests a model for selecting interventions that match the context of classrooms.

Validated a measure of clinical supervision practices, further validated a measure of therapist adherence, and examined the association between supervisory practices and therapist adherence to an evidence-based treatment model (i.e., multisystemic therapy [MST]) in real-world clinical settings. 

This article discusses the current focus on using teacher observation instruments as part of new teacher evaluation systems being considered and implemented by states and districts. 

Used a direct observation-based approach to identify behavioral conditions in sending (i.e., special education) and in receiving (i.e., regular education) classrooms and to identify targets for intervention that might facilitate mainstreaming of behavior-disordered (BD) children.

This document presents a set of criteria to be used in evaluating treatment guidelines that have been promulgated by health care organizations, government agencies, professional associations, or other entities.1  The purpose of treatment guidelines is to educate health care professionals2 and health care systems about the most effective treatments available

This piece describes the widely held perception among education leaders that we already  know how to help teachers improve, and that we could achieve our goal of great teaching in far more classrooms if we just applied what we know more widely. 

This study examined the effects of performance feedback on treatment integrity.

The Good Behavior Game (GBG) is a well-documented group contingency designed to reduce disruptive behavior in classroom settings. However, few studies have evaluated the GBG with students who engage in severe problem behavior in alternative schools, and there are few demonstrations of training teachers in those settings to implement the GBG.

This book provides research as well as case studies of successful professional development strategies and practices for educators.

Considers design issues and strategies by comparative outcome studies, including the conceptualization, implementation, and evaluation of alternative treatments; assessment of treatment-specific processes and outcomes; and evaluation of the results. It is argued that addressing these and other issues may increase the yield from comparative outcome studies and may attenuate controversies regarding the adequacy of the demonstrations.

The authors proposed a preliminary FI theory (FIT) and tested it with moderator analyses. The central assumption of FIT is that FIs change the locus of attention among 3 general and hierarchically organized levels of control: task learning, task motivation, and meta-tasks (including self-related) processes.

This book examines the use of video recording to to improve teacher performance. The book shows how every classroom can easily benefit from setting up a camera and hitting “record”.  

Scientists have discredited claims that listening to classical music enhances intelligence, yet this so-called "Mozart Effect" has actually exploded in popularity over the years.

An old disagreement over how to teach children to read -- whole-language versus phonics -- has re-emerged in California, in a new form. Previously confined largely to education, the dispute is now a full-fledged political issue there, and is likely to become one in other states.

This study evaluated the effects of an intensive training progam for paraeducators responsible for implementing a group contingency intervention for classroom behavior.

This study provides descriptive data on the rates of office discipline referrals and beginning reading skills for students in grades K—3 for one school district that is implementing a three-tier prevention model for both reading and behavior support. 

This study provides descriptive data on the rates of office discipline referrals and beginning reading skills for students in grades K—3 for one school district that is implementing a three-tier prevention model for both reading and behavior support. 

We should stop being so embarrassed by uncertainty and embrace it as a strength rather than a weakness of scientific reasoning

This position paper contends that the whole language approach to reading instruction has been disproved by research and evaluation but still pervades textbooks for teachers, instructional materials for classroom use, some states' language-arts standards and other policy documents, teacher licensing requirements and preparation programs, and the professional context in which teachers work. 

Fidelity of treatment in outcome research refers to confirmation that the manipulation of the independent variable occurred as planned. Verification of fidelity is needed to ensure that fair, powerful, and valid comparisons of replicable treatments can be made.

This study evaluated the effects of performance feedback on the implementation of a classroom intervention.

This study investigated treatment fidelity in social work research. The authors systematically reviewed all articles published in five prominent social work journals over a 5- year period.

This study compared the effects of discussing issues of implementation challenges and performance feedback on increasing the integrity of implementation. Performance feedback was more effective than discussion in increasing integrity.

This study contributes to the data-base on the use of performance feedback to increase treatment integrity.

This study examined the impact of three levels of treatment integrity on students'
responding on mathematics tasks.

This study evaluated the impact of training on treatment integrity.  After finding that positive effects lasted 2-4 days, performance feedback was used to increase treatment integrity.

Examined the treatment integrity with which general education teachers implemented a reinforcement based intervention designed to improve the academic performance of elementary school students

This study evaluated three approaches to behavioral consultation and their impact on treatment integrity.  Performance feedback was associated with superior treatment implementation and child behavioral outcomes.

Naomi Oreskes offers a bold and compelling defense of science, revealing why the social character of scientific
knowledge is its greatest strength—and the greatest reason we can trust it.

This study evaluated the effects of a pyramidal training model to improve teachers’ implementation of functional analysis components.  An expert trained a group of teachers who then trained another group of teachers.  All teachers improved their ability to conduct functional analyses.

The purpose of the current investigation was to assess the relationship between the integrity with which social skills interventions were implemented in early childhood special education classrooms and 3 factors: teacher ratings of intervention acceptability, consultative support for implementation, and individual child outcomes.

This study evaluated the effects of performance feedback to pre-service teachers to increase their rates of positive and negative communication with students.

The focus of the book is on essential concepts in educational statistics, understanding when to use various statistical tests, and how to interpret results. This book introduces educational students and practitioners to the use of statistics in education and basic concepts in statistics are explained in clear language. 

This study evaluted the impact of coaching on the implementation of an intervention.  Coaching with higher rates of performance feedback resulted in the highest level of treatment integrity.

This page describes the conflict of interest and what should we do about it.

An alternating treatments design was used to evaluate the effectiveness of an educational program that combined timings (via chess clocks), peer tutoring (i.e., peer-delivered immediate feedback), positive-practice overcorrection, and performance feedback on mathematics fluency (i.e., speed of accurate responding) in four elementary students with mathematics skills deficits.

It is proposed in this paper that interventions are most likely to be implemented when they draw from existing practices in a classroom.

First Step to Success is an empirically supported intervention for young elementary school students that can be implemented by public school teachers with training. This study evaluated the effects of coaching feedback on teachers who not effectively implemented First Step following training.

This book looks at how new ideas spread via communication channels over time. Such innovations are initially perceived as uncertain and even risky. To overcome this uncertainty, most people seek out others like themselves who have already adopted the new idea. Thus the diffusion process typically takes months or years. But there are exceptions: use of the Internet in the 1990s, for example, may have spread more rapidly than any other innovation in the history of humankind. 

This review will briefly address the nature of conflicts of interest in research, including the importance of both financial and non-financial conflicts, and the potential effectiveness and limits of various strategies for managing such conflicts.

The purpose of this chapter is to explain the role of treatment integrity assessment within the “implementing solutions” stage of a problem-solving model.

This paper describes a multi-tier system of supports for teachers as they implement an intervention.

This study evaluated the Treatment Integiry Planning Protocol as a means for increasing treatment integrity.

This paper describes treatment integrity assessment and intervention for practicing school psychologists.

This paper reviews options for treatment integrity measurement emphasizing how direct behavior rating technology might be incorporated within a multi-tiered model of intervention delivery.

This study evaluated four methods for teachers self reporting how well they implemented an intervention.

Both student outcomes and treatment fidelity data are necessary to draw valid conclusions about intervention effectiveness. Reviews of the intervention outcome literature in related fields, and prior reviews of the school psychology literature, suggest that many researchers failed to report treatment fidelity data.

This paper evaluated the impact of Implementation Planning on teacher level of treatment integrity.

The authors reviewed all intervention studies published in the Journal of Positive Behavior Interventions between 1999-2009 to determine the percent of those studies that reported a measure of treatment integrity. Slightly more than 40% reported a measure of treatment integrity.

If educational programs are to be effective they must be implemented with sufficient integrity to assure benefits. To have a significant impact on schools, solutions must be scalable. This study evaluated the effects of using existing school personnel to provide performance feedback to teachers regarding the quality of implementation.

The authors reviewed four school psychology journals between 1995-2008 to estimate the percent of intervention studies that reported some measure of treatment integrity. About 50% reported a measure of treatment integrity.

This study evaluated the relative benefits of verbal feedback and verbal plus grahic feedback as a means for increasing treatment integrity.  The verbal plus graphic feedback was more effective than verbal feedback alone.

This study utilized a “bug in the ear” device to provide immediate feedback on implementation of specific teaching practices.

This paper describes the use of wireless technology to give feedback to students in a teacher prep program about their integrity of implementation.

A commonly used research design in applied behavior analysis involves comparing two or more independent variables. There were some consistencies across studies, with half resulting in equivalent outcomes across comparisons. In addition, most studies employed the use of an alternating treatments or multi-element single-subject design and compared a teaching methodology. 

This paper outlines the best practices for researchers and practitioners translating research to practice as well as recommendations for improving the process.

The effects of two error-correction procedures on oral reading errors and a control condition were compared in an alternating treatments design with three students who were moderately mentally retarded. The two procedures evaluated were word supply and sentence repeat.

This literature review was conducted to evaluate the current state of evidence supporting communication interventions for individuals with severe intellectual and developmental disabilities. Many researchers failed to report treatment fidelity or to assess basic aspects of intervention effects, including generalization, maintenance, and social validity.

The current study extracted and aggregated data from single-case studies that used Performance feedback (PF) in school settings to increase teachers' use of classroom-based interventions.

This study is a meta-analysis of studies using performance feedback to improve treatment integrity.  The overall result was that performance feedback had moderate effects on integrity.

Observational measurement of treatment adherence has long been considered the gold standard. However, little is known about either the generalizability of the scores from extant observational instruments or the sampling needed. Results suggested that reliable cognitive–behavioral therapy adherence studies require at least 10 sessions per patient, assuming 12 patients per therapists and two coders—a challenging threshold even in well-funded research. Implications, including the importance of evaluating alternatives to observational measurement, are discussed.

This book is written for school administrators, staff developers, behavior specialists, and instructional coaches to offer guidance in implementing research-based practices that establish effective classroom management in schools. The book provides administrators with practical strategies to maximize the impact of professional development. 

This paper examines a range of education failures: common mistakes in how new practices are selected, implemented, and monitored. The goal is not a comprehensive listing of all education failures but rather to provide education stakeholders with an understanding of the importance of vigilance when implementing new practices.

Inattention to treatment integrity is a primary factor of failure during implementation. Treatment integrity is defined as the extent to which an intervention is executed as designed, and the accuracy and consistency with which the intervention is implemented 

The debate about the safety of calcium-channel antagonists provided an opportunity to study financial conflicts of interest in medicine. This project was designed to examine the relation between authors' published positions on the safety of calcium-channel antagonists and their financial interactions with the pharmaceutical industry.

This study compared indirect training and direct training methods as a means of impacting levels of treatment integrity.  Direct training methods produced better outcomes.

The present study was conducted to investigate the relationship between training procedures and treatment integrity. 

This is a systematic review of the  effects of coaching teachers to implement social behavior interventions.

This paper describes the problem of publication bias with reference to its history in a number of fields, with special reference to the area of educational research.

This experiment evaluated the effects of requiring overt answer construction in computer-based programmed instruction using an alternating treatments design.

Using the data from the Program for the International Assessment of Adult Competencies (PIAAC), this Data Point summarizes the number of U.S. adults with low levels of English literacy and describes how they differ by nativity status¹ and race/ethnicity.

The National Assessment of Educational Progress (NAEP) is the largest nationally representative and continuing assessment of what America's students know and can do in various subject areas.

Simple experimental studies randomize study participants into two groups: a treatment group that includes participants who receive the offer to participate in a program or intervention, and a control group that includes participants who do not receive that offer. Such studies primarily address questions about the program impacts on the average outcomes of participants.

A review of 20 experimental, shared book reading (SBR) interventions using questioning strategies with preschool children was conducted. The studies were analyzed in terms of their quality, focus, and the questioning strategies employed. Although there were few methodological concerns about the studies conducted, treatment fidelity and replicability of the reported interventions are raised as issues needing attention in future research.

The purpose of this study was to assess the degree to which behavioral intervention studies conducted with persons with mental retardation operationally defined the independent variables and evaluated and reported measures of treatment integrity. The study expands the previous work in this area reported by Gresham, Gansle, and Noell (1993) and Wheeler, Baggett, Fox, and Blevins (2006) by providing an evaluation of empirical investigations published in multiple journals in the fields of applied behavior analysis and mental retardation from 1996–2006. Results of the review indicated that relatively few of the studies fully reported data on treatment integrity.

The relationships among independent variables and three measures of treatment integrity were evaluated.

Treatment integrity is essential for the implementation of interventions in schools as it determines the accuracy or consistency with which different components of a treatment are implemented. There are no current standards regarding the best practices in treatment integrity measurement; however, higher integrity is associated with enhanced student outcomes.

This study evaluated the effects of performance feedback on increasing the quality of implementation of interventions by teachers in a public school setting.

This study evaluated the effects of graphed feedback alone compared to the effects of graphed feedback plus verbal feedback.  The combined graphed and verbal resulted in slightly better performance.