Treatment Integrity Dimensions

Treatment integrity is a core component of data-based decision making (Detrich, 2013). The usual approach is to consider student data when making decisions about an intervention; however, if there are no data about how well the intervention was implemented, then meaningful judgments cannot be made about effectiveness.

This book is compiled from the proceedings of the sixth summit entitled “Performance Feedback: Using Data to Improve Educator Performance.” The 2011 summit topic was selected to help answer the following question: What basic practice has the potential for the greatest impact on changing the behavior of students, teachers, and school administrative personnel?

We reviewed parametric analyses of treatment integrity levels published in 10 behavior analytic journals through 2017. We discuss the general findings from the identified literature as well as directions for future research.

Treatment fidelity data (descriptive and statistical) are critical to interpreting and generalizing outcomes of intervention research. Despite recommendations for treatment fidelity reporting from funding agencies and researchers, past syntheses have found treatment fidelity is frequently unreported in educational interventions and fidelity data are seldom used to analyze its relation to student outcomes.

Collecting treatment integrity data is critical for (a) strengthening internal validity within a research study, (b) determining the impact of an intervention on student outcomes, and (c) assessing the need for implementation supports. Although researchers have noted the increased inclusion of treatment integrity data in published articles, there has been limited attention to how treatment integrity is assessed.

Dane and Schneider propose treatment integrity as a multi-dimensional construct and describe five dimensions that constitute the construct.

To prevent academic failure and promote long-term success, response-to-intervention (RtI) is designed to systematically increase the intensity of delivering research-based interventions. Interventions within an RtI framework must not only be effective but also be implemented with treatment fidelity and delivered with the appropriate level of treatment intensity to improve student mathematics achievement.

The impact of an intervention is influenced by how well it fis into the context of a classroom.  This paper suggests a number of variables to consider and how they might be measured prior to the development of an intervention.

To produce better outcomes for students two things are necessary: (1) effective, scientifically supported interventions (2) those interventions implemented with high integrity.  Typically, much greater attention has been given to identifying effective practices.  This review focuses on features of high quality implementation.

Presentation by Wing Institute with goals: Make the case that treatment integrity monitoring is a necessary part of service delivery; describe dimensions of treatment integrity; suggest methods for increasing treatment integrity; place treatment integrity within systems framework . 

Over the last fifty years, there have been many educational reform efforts, most of which have had a relatively short lifespan and failed to produce the promised results. One possible reason for this is for the most part these innovations have been poorly implemented. In this chapter, the author proposes a data-based decision making approach to assuring high quality implementation.

In this conceptualization of treatment integrity, there are four dimensions relevant to practice: (a) exposure (dosage), (b) adherence, (c) quality of delivery, and (d) student responsiveness.  It is important to understand that these dimensions do not stand alone but rather interact to impact the ultimate effectiveness of an intervention. 

The usual approach to determining if an intervention is effective for a student is to review student outcome data; however, this is only part of the task. Student data can only be understood if we know something about how well the intervention was implemented. Student data without treatment integrity data are largely meaningless because without knowing how well an intervention has been implemented, no judgments can be made about the effectiveness of the intervention. Poor outcomes can be a function of an ineffective intervention or poor implementation of the intervention. Without treatment integrity data, there is a risk that an intervention will be judged as ineffective when, in fact, the quality of implementation was so inadequate that it would be unreasonable to expect positive outcomes.

For the best chance of producing positive educational outcomes for all children, two conditions must be met: (a) adopting effective empirically supported (evidence-based) practices and (b) implementing those practices with sufficient quality that they make a difference (treatment integrity)

This study evaluated the impact of public feedback in RtI team meetings on the quality of implementation.  Feedback improved poor implementation and maintained high level implementation.

The first purpose of this review is to assess the impact of implementation on program outcomes, and the second purpose is to identify factors affecting the implementation process.

This paper summarizes survey results about the acceptability of different methods for monitoring treatment integrity and performance feedback.

This paper discusses common elements of successfully sustaining effective practices across a variety of disciplines.

This is a comprehensive literature review of the topic of Implementation examining all stages beginning with adoption and ending with sustainability.

Many students with autism spectrum disorder (ASD) receive behavioral interventions to improve academic and prosocial functioning and remediate current skill deficits. Sufficient treatment integrity is necessary for these interventions to be successful.

Technical issues (specification of treatment components, deviations from treatment protocols and amount of behavior change, and psychometric issues in assessing Treatment Integrity) involved in the measurement of Treatment Integrity are discussed.

The concept of treatment fidelity (integrity) is important across a diversity of fields that are involved with providing treatments or interventions to individuals. Despite variations in terminology across these diverse fields, the concern that treatments or interventions are delivered as prescribed or intended is of paramount importance to document that changes in individuals’ functioning (medical, nutritional, psychological, or behavioral) are due to treatments and not from uncontrolled, extraneous variables.

The concept of treatment integrity is an essential component to data-based decision making within a response-to-intervention model. Although treatment integrity is a topic receiving increased attention in the school-based intervention literature, relatively few studies have been conducted regarding the technical adequacy of treatment integrity assessment methods.

This study reviewed all intervention studies published between 1980-1990 in Journal of Applied Behavior Analysis in which children were the subjects of the study. The authors found that treatment integrity was reported in only 16% of the studies.

The authors reviewed three learning disabilities journals between 1995-1999 to determine what percent of the intervention studies reported measures of treatment integrity. Only 18.5% reported treatment integrity measures.

This review examines the treatment integrity data of literacy interventions for students with emotional and/or behavioral disorders (EBD). Findings indicate that studies focusing on literacy interventions for students with EBD included clear operational definitions and data on treatment integrity to a higher degree than have been found in other disciplines.

This study evaluated the differences in estimates of treatment integrity be measuring different dimensions of it.

Determining whether or not the treatment or innovation under study is actually in use, and if so, how it is being used, is essential to the interpretation of any study. The concept of Levels of Use of the Innovation (LoU) permits an operational, cost-feasible description and documentation of whether or not an innovation or treatment is being implemented.

This study examines adoption and implementation of the US Department of Education's new policy, the `Principles of Effectiveness', from a diffusion of innovations theoretical framework. In this report, we evaluate adoption in relation to Principle 3: the requirement to select research-based programs.

This article discusses the current focus on using teacher observation instruments as part of new teacher evaluation systems being considered and implemented by states and districts. 

Used a direct observation-based approach to identify behavioral conditions in sending (i.e., special education) and in receiving (i.e., regular education) classrooms and to identify targets for intervention that might facilitate mainstreaming of behavior-disordered (BD) children.

This piece describes the widely held perception among education leaders that we already  know how to help teachers improve, and that we could achieve our goal of great teaching in far more classrooms if we just applied what we know more widely. 

The Good Behavior Game (GBG) is a well-documented group contingency designed to reduce disruptive behavior in classroom settings. However, few studies have evaluated the GBG with students who engage in severe problem behavior in alternative schools, and there are few demonstrations of training teachers in those settings to implement the GBG.

This book provides research as well as case studies of successful professional development strategies and practices for educators.

Considers design issues and strategies by comparative outcome studies, including the conceptualization, implementation, and evaluation of alternative treatments; assessment of treatment-specific processes and outcomes; and evaluation of the results. It is argued that addressing these and other issues may increase the yield from comparative outcome studies and may attenuate controversies regarding the adequacy of the demonstrations.

The authors proposed a preliminary FI theory (FIT) and tested it with moderator analyses. The central assumption of FIT is that FIs change the locus of attention among 3 general and hierarchically organized levels of control: task learning, task motivation, and meta-tasks (including self-related) processes.

This book examines the use of video recording to to improve teacher performance. The book shows how every classroom can easily benefit from setting up a camera and hitting “record”.  

Fidelity of treatment in outcome research refers to confirmation that the manipulation of the independent variable occurred as planned. Verification of fidelity is needed to ensure that fair, powerful, and valid comparisons of replicable treatments can be made.

This study investigated treatment fidelity in social work research. The authors systematically reviewed all articles published in five prominent social work journals over a 5- year period.

This study compared the effects of discussing issues of implementation challenges and performance feedback on increasing the integrity of implementation. Performance feedback was more effective than discussion in increasing integrity.

Examined the treatment integrity with which general education teachers implemented a reinforcement based intervention designed to improve the academic performance of elementary school students

This book looks at how new ideas spread via communication channels over time. Such innovations are initially perceived as uncertain and even risky. To overcome this uncertainty, most people seek out others like themselves who have already adopted the new idea. Thus the diffusion process typically takes months or years. But there are exceptions: use of the Internet in the 1990s, for example, may have spread more rapidly than any other innovation in the history of humankind. 

The purpose of this chapter is to explain the role of treatment integrity assessment within the “implementing solutions” stage of a problem-solving model.

This paper reviews options for treatment integrity measurement emphasizing how direct behavior rating technology might be incorporated within a multi-tiered model of intervention delivery.

Both student outcomes and treatment fidelity data are necessary to draw valid conclusions about intervention effectiveness. Reviews of the intervention outcome literature in related fields, and prior reviews of the school psychology literature, suggest that many researchers failed to report treatment fidelity data.

Over the past two decades, the role of school psychologists internationally has shifted from a more narrow focus on assessment to a broader emphasis on problem solving and delivering intervention services via consultation. Defining interventions is important for replication and translation of practice. Further, to make valid, data-based decisions about intervention effectiveness, school psychologists need to consider student outcomes in light of treatment integrity data.

Over the past two decades, the role of school psychologists internationally has shifted from a more narrow focus on assessment to a broader emphasis on problem solving and delivering intervention services via consultation. Defining interventions is important for replication and translation of practice. Further, to make valid, data-based decisions about intervention effectiveness, school psychologists need to consider student outcomes in light of treatment integrity data.

The authors reviewed all intervention studies published in the Journal of Positive Behavior Interventions between 1999-2009 to determine the percent of those studies that reported a measure of treatment integrity. Slightly more than 40% reported a measure of treatment integrity.

The authors reviewed four school psychology journals between 1995-2008 to estimate the percent of intervention studies that reported some measure of treatment integrity. About 50% reported a measure of treatment integrity.

A commonly used research design in applied behavior analysis involves comparing two or more independent variables. There were some consistencies across studies, with half resulting in equivalent outcomes across comparisons. In addition, most studies employed the use of an alternating treatments or multi-element single-subject design and compared a teaching methodology. 

This literature review was conducted to evaluate the current state of evidence supporting communication interventions for individuals with severe intellectual and developmental disabilities. Many researchers failed to report treatment fidelity or to assess basic aspects of intervention effects, including generalization, maintenance, and social validity.

This book is written for school administrators, staff developers, behavior specialists, and instructional coaches to offer guidance in implementing research-based practices that establish effective classroom management in schools. The book provides administrators with practical strategies to maximize the impact of professional development. 

This paper examines a range of education failures: common mistakes in how new practices are selected, implemented, and monitored. The goal is not a comprehensive listing of all education failures but rather to provide education stakeholders with an understanding of the importance of vigilance when implementing new practices.

Inattention to treatment integrity is a primary factor of failure during implementation. Treatment integrity is defined as the extent to which an intervention is executed as designed, and the accuracy and consistency with which the intervention is implemented 

This study compared indirect training and direct training methods as a means of impacting levels of treatment integrity.  Direct training methods produced better outcomes.

The present study was conducted to investigate the relationship between training procedures and treatment integrity. 

A review of 20 experimental, shared book reading (SBR) interventions using questioning strategies with preschool children was conducted. The studies were analyzed in terms of their quality, focus, and the questioning strategies employed. Although there were few methodological concerns about the studies conducted, treatment fidelity and replicability of the reported interventions are raised as issues needing attention in future research.

The relationships among independent variables and three measures of treatment integrity were evaluated.

Treatment integrity is essential for the implementation of interventions in schools as it determines the accuracy or consistency with which different components of a treatment are implemented. There are no current standards regarding the best practices in treatment integrity measurement; however, higher integrity is associated with enhanced student outcomes.

This study evaluated the effects of performance feedback on increasing the quality of implementation of interventions by teachers in a public school setting.