Program Fidelity

This book is compiled from the proceedings of the sixth summit entitled “Performance Feedback: Using Data to Improve Educator Performance.” The 2011 summit topic was selected to help answer the following question: What basic practice has the potential for the greatest impact on changing the behavior of students, teachers, and school administrative personnel?

A 10-year comparison of graduates from 4- and 5-year teacher education programs at the same institution revealed significant differences between graduates of the two programs. Limitations of the study and alternative explanations for these differences are discussed.

This study evaluated institutional sustainability of the Early Risers “Skills for Success” conduct problems prevention program.

Differential reinforcement of appropriate behavior is an important skill for classroom teachers. This study examined the use of performance feedback to increase the rate of differential reinforcement by pre-service teachers.

In this paper, the author surveys 117 published and unpublished studies spanning more than 25 years. He structures the literature review using a conceptual model shown as Program Implementation Stages

The modified Research, Development, and Diffusion (RD&D) model as exemplified by
change agents in federal organizations were examined as a viable strategy for
disseminating social program innovations.

This paper review the promise of fidelity measures for advancing the research and practice of one area of mental health services, namely, psychiatric rehabilitation. This paper also provide a historical context for the development of fidelity measurement.

This study examines the importance of implementation integrity for problem-solving teams (PST) and response-to-intervention models.

The current study hypothesized that providing performance feedback, which has consistently been shown to increase implementation integrity, to PSTs would enhance the procedural integrity of the process.

This paper discusses the search for a “magic metric” in education: an index/number that would be generally accepted as the most efficient descriptor of school’s performance in a district.

The purpose of this study is to examine the effects of feedback on treatment integrity for implementing behavior support plans.

The authors examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. 

No Child Left Behind (NCLB) as federal policy required the use of scientifically based instruction to improve educational outcomes. It is a long way from policy to impacting what actually happens in a classroom. In this chapter, the author reviews what is known from implementation science to assure that policy becomes practice.

Over the last fifty years, there have been many educational reform efforts, most of which have had a relatively short lifespan and failed to produce the promised results. One possible reason for this is for the most part these innovations have been poorly implemented. In this chapter, the author proposes a data-based decision making approach to assuring high quality implementation.

This article reviews implementation issues in prevention trials and specifically highlights the study of implementation in the 34 programs determined to be effective in a recent review conducted by the Prevention Research Center for the Center for Mental Health Services.

The National Council of Teacher Quality (NCTQ) review examines teacher preparation program progress in adopting the necessary components of evidence-based reading instruction. The report continues the effort of two previous reports offering educators a look at trends on preparation program progress on providing this essential training.

a brief overview of findings from the Blueprints for Violence Prevention-Replication Initiative is presented, identifying factors that enhance or impede a successful implementation of these programs.

Part One provides the reader with information essential to understanding not only the content of the section that follow but also the wealth of material that exists in the literature on  program evaluation. Part Two introduces you to different approaches to evaluation to enlarge your understanding of the diversity of choices that evaluators and stakeholders make in undertaking evaluation.

This paper discusses common elements of successfully sustaining effective practices across a variety of disciplines.

This is a comprehensive literature review of the topic of Implementation examining all stages beginning with adoption and ending with sustainability.

This article presents the Fidelity of Implementation Rating System (FIMP), an observation-based measure assessing competent adherence to the Oregon model of Parent Management Training (PMTO). 

This commentary offers such a framework and then they consider how the articles constituting this special issue address the various questions posed in the framework. 

The authors have been evaluating the impact of five principal preparation programs in the United States on student outcomes. This information should be considered as one aspect of preparation program improvement and accountability. The study team lays out its recommendations in this policy paper. 

This study reviewed the reporting rates of treatment integrity for school based interventions.

This study reviewed all intervention studies published between 1980-1990 in Journal of Applied Behavior Analysis in which children were the subjects of the study. The authors found that treatment integrity was reported in only 16% of the studies.

The authors reviewed three learning disabilities journals between 1995-1999 to determine what percent of the intervention studies reported measures of treatment integrity. Only 18.5% reported treatment integrity measures.

This paper suggests a model for selecting interventions that match the context of classrooms.

In this review, we explore the extent to which researchers evaluating the efficacy of Tier 2 elementary reading interventions within the framework of Response to Intervention reported on fidelity of implementation and alignment of instruction between tiers. However, researchers frequently neglect to report on fidelity of intervention in Tier 1, potentially limiting claims that can be made about the efficacy of subsequent Tier 2 intervention.

The purpose of this study was to compare the effects of constant time delay delivered with high procedural fidelity to constant time delay with high procedural fidelity on all variables except delivery of the controlling prompt

Response to Intervention (RtI) has gained increased attention with the reauthorization of the Individuals with Disabilities Education Improvement Act. Since RtI was introduced at the policy level as a mechanism for use in the learning disability identification process, much of the implementation work has focused on this application. 

The present study used cross-sectional data from 1,438 schools to examine relations between fidelity self-assessment and team-based fidelity measures in the first 4 years of implementation of School-Wide Positive Behavioral Interventions and Supports (SWPBIS). Results showed strong positive correlations between fidelity self-assessments and a team-based measure of fidelity at each year of implementation.

In 1990, the Commonwealth of Pennsylvania implemented a statewide instructional support team (IST) process to provide prereferral assessment and intervention for at-risk students in 500 school districts. The current study examined the academic performance of students affected by this process as contrasted with other at-risk students who did not have access to it.

The authors conducted a comprehensive review of research to identify the impact of coaching on changes in preservice and in-service teachers’ implementation of evidence-based practices.

The authors conducted a comprehensive review of research to identify the impact of coaching on changes in preservice and in-service teachers’ implementation of evidence-based practices. They identified a total of 13 studies from the 20 years of literature they searched.

Assessing fidelity of implementation of school-based interventions is a critical factor in successful implementation and sustainability. The Tiered Fidelity Inventory (TFI) was developed as a comprehensive measure of all three tiers of School-Wide Positive Behavioral Interventions and Supports (SWPBIS) and is intended to measure the extent to which the core features of SWPBIS are implemented with fidelity.

This study examined whether children receiving one year of EIBI (N = 35) would make larger gains in adaptive behaviors than a group of children receiving treatment as usual (TAU; N = 24).

Full and durable implementation of school-based interventions is supported by regular evaluation of fidelity of implementation. Multiple assessments have been developed to evaluate the extent to which schools are applying the core features of school-wide positive behavioral interventions and supports (SWPBIS).

Several reliable and valid fidelity surveys are commonly used to assess Tier 1 implementation in School-Wide Positive Behavioral Interventions and Supports (SWPBIS); however, differences across surveys complicate consequential decisions regarding school implementation status when multiple measures are compared. Compared with other measures, the PBIS Self-Assessment Survey (SAS) was more sensitive to differences among schools at higher levels of implementation. Implications for SWPBIS research and fidelity assessment are discussed.

Fidelity of treatment in outcome research refers to confirmation that the manipulation of the independent variable occurred as planned. Verification of fidelity is needed to ensure that fair, powerful, and valid comparisons of replicable treatments can be made.

This study evaluated the effects of performance feedback on the implementation of a classroom intervention.

This literature review highlights central elements of the residency component of school leader preparation programs by aggregating the results of studies conducted on existing principal preparation programs. 

This literature review highlights central elements of the residency component of school leader preparation programs by aggregating the results of studies conducted on existing principal preparation programs. 

This study compared the effects of discussing issues of implementation challenges and performance feedback on increasing the integrity of implementation. Performance feedback was more effective than discussion in increasing integrity.

This study evaluated the effects of consultation and performance feedback in a public school setting.

This study looks at treatment integrity with which general education teachers implemented a reinforcement based intervention designed to improve the academic performance of elementary school students.

Education researchers are being asked to conduct rigorous, scientifically based studies of K–12 curriculum interventions; therefore, the need for measuring fidelity of implementation and empirically relating it to outcomes (the chief rationale for this review) is warranted to ensure internal and external validity. The results of this review indicate that there are too few studies to guide researchers on how fidelity of implementation to core curriculum interventions can be measured and related to outcomes, particularly within efficacy and effectiveness studies, where the requirements for fidelity measures differ. 

This article describes early aspects of the nationwide implementation of an evidence‐based program (EBP) in Norway and the design for studying program fidelity over time. 

We sought to identify, examine, and summarize empirical literature focused on early childhood behavior interventions examined using a single case research designs (SCD) and published between 2001 and 2018. The findings of the current review suggest: promoting implementation fidelity through implementation support to improve social validity outcomes, providing guidelines for timing and frequency of social validity assessment, and development of social validity assessment tools designed to assess each of the social validity dimensions.

The authors reviewed all of the intervention studies published in the Journal of Applied Behavior Analysis between 1968-1980 to determine the percent of studies that reported measures of treatment integrity. The overall average across those years was 16%.

First Step to Success is an empirically supported intervention for young elementary school students that can be implemented by public school teachers with training. This study evaluated the effects of coaching feedback on teachers who not effectively implemented First Step following training.

The authors reviewed all intervention studies published in the Journal of Positive Behavior Interventions between 1999-2009 to determine the percent of those studies that reported a measure of treatment integrity. Slightly more than 40% reported a measure of treatment integrity.

If educational programs are to be effective they must be implemented with sufficient integrity to assure benefits. To have a significant impact on schools, solutions must be scalable. This study evaluated the effects of using existing school personnel to provide performance feedback to teachers regarding the quality of implementation.

The authors reviewed four school psychology journals between 1995-2008 to estimate the percent of intervention studies that reported some measure of treatment integrity. About 50% reported a measure of treatment integrity.

A critical review of reading programs requires objective and in-depth analysis. For these reasons, the authors offer the following recommendations and procedures for analyzing critical elements of programs.

These publications have laid out guidelines to successfully implement the changes that they suggest will improve students' understandings of science. In keeping with these suggestions, several research groups have created, piloted, and implemented curricular programs in schools in the hopes of increasing students' science content understanding as well as their complex reasoning skills.

We report findings from a quasi-experimental evaluation of the recently implemented US$5,000 retention bonus program for effective teachers in Tennessee’s Priority Schools. We estimate the impact of the program on teacher retention using a fuzzy regression discontinuity design by exploiting a discontinuity in the probability of treatment conditional on the composite teacher effectiveness rating that assigns bonus eligibility.

In this article, the authors address the following questions: How does the level of on-site technical assistance affect student outcomes? Do teachers’ fidelity of treatment implementation and their perceptions of school climate mediate effects on student performance?

Treatment fidelity reporting practices are described for journals that published general and special education intervention research with high impact factors from 2005 through 2009. The authors reviewed research articles, reported the proportion of intervention studies that described fidelity measurement, detailed the components of fidelity measurement reported, and determined whether the components of fidelity reported differed based on the research design, the type of intervention, or the number of intervention sessions.

This study examined the fidelity of problem-solving implementation by multidisciplinary
teams (MDTs) in 227 schools and the relationship to student outcomes.

Intended for state officials involved in the assessment and approval of university and other programs to train future school principals, this report describes five design principles for effective program evaluation.

Implementation fidelity is often thought of as a necessary condition to achieve internal validity and as having a relation to student outcomes. To examine the nature of this relation, we reviewed reading intervention studies for students in K-12 in which measures of implementation fidelity were included in final data analysis.

This study evaluated the effects of performance feedback on increasing the quality of implementation of interventions by teachers in a public school setting.