What is the impact of reform efforts over time?

Why is this question important? The past 30 years in American education is best described as a continuous effort to reform the nation's schools (Yeh, 2007). Unfortunately, these efforts have delivered little and can be characterized as a case of jumping from one fad to the next (Paul & Elder, 2007; Slavin & Madden, 1989). The upshot has been the adoption of too many practices without evidence of effectiveness, a superficial execution of practices implemented without regard to treatment integrity (Detrich, 2014), and a never-ending quest for quick, easy fixes to solve the problems of a system that has produced few notable results for more than 30 years (National Center for Education Statistics, 2013). The science of implementation informs us that the process of solving complex problems is not simple, and that it requires time and a willingness to adapt practices to the reality of the novel circumstances found in each new setting. Data from other fields indicate that three to five cycles may be required to find an acceptable and effective solution (Blase, Fixsen, Sims, & Ward, 2014) and from 2 to 4 years for full implementation to be achieved (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005).

The typical intervention is short lived, lasting on average only 14 to 18 months (Latham, 1988), and research shows the greatest impact for comprehensive school reform (CSR) takes place after 5 years of implementation (Borman, Hewes, Overman, & Brown, 2013). This finding implies that even when we select the best evidence-based practices, we are habitually abandoning potentially valuable solutions years before we could expect to reap the benefits of our labors.

See further discussion below.

Results: The Borman et al. study found that schools implementing CSR models for 5 years or more showed particularly strong effects, and the benefits were consistent across schools of varying economic levels.

The study analyzed the relationship between years of implementation and effect size. The study showed a small but significant 0.17 effect size during the first year of implementation. The effect size declined slightly during the second, third, and fourth years of implementation. But from the fifth year onward, effect sizes increased substantially and were nearly twice those found for CSR schools in general. From 7 years onward, the impact was more than two and a half times the magnitude of the overall CSR impact of d = 0.15. The effect continued to increase for those few schools with outcome data beyond 8 years of CSR model implementation and was three and a third times larger than the overall CSR effect.

Implications: The Borman et al. research found that comprehensive system-wide efforts to improve schools can produce results. An important lesson is that efforts require time to mature if they are to produce significant results. The study points to the importance of designing effective implementation strategies that must be sustained over years and despite changes in personnel. This might prove to be the greatest challenge to school reform as the short tenure of a school principal and teachers is less than the time an intervention requires to produce results. (See Wing Institute Data Mining item: What is the turnover rate for school principals?) Bringing about the dramatic improvements in American schools that data show are necessary requires a change in the culture of education, which currently values a radical restructuring over sustained interventions lasting beyond 5 or 10 years. It requires shifting to a culture that values monitoring progress, and rewards incremental adaptation and improvement.

Study Description: The Borman et al. study is a meta-analysis of 232 studies on the impact of CSR and the effects of 29 widely implemented reform models. The meta-analysis also examined components as well as contextual and methodological factors associated with CSR. Those reform models sustained for 5 years or more showed the strongest effects. The interventions benefited equally students of higher and lower socio-economic status. To be included in the study the reform models needed to meet the U.S. Department of Education definition for comprehensive school reform using 11 components that, when implemented, represent a "comprehensive" and "scientifically based" approach to school reform.

Citation:
Blase, K. A., Fixsen, D. L., Sims, B. J., & Ward, C. S. (2014). Implementation science: Changing hearts, minds, behavior, and systems to improve educational outcomes. Presented at the Wing Institute Summit, Berkeley, CA. *

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73(2), 125–230.

Detrich, R. (2014). Treatment integrity: Fundamental to education reform. Journal of Cognitive Education and Psychology, 13(2), 258–271.

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

Latham, G. (1988). The birth and death cycles of educational innovations. Principal, 68(1), 41–43.

National Center for Education Statistics (NCES). (2013). The nation's report card: Trends in academic progress 2012 (NCES 2013-456). Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013456.

Paul, R., and Elder, L. (2007). A critical thinker's guide to educational fads: How to get beyond educational glitz and glitter. The Foundation for Critical Thinking Press. Retrieved May 11, 2015.

Slavin, R. E., & Madden, N. A. (1989). What works for students at risk: A research synthesis. Educational leadership, 46(5), 4–13.

Yeh, S. S. (2007). The cost-effectiveness of five policies for improving student achievement, American Journal of Evaluation, 28(4), 416–436.

* study from which graph data were derived