Which School Reform Programs Produce the Best Results?

Why is this question important? The continuing predicament of underperforming schools, especially notable in schools with a history of chronic underachievement and often in high-poverty communities, calls for a fundamental change in how reform efforts are chosen. Reform by its nature is complex; it demands practices that have the greatest likelihood of succeeding. Too often, reforms have failed to achieve important outcomes, such as improved test scores, reduced dropouts, and improved student behavior. Past selection criteria for these efforts have been based on factors such as fads, "common wisdom," or corporate lobbying. These methods have not worked. The need for substantive change that produces results is enormous. Such change requires the adoption of an evidence-based selection process built on scientific research employing practices that have been shown to produce tangible results in real schools.

See further discussion below.

Source(s): The Comprehensive School Reform Quality (CSRQ) Center: Report on Elementary School CSR Models, 2006

Result(s): The study rated the positive impact of 22 programs on student achievement:

1. Only two programs met the standards for having a moderately strong positive impact: Direct Instruction and Success for All.
2. Seven programs had a moderate positive impact: Accelerated Schools Plus, America's Choice School Design, Core Knowledge, Literacy Collaborative, National Writing Project, School Development Program, and School Renaissance.
3. Six programs had a limited positive impact: ATLAS Learning Communities, Different Ways of Knowing, Integrated Thematic Instruction, Modern Red Schoolhouse, Pearson Achievement Solutions (formerly Co-nect), and Ventures Initiative and Focus System.
4. Seven programs had zero positive impact: Breakthrough to Literacy, Coalition of Essential Schools, Community for Learning, Comprehensive Early Literacy Learning, Expeditionary Learning, First Steps, and Onward to Excellence II.

For descriptions and summary of results for each program, see narrative found in the CSRQ study

Implication(s): Despite billions of dollars and countless hours of well-intentioned efforts, educators and policy makers have been not been successful in achieving desired improvements. Past reform efforts have all too often been based on unproven programs and practices that were abandoned after offering educators disappointing results. Fortunately, there is a growing body of research that evidence-based practices when used in combination with practitioners sound professional judgment offer educators, policy makers, and the public support in making more productive choices in programs. The CSRQ report is an example of a well-thought-out analysis that is a practical tool for selecting interventions offering the greatest likelihood of success for reforms efforts that are based on programs used in actual schools.

Author(s): CSRQ, operated by the American Institutes for Research (AIR)

Publisher(s): American Institutes for Research

Study Description: The report offers basic information on reform models, including mission and focus, year introduced in schools, grade levels served, number of schools served, and costs. The report examined the following five categories to establish ratings on the effectiveness of the reform models:

1. Positive effects on student achievement.
2. Positive effects on additional outcomes (student discipline, student attendance, school climate, teacher retention/promotion rates, and teacher satisfaction).
3. Positive effects on parent, family, and community involvement.
4. Link between research and the model's design.
5. Evidence of services and support to schools to enable successful implementation.

The study was designed to measure and report quality and effectiveness for each category combining the following two elements to provide a single rating for each of the categories:

1. The strength of the evidence based on the causal validity of the research design. The strength of evidence depended on the rigor of the research method as defined by the reliability of the evidence, the quantity of evidence, and the consistency of evidence.
2. The strength of the impact calculated as an effect size (for example, whether the model raised student achievement a little or a lot).

Taken from the CSRQ study, the table shows how five hypothetical models might be rated.

A rating summarizing the data from each model reviewed in the study is provided in the chart entitled Evidence of Positive Effect for School Reform Models. It is the measure of each model's overall impact on student achievement. The rating was derived from a number of studies rated as suggestive and conclusive, the percentage of findings in the suggestive and conclusive studies that demonstrated a positive impact, and the average effect size of those findings. The final rating represents the volume of rigorous research and the strength of the effects reported in that research.

Definition(s):

CSRQ Center ratings

1. Very strong: The highest rating, meaning that the model demonstrates very strong (highly credible) evidence of a very strong impact.
2. Moderate-strong: The next highest rating, indicating that the combination of strength of evidence and strength of impact is moderately strong. The evidence base is not as rigorous or the overall impact as large as for very strong models.
3. Moderate: When either strength of evidence or strength of impact, or both, do not meet the higher standards described above. Models receiving this rating may still have significant evidence because of rigor or impact.
4. Limited: An indication that, while some evidence of effectiveness was found, more rigorous research needs to be conducted.
5. Zero: None of the studies were of sufficient quality to be counted as reliable evidence.

Citation: The Comprehensive School Reform Quality Center (2006), CSRQ Center: Report on Elementary School Comprehensive School Reform Models, retrieved from http://www.csrq.org/reports.asp