Continuum of Evidence

Evidence-based, empirically-supported, and best practice are often used interchangeably. A case is made that for clarity each term should have a separate and distinct meaning.

Increasing education’s reliance on evidence to guide decisions requires a significant change in the culture of districts and schools. This paper reviews the implications of moving toward evidence-based education.

Evidence-based practice is a framework for decision making.  Even with high quality evidence there are likely sources of uncertainty that practitioners must confront.

This paper outlines the best practices for researchers and practitioners translating research to practice as well as recommendations for improving the process.

The systematic review process is an assessment and, as such, concerns about validity of the assessment are paramount.  In this paper, we review the considerations that are important in reaching conclusions about the adequacy of a systematic review.

Applied behavior analysis emphasizes being scientifically-based In this paper, we discuss how the core features of evidence-based practice can be integrated into applied behavior analysis.

The notion of best available evidence implies that some evidence is better than other. This paper reviews different sources of evidence and the relative strengths and limitations of each type.

Synopsis: Evidence-based practice is characterized as a framework for decision-making integrating best available evidence, clinical expertise, and client values and context.  This paper reviews how these three dimensions interact to inform decisions.

Multitiered system of support (MTSS) is a framework for organizing service delivery. At the core of MTSS is the adoption and implementation of a continuum of evidence-based interventions that result in improved academic and behavioral outcomes for all students. MTSS is a data-based decision making approach based on the frequent screening of progress for all students and intervention for students who are not making adequate progress.

The purpose of this article is to describe how effective practices are incorporated into an approach termed schoolwide positive behavior supports (SWPBS)

This paper examines the types of research to consider when evaluating programs, how to know what “evidence’ to use, and continuums of evidence (quantity of the evidence, quality of the evidence, and program development).

This paper outlines the best practices for researchers and practitioners translating research to practice as well as recommendations for improving the process.

 The present paper makes the case for systematic assessment and evaluation in clinical practice. The purpose of systematic evaluation is to enhance client care and to improve the basis for drawing inferences about treatment and therapeutic change.

We analyze the relationship between inequality and economic growth from two directions. The first part of the survey examines the effect of inequality on growth. The second part analyzes several mechanisms whereby growth may increase wage inequality, both across and within education cohorts.

This paper examines the benefits and challenges inherent in using randomized clinical trials and quasi-experimental designs in the field of education research.

Evaluated the use of the N. S. Jacobson et al (see record 1985-00073-001) criteria for clinical significance in psychotherapy data analysis.

Objective: To review alternative treatments (Tx) of Attention-Deficit/Hyperactivity Disorder (ADHD)those other than psychoactive medication and behavioral/psychosocial Tx-for the November, 1998 National Institute of Health (NIH) Consensus Development Conference on ADHD.

This chapter reviews a set of behavioral science findings derived from the November 1993 NIDA Technical Review, “Reviewing the Behavioral Science Knowledge Base on Technology Transfer.” This is not intended to be a complete recapitulation of the arguments and conclusions drawn by the authors of the 14 papers presented in this monograph.

In this article, we attempt to distinguish between the properties of moderator and mediator variables at a number of levels.

s. In this study, the authors used meta-analytic procedures to test one possible factor contributing to the attenuation of effects: structural inequalities between placebo and active treatments.

The author discusses what he believes to be the promise of this approach, its influence on the role of the school psychologist, and what educators can do if they choose to pursue the leads offered by this group. 

This research evaluated the outcomes of a school psychology training practicum by replicating intervention-based service delivery procedures established in prior research. 

Observational data collected in ecologically valid measurement contexts are likely to be influenced by contextual factors irrelevant to the research question. Using multiple sessions and raters often improves the stability of scores for variables from such contexts.

This article describes an evaluation of a prisoner-run delinquency prevention program at Hawaii's major prison.

As pressure increases for the demonstration of effective treatment for children with mental disorders, it is essential that the field has an understanding of the evidence base. To address this aim, the authors searched the published literature for effective interventions for children and adolescents and organized this review

Over many decades, educators have developed countless interventions and theories about how to create lasting change. In other words, implementation research is an endeavor to understand if and how educational efforts are accomplishing their goals. 

Eight comprehensive chapters cover the common problems of disruptive behavior, anxiety, sleep disorders, nocturnal enuresis, encopresis, habit disorders (such as tics and thumbsucking), the treatment of pain and, finally, helping children adhere to medical regimens. The book describes diagnosis and treatment, with an emphasis on practicality. 

This guide presents the tools therapists need to incorporate outcomes measurement effectively and meaningfully into everyday clinical work.

The effects of changes in depression-relevant cognition were examined in relation to subsequent change in depressive symptoms for outpatients with major depressive disorder randomly assigned to cognitive therapy (COT; n = 32) vs those assigned to pharmacotherapy only (NoCT; n = 32). 

This article explored developmental and intervention evidence relevant to iatrogenic effects in peer-group interventions. Longitudinal research revealed that "deviancy training" within adolescent friendships predicts increases in delinquency, substance use, violence, and adult maladjustment.

This comprehensive textbook is an essential primer for all practitioners and students who are grappling with the new age of evidence-based practice. The contributors explore some of the complex challenges in implementing EBPs, and highlight the meaningful opportunities that are inherent in this paradigm shift.

This article discusses the use of randomized controlled trials as required by the Department of Education in evaluating the effectiveness of educational practices.

The authors argue that important evidence about best practice comes from case-based research, which builds knowledge in a clinically useful manner and complements what is achieved by multivariate research methods. 

Cover, copy, compare (CCC) has been used with success to improve spelling skills. This study adds to existing research by completing an analysis of the rewriting component of the intervention. The impact of varying the number of times a subject copied a word following an error was examined with four elementary age students.

This book analyzes the findings of a treatment program which integrated antisocial and delinquent youths into prosocial peer groups in a suburban community center in St. Louis.

In this paper we will review some of the examples from industrial innovation and dissemination, provide some data on replications of the Achievement Place/Teaching-Family Model over 20 years, and try to share some of the philosophical, practical, and technological guidelines we have come to accept.

Predictions are made in the form of if-then statements. If reliable predictions define science and testing predictions is the work of scientists, then implementation science is a science to the extent that 1) predictions are made and 2) those predictions are tested in practice.

The resource can be found on the "Resources" tab of the Active Implementation website.   

The standard reference in the field, this acclaimed work synthesizes findings from hundreds of carefully selected studies of mental health treatments for children and adolescents.

This article introduces a special section addressing these resource allocation issues in the context of prevalent disorders 

This study examines adoption and implementation of the US Department of Education's new policy, the `Principles of Effectiveness', from a diffusion of innovations theoretical framework. In this report, we evaluate adoption in relation to Principle 3: the requirement to select research-based programs.

Hattie’s book is designed as a meta-meta-study that collects, compares and analyses the findings of many previous studies in education. Hattie focuses on schools in the English-speaking world but most aspects of the underlying story should be transferable to other countries and school systems as well. Visible Learning is nothing less than a synthesis of more than 50.000 studies covering more than 80 million pupils. Hattie uses the statistical measure effect size to compare the impact of many influences on students’ achievement, e.g. class size, holidays, feedback, and learning strategies.

This unique and ground-breaking book is the result of 15 years research and synthesises over 800 meta-analyses on the influences on achievement in school-aged students. It builds a story about the power of teachers, feedback, and a model of learning and understanding.

This book examines the use of randomized controlled trial (RCT) studies in education.

A statistics textbook appropriate for graduate students and researchers conducting quasi-experimental design and analysis.

This essay discusses issues and concerns that too many research findings may be false. The paper examines reasons a study may prove inaccurate including: the study power and bias, the number of other studies on the same question, and the ratio of true to no relationships. Finally, it considers the implications these problems create for conducting and interpreting research.

A comprehensive, easily digestible introduction to research methods for undergraduate and graduate students. Readers will develop an understanding of the multiple research methods and strategies used in education and related fields, including how to read and critically evaluate published research and how to write a proposal, construct a questionnaire, and conduct an empirical research.

This paper is a review of primary research investigating the Feingold hypothesis which suggests diet modification as an efficacious treatment for hyperactivity. 

This chapter traces the history of behavior modification as a general movement. Individual conceptual approaches and techniques that comprise behavior modification are obviously important in tracing the history, but they are examined as part of the larger development rather than as ends in their own right. 

The previous articles in this special section make the case for the importance of evaluating the clinical significance of the therapeutic change, present key measures and innovative ways in which they are applied, and more generally provide important guidelines for evaluating therapeutic change. 

The review by Sheldrick et al. evaluates treatments for children and adolescents with conduct disorder and whether they produce clinically significant change

The focus of this chapter is on psychotherapy research and a call for research on mechanisms of therapeutic change. 

In this article, we discuss the importance of studying mechanisms, the logical and methodological requirements, and why almost no studies to date provide evidence for why or how treatment works.

The authors developed a methodological basis for investigating how risk factors work together. Better methods are needed for understanding the etiology of disorders, such as psychiatric syndromes, that presumably are the result of complex causal chains.

This paper describes an analytic framework to identify and distinguish between moderators and mediators in RCTs when outcomes are measured dimensionally. 

This paper by a What Works Clearinghouse the panel provides an overview of singlr-subject designs (SCDs), specifies the types of questions that SCDs are designed to answer, and discusses the internal validity of SCDs. The panel then proposes standards to be implemented by the WWC.

The present study attempted to examine the causal relationships among changes in automatic thoughts, dysfunctional attitudes, and depressive symptoms in a 12-week group cognitive behavior therapy (GCBT) program for depression.

This bestselling resource presents authoritative thinking on the pressing questions, issues, and controversies in psychotherapy research and practice today.

In the present correlational study of 199 treated adolescents, the authors used a multitrait-multimethod analysis to examine psychometrically measured pathology change (pre- and postassessment of symptoms and functioning), consumer satisfaction, and perceived improvement reported by multiple informants.

The chapter focuses on the historically perceived poor methodological rigor and low scientific credibility of most educational/psychological intervention research.

Despite increased attention to methodological rigor in education research, the field has focused heavily on experimental design and not on the merit of replicating important results. The present study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications. Contrary to previous findings in medicine, but similar to psychology, the majority of education replications successfully replicated the original studies. However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. The results emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.

Used meta‐analysis to examine the efficacy of bibliotherapy. Bibliotherapy treatments were compared to control groups and therapist‐administered treatments. 

In a large sample of children from the general population this research found no association between parent, teacher, and self-reports of ADDH behaviors and a history of allergic disorders (asthma, eczema, rhinitis, and urticaria) at ages 9 or 13 years.

Developed a fidelity index of program implementation for assertive community treatment (ACT). In Study 1, 20 experts rated the importance of 73 elements proposed as critical ACT ingredients, also indicating ideal model specifications for elements.

A variety of researches are examined from the standpoint of information theory. It is shown that the unaided observer is severely limited in terms of the amount of information he can receive, process, and remember. However, it is shown that by the use of various techniques, e.g., use of several stimulus dimensions, recoding, and various mnemonic devices, this informational bottleneck can be broken.

This article covers current efforts by the National Institute of Mental Health to bridge this gap. Included are discussions of problems with the current research portfolio and new efforts in expanding the research portfolio, innovative methodological research, and expansion of training programs. 

The relationship of client satisfaction to outcome was investigated for adult outpatients (N  =  152) from 3 urban community mental health centers. Clients completed a problem self-rating and the Brief Symptom Inventory (BSI) at intake, 10 weeks later, and 5 months later.

This award-winning twelve-volume reference covers every aspect of the ever-fascinating discipline of psychology and represents the most current knowledge in the field. This ten-year revision now covers discoveries based in neuroscience, clinical psychology's new interest in evidence-based practice and mindfulness, and new findings in social, developmental, and forensic psychology.

Describes treatment of autism, a severe, chronic developmental disorder that results in significant lifelong disability for most persons, with few persons ever functioning in an independent and typical lifestyle.

Current systems for listing empirically supported therapies (ESTs) provide recognition to treatment packages, many of them proprietary and trademarked, without regard to the principles of change believed to account for their effectiveness.

Two messages are conveyed in the report: Mental health is fundamental to health, and mental disorders are real health conditions. The surgeon general's report summarizes the Office's detailed review of more than 3,000 research articles, plus 1st-person accounts from individuals who have been afflicted with mental disorders. 

This article suggests the routine use of replications in field studies. 

The challenges of specifying a complex and individualized treatment model and measuring fidelity thereto are described, using multisystemic therapy (MST) as an example.

This study evaluated the impact of intensive behavioral treatment on the development of young autistic children.

This paper outlines the best practices for researchers and practitioners translating research to practice as well as recommendations for improving the process.

For many centuries, the professor was the primary source of information, the font of knowledge. Books were nonexistent or scarce, as they still are today in developing countries of the world, and information was passed orally from teacher to pupil. The didactic lecture is an effective method for conveying information from one person to a larger number of students, but, as most of us have experienced, simply telling information to someone does not ensure that learning takes place.

Did you know that plants and plant products can be used to improve people’s cognitive, physical, psychological, and social functioning? Well, they can, and Horticulture as Therapy is the book to show you how! 

A recent large-scale evaluation of Reading Recovery, a supplemental reading program for young struggling readers, supports previous research that found it to be effective.  In a 4 year, federally funded project, almost 3,500 students in 685 schools found that generally students benefitted from the intervention. Students receiving Reading Recovery receive supplemental services in a 1:1 instructional setting for 30 minutes 5 days a week from an instructor trained in Reading Recovery.  In the study reported here, students who received Reading Recovery had effect sizes of .35-.37 relative to a control group across a number of measures of reading.  These represent moderate effect sizes and account for about a 1.5 month increase in skill relative to the control group.  Even though the research supports the efficacy of the intervention, it also raises questions about its efficiency.  The schools that participated in the study served about 5 students and the estimated cost per student has ranged from $2,000-$5,000.  These data raise questions about the wisdom of spending this much money per student for growth of about a month and a half.

Since 1980, 12 peer-reviewed outcome studies (nine on behavior analytic programs, one on Project TEACCH, and two on Colorado Health Sciences) have focused on early intervention for children with autism. Mean 10 gains of 7-28 points were reported in studies of behavior analytic programs, and 3-9 in studies on TEACCH and Colorado. 

A soon to be published meta-analysis of Direct Instruction (DI) curricula that reviews research on DI curricula between 1966-2016 reports that DI curricula produced moderate to large effect sizes across the curriculum areas reading, math, language, and spelling.  The review is notable because it reviews a much larger body of DI research than has occurred in the past and covers a wide range of experimental designs (from single subject to randomized trials).  328 studies were reviewed and almost 4,000 effects were considered.  Given the variability in research designs and the breadth of the effects considered, it suggests that DI curricula produce robust results.  There was very little decline during maintenance phases of the study and greater exposure to the curricula resulted in greater effects.

The purpose of this article is to describe how effective practices are incorporated into an approach termed schoolwide positive behavior supports (SWPBS)

At the request of David Barlow, President of Division 12, and under the aegis of Section III, this task force was constituted to consider methods for educating clinical psychologists, third party payors, and the public about effective psychotherapies

This paper examines the types of research to consider when evaluating programs, how to know what “evidence’ to use, and continuums of evidence (quantity of the evidence, quality of the evidence, and program development).

Simple experimental studies randomize study participants into two groups: a treatment group that includes participants who receive the offer to participate in a program or intervention, and a control group that includes participants who do not receive that offer. Such studies primarily address questions about the program impacts on the average outcomes of participants.

The text and graphics contained in the 26th Annual Report to Congress were developed primarily from data from the Office of Special Education Programs (OSEP) Data Analysis System (DANS). DANS is a repository for all the data mandated by the Individuals with Disabilities Education Act (IDEA) to be collected from states annually. 

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.

The Innovation Journey presents the results of a major longitudinal study that examined the process of innovation from concept to implementation of new technologies, products, processes, and administrative arrangements.

The Child Task Force report represents an important initial step in this direction. Here they offer both praise and critique, suggesting a number of ways the task force process and product may be improved. 

This article addresses the gap between clinical practice and the research laboratory. We focus on the issue as it relates specifically to interventions for children and adolescents.

The study does suggest that "more is not always better" (L. Bickman, 1996), but more of what? Little is known about the specific interventions that were combined to form the Fort Bragg system of care, so the study does not really reveal what failed or what needs to be changed. 

The Society of Clinical Psychology's task forces on psychological intervention developed criteria for evaluating clinical trials, applied those criteria, and generated lists of empirically supported treatments. Building on this strong base, the task force successor, the Committee on Science and Practice, now pursues a three‐part agenda

This slide show presents what is EBE and what are EBE goals in education. 

The goal of the WWC is a resource for informed education decision-making. The WWC identifies evidence-based practice, program, or policy, and disseminates summary information on the WWC website.