How often are treatment integrity measures reported in published research?

Why is this question important? Educators are mandated to use research-based practices to improve student performance. One of the primary ways to identify research-based practices is through peer-reviewed journals. Research-based practices can be identified by reading the articles or reviewing meta-analyses of the effects of specific practices. Anyone doing either might assume that the researchers responsible for implementing the practices actually followed implementation protocols, but that person would know for sure only if the researchers measured and reported on the adequacy of implementation. Without these data, it is not possible to know how well the implementation protocol was followed, which reduces confidence in the effects claimed.

See further discussion below.

Results: Regardless of the characteristics of the review—type of subjects, setting, and number of years—reports of treatment integrity measures were meager. Of all the listings in the graph, the 2007 review by McIntyre, Gresham, DiGennaro, and Reed was the first to find more than 20% of the published reports addressing measures of treatment integrity. In their 2011 review, Sanetti, Gritter, and Dobey found the highest percentage of articles reporting on treatment integrity—50%, still a troublingly low level. The reviews calculated only the percentage of published reports that included measures of treatment integrity and did not say anything about the quality of treatment integrity attained in the research studies. Anyone wanting to find out about the quality would have to read each individual study to find out how well the intervention was implemented, if indeed that information was included in the article.

Implications: If educators are to adopt a research-based intervention, they must be confident that the intervention was actually implemented according to protocol. As shown in the graph, at best only 50% of published studies reported any measure of treatment integrity, so half or more of the published studies that were reviewed did not provide any information about how well procedures were implemented. Without this information, it is impossible to know if the outcome of a study was a function of the described intervention procedures or some variation of those procedures. There is no compelling reason for published research articles to omit reports on measures of treatment integrity. No reputable journal publishes studies that omit reports on the reliability of measures of outcomes. Measures of how well interventions are implemented should be held to the same standard. If journals' publication policies required reporting on treatment integrity, the low report rates found in reviews would be a thing of the past.

Study Description: Each group of authors listed in the graph reviewed articles on a specified subject published in a selected journal over a set period of years. They coded the articles for the years included in the review, the population involved in the studies (e.g., children), and the settings in which the studies were done (e.g., schools). Then they calculated the percentage of the articles that reported on some measure of treatment integrity.

Definitions: Treatment integrity: Implementing an intervention as prescribed.

Citation:
Gresham, F. M., Gansle, K. A., & Noell, G. H. (1993a). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26(2), 257–263.

Gresham, F. M., Gansle, K. A., Noell, G. H., Cohen, S., & Rosenblum, S. (1993b). Treatment integrity of school-based behavioral intervention studies: 1980–1990. School Psychology Review, 22, 254–272.

Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., & Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research & Practice, 15(4), 198–205.

McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991–2005. Journal of Applied Behavior Analysis, 40(4), 659–672.

Peterson, L., Homer, A. L., & Wonderlich, S. A. (1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15(4), 477–492.

Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40(1), 72–84.

Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29–46.

Wheeler, J. J., Baggett, B. A., Fox, J., & Blevins, L. (2006). Treatment integrity: A review of intervention studies conducted with children with autism. Focus on Autism & Other Developmental Disabilities, 21(1), 45–54.