What is Measured for Treatment Integrity at Tier 1 in Multi-tiered Systems of Support?

The measurement of treatment integrity is important any time an intervention is implemented.  The measurement of treatment integrity is complex when assessing it at the level of universal intervention for an entire school.  Should we measure integrity at the level of the school or at the level of the individual classroom?  When assessed at the level of the whole school, we know how the school, in general, is performing; however, this may obscure how well an individual classroom is implementing the universal intervention.  Assessment at the level of the classroom is important for making decisions regarding an individual student’s need for more intensive interventions (Tier 2 or Tier 3).  If the universal intervention has not been implemented well, then it is difficult to know if the student’s failure to perform is a function of poor implementation or if the student requires more intensive support. 

Given the importance of decision-making in multi-tiered systems of support, little is known about how integrity is measured.  In a recent study by Buckman et al. (2021), treatment integrity of universal interventions was mapped in terms of frequency of measuring, the method used to assess, and unit of analysis (whole school or individual classroom).  A systematic review of the published literature since 1996 resulted in 42 articles being included in this review.  Over 86% of the articles reported procedures for monitoring integrity, and 76% reported quantifiable data for Tier 1 treatment integrity.  These are encouraging data.  The most common method for assessing treatment integrity was self-report (90%).  Self-report measures are efficient, but there is the risk of the reports being inflated over what actually occurred.  It is easy to understand why self-report is utilized so commonly, given the resource demands associated with measuring integrity across an entire system; however, much more research needs to be done to establish conditions for the self-reports to be valid measures.  Direct observation was used least often to assess treatment integrity (18.75%).  The resource demands make it very difficult to use even though it is most likely to yield the most valid data.    Procedures to balance the efficiency and effectiveness of different methods for assessing integrity have yet to be fully developed.

Monitoring of treatment integrity occurred 81% of the time at the school level.  40% of the studies assessed treatment integrity at the level of the individual classroom.  These measures are not mutually exclusive.  In some instances, integrity was measured at both levels.  Of the studies reviewed, 57% measured integrity one time per year.  This raises questions about the representativeness of the data, especially when the data were most often collected at the level of the entire school.  School-wide measurement obscures implementation at the classroom level, and measuring only one time per year may further obscure variables that influence the obtained data point.  There is no established standard for the frequency of measuring integrity at the universal level of intervention.  It could be argued that these measures should be employed at the same frequency at which decisions are made regarding students’ need for additional services.  For example, if school-wide data are reviewed three times per year, then integrity measures should occur three times per year.  This would allow decision-makers to track changes in integrity across time and determine if student performance reflects changes in integrity.  All of this is done to increase the validity of decisions regarding the level of support required for individual students. 

There are challenges to assessing integrity at the universal level.  Considerable resources are required to assess across an entire school, especially when measuring at the level of the individual classroom.  Efficient and effective systems that can be employed by existing school resources are necessary and have yet to be developed.  The importance of these systems cannot be overstated.  High-stakes decisions about students’ futures are being made based on their performance at the universal level of instruction.  It is essential that the decisions are based on valid data, including treatment integrity data.

Citation: Buckman, M. M., Lane, K. L., Common, E. A., Royer, D. J., Oakes, W. P., Allen, G. E., … & Brunsting, N. C. (2021).  Treatment integrity of primary (tier 1) prevention efforts in tiered systems: Mapping the literature.  Education and Treatment of Children, 44(3), 145-168.

Link to article: https://link.springer.com/article/10.1007/s43494-021-00044-4