| |
Monitor: To observe a behavior for any changes that may occur over time and/or for effects an intervention may have on the observed behavior.
Is it working?
To assure that students are benefiting from educational interventions it is necessary to monitor performance. We can only judge success after an intervention has been implemented.
We monitor interventions for two important reasons:
- To make macro level (systems) decisions and policy choices
- To make decisions at the individual student level
A group’s performance is the focus at the macro level as opposed to the individual student.
Monitoring at this level helps stakeholders see the systems effectiveness in the following areas:
- Student achievement in relation to standards
- Continuous improvement against past performance
Types of Macro-Level Monitoring (Systems)
High stakes tests: A system of tests conducted for the purpose of making important decisions that have significant consequences for state systems, districts, school, teachers, and/or students.
Performance outcomes measures: The quantifiable result or end product of worthy performance.
There are two types of Performance Outcome Measures:
- Performance outcomes: A direct measurement of the worthy and intended purpose of an intervention.
Example: high school graduation - Process outcomes: A measurement of a mission critical process that leads toward a process outcome.
Example: school attendance
Progress Monitoring (Platinum Standard)
Monitoring individual student performance is referred to as the “platinum standard”.
The “platinum standard” offers an important counter point to rigorous standards of proof by offering options that focus on safeguards for the success of each child.
Types of progress monitoring: This type of monitoring is focused on evaluations of individual student performance.
- Student behavior
- Curriculum-based measurement (CBM)
- Dynamic Indicators of Basic Early Literacy Skills (DIBELS)
- Report cards
Whether the focus is on the group or the individual, monitoring is designed to answer the most important question of all, Is the Intervention Working?
To assure that students are benefiting from educational interventions it is necessary to monitor performance. In the evidence-based model we adhere to the notion that it is best to select interventions based upon rigorous empirical research. When we adopt interventions based on this standard we increase the odds that the practice will produce successful results, but this is not a certainty. We can only judge success after an intervention has been implemented. Only from evaluating the results of the intervention can we be sure that the intended outcomes have been realized. Conditions that exist outside the laboratory have a profound effect on all interventions. Failure is common and success often limited to specific populations under specific conditions. Even the positive effects of proven interventions may diminish over time due to changes in the environment. Unfortunately, there is no guarantee that an intervention will work for everyone, everywhere, every time. For these reasons, monitoring is an indispensable component of the evidence-based education model.
When combined with systematic monitoring, evidenced-based decision-making remains the educator’s most powerful model for selecting and sustaining practices that “work”.
We monitor interventions for two important reasons:
- To make macro level (systems) decisions and policy choices
- To make decisions at the individual student level
Stakeholders employ macro level monitoring to assure that schools and the education system meet expectations. The performance of all students is the focus at the macro level. Tracking system-wide achievements can reveal how well a group is performing against expectations as well as highlight overall trends. Monitoring at this level helps stakeholders see the system’s effectiveness in the following areas:
- Student achievement in relation to standards
- Continuous improvement against past performance
Macro level monitoring works primarily as a feedback mechanism to keep stakeholders apprised of progress. This requires:
- Collection of accurate and objective data, representative of the performance of students and of those who provide the educational service.
- Dissemination of the data to stakeholders in a timely manner and in a form that is easily understood.
- Stakeholder interaction with the data for the purpose of making future decisions.
Decisions made when feedback is non-existent or insufficient, or when information about variables and the relationships among these variables is absent may lead to faulty conclusions.
Macro level monitoring contributes to effective performance when:
- It informs decision makers that an intervention has been successful. This allows the decision maker to replicate the intervention under similar circumstances to achieve additional positive results.
- The information can be shared with stakeholders to increase support required for sustaining interventions.
- It provides specific feedback on exemplary performance, reinforcing good performance.
- It identifies problems in a timely manner, enabling prompt action and use of problem-solving strategies.
- It identifies failed interventions so more effective alternatives can replace them.
- High stakes tests
- Performance outcome measures
High stakes tests: Tests in which the results are used to make important decisions that have significant consequences for state systems, districts, schools, teachers, and/or students:
- High school graduation
- Promotion to the next grade
- Access to resources and special opportunities
- Access to teacher compensation or promotion
Examples: high stakes tests
- NAEP - National Assessment of Educational Progress
- State high school exit exams – 19 states have exit exams (CAHSEE - California High School Exit Examination)
- Alternate assessment for children with disabilities – (CAPA – California Alternative Performance Assessment, UAA - Utah Alternative Assessment)
- SAT - Scholastic Aptitude Test
- ACT - American College Testing Program or American College Test
- MCAT - Medical College Admission Test
- GMAT - Graduate Management Admission Test
- TOEFL - Test of English as a Foreign Language
Performance outcomes measures: measures of the benefits or changes to populations or groups resulting from an intervention, program, or practice compared to its intended outcome.
There are two types of performance outcome measures:
- Performance outcomes: A direct measurement of the worthy and intended purpose of an intervention.
Example: high school graduation - Process outcomes: A measurement of a mission critical process that leads toward a performance outcome.
Example: school attendance
Examples: performance outcomes
- Graduation rates
- College entrance
- Violent crime victimization rate
- Discipline – referral to the office
- Drug use
- Fully credentialed teachers
Examples: process outcomes
- School attendance
- Percent of students enrolled in college preparation classes
- Percent of students completing conflict resolution curriculum
- Percent of classroom staff completing Positive Behavior Intervention Services (PBIS) training
- Percent of students completing drug prevention curriculum
- Number of job fairs attended
Rigorous standards of proof, including randomized trials, are designed for the selection of practices that will meet the needs of the population-at-large. Interventions selected on this criterion are known as “universal interventions”. When we adopt universal interventions we increase the odds that the practice will “work” for the majority of students but there are no guarantees for the individual. No practice will necessarily work for all. Each student is unique. Inevitably, some students will not respond as we wish. The critical question then becomes: “Is it working?” for an individual student. Progress monitoring focuses on the individual. Monitoring each student’s performance with concern for maximizing success for each student is referred to as the “platinum standard”.
The same holds true in medicine. When a physician selects a medication that is demonstrated to be effective they need to follow-up to assure that the treatment is working for a specific patient. Not infrequently, a physician will need to adjust the dosage or even prescribe a different medication altogether for a particular patient. The fact is not all people respond the same. We all react a little differently.
Our response to educational interventions follows the same pattern. We must frequently monitor performance to assure that each student is benefiting from our efforts. It is one thing to talk of an intervention’s positive effect upon a group of students but it often misses the fact that some children will not respond as desired. If a child is one of a few for whom the practice is not working, failure just isn’t an acceptable option. The historic model of “wait-to-fail” requires that the student first fail before we use alternative methods to remediate a problem. Special education services have traditionally followed this paradigm. A student normally is referred for special education services only after experiencing extensive and extended occurrences of behavioral difficulty and/or a persistent and serious gaps between current academic performance and the child’s grade level. It was not until the reauthorization of the Individuals with Disabilities Education Improvement Act of 2004 (IDEA) that alternative models such as “response to intervention” received endorsement. Adequate, ongoing monitoring of a child’s response to interventions is central to the response-to-intervention approach.
Monitoring individual performance is the most effective method for early identification of those children not responding to a universal intervention. Focusing on the individual in progress monitoring connects students who do not benefit from a universal intervention with alternative evidence-based interventions that are better suited to that child.
Types of Progress Monitoring: This type of monitoring is focused on evaluations of individual student performance.
Student behavior: Commonly used methods to measure student behavior progress are:
- Frequency counts: counting the number of occurrences of a behavior
- Duration: recording the duration of occurrences
- Intensity: recording the intensity of an occurrence
Curriculum-based measurement (CBM): a tool for weekly measurements of student competency and progress in the basic skill areas against the content of the annual curriculum. If a child’s performance is not meeting expectations, changes can be initiated quickly to adjust the teaching methods to the student’s needs so that the annual academic goals are achieved.
- Pre-reading – measures phoneme segmentation fluency, letter sound fluency, letter name fluency, and nonsense word fluency
- Reading fluency - measures oral reading fluencyandmaze fluency
- Spelling
- Mathematics - measures computation and concepts and applications
- Written expression – measures word sequence, spelling, grammar, and semantics
Dynamic Indicators of Basic Early Literacy Skills (DIBELS): a set of standardized, individually administered measures of early literacy development. DIBELS consists of short (one-minute) fluency measures used to regularly monitor the development of pre-reading and early reading skills. Each measure has been thoroughly researched and demonstrated to be reliable, a valid indicator of early literacy development, and predictive of later reading proficiency; thus, aiding in the early identification of students who are not progressing as expected. The measures assess student development in the following areas:
- phonological awareness – the ability to identify the different sounds that make words and to associate these sounds with written words
- alphabetic understanding – facility in the use of the complete standardized set of letters or basic written symbols used in a language. Each letter roughly represents a phoneme of a spoken language
- automaticity: the ability to effortlessly complete everyday tasks with low interference of other simultaneous activities and without conscious thought to step-by-step process
- fluency with the code – the accuracy and speed of responding
Individual Education Program (IEP): an IEP describes the special education and related services specifically designed to meet the unique educational needs of a student with a disability. An IEP includes the following:
- Present Level of Educational Performance
- Annual Goal(s)
- Short-Term Objectives
- Evaluation Criterion
- Evaluation Frequency (Schedules)
- Evaluation Procedure
Report card: measures student progress in specific subject areas on a specified monthly, quarterly, semester, or annual schedule
Mean Length of Utterance (MLU): means of calculating the linguistic activity or proficiency of an individual by adding up the total number of words spoken and dividing by the number of times that the person has spoken
Whether the focus is on the group or the individual, monitoring is designed to answer the most important question of all: Is the Intervention Working?
| |
