Newsletter
Volume 31 | 2008 | Number 3
Critical Issues in the Use of Randomized Clinical Trials and Control Groups Within Applied Settings: Rationale, Challenges, and Benefits
By Dr. Hill M. Walker, Ph.D., Center on Human Development
School districts and educators have recently come under intense pressures to accommodate the use of randomized control trials (RCTs) due primarily to the public demand for access to evidence-based interventions that can make a difference in the school success of all students, and especially those who may be marginalized and at-risk for destructive outcomes. Randomized control trials are considered the gold standard in terms of levels of scientific evidence in regards to evaluating the efficacy and effectiveness of interventions. RCTs have a long history of effective usage in fields such as medicine and psychology but this has not been the case in education. However, since schools are so well suited for the conduct of large-scale research in terms of providing broad access to child-youth populations, having a cadre of professionals who can serve intervention delivery roles (e.g. teachers, counselors, school psychologists, behavioral specialists), and which are expected to produce important social and academic outcomes for all students, they are increasingly seen by researchers as relatively ideal laboratory-type settings for testing interventions and new practices.
In spite of its suitability as a research site, the typical school district usually provides a number of challenges to the appropriate use of RCTs. For example, educators often have a visceral dislike for control groups and conditions where certain students are denied access to a new treatment or practice. RCTs with waitlist control groups are often invoked as a compromise for this dilemma; however, these designs do not allow for control conditions in follow-up assessments as the control group is provided access to the intervention immediately following exposure to it of the treatment group and the recording of post assessments. Further, the demands of random assignment create problems for both educators and parents who often terminate their study involvement when they learn they or their children have been assigned to a control group. Finally, the logistics and complex politics of school districts frequently result in demands for compromise of school-based, RCT study designs that reduce their integrity.
On balance, it is far easier to conduct single subject research within the context of schooling than it is to implement larger scale RCTs. However, both types of research are ultimately essential for developing and testing evidence-based interventions and practices in which parents and school professionals can have confidence. The purpose of this keynote is to address some of the critical issues and challenges involved in the use of randomized control trials within school settings. The following topics are discussed herein: (a) randomized control trials—definitions and key characteristics, (b) advantages and disadvantages of RCTs, (c) single subject vs. RCT designs in schools, (d) integrating single subject and RCT designs in a 3-stage R & D process, and (e) criteria used to classify evidence-based interventions and practice approaches.
Definitions and Key Characteristics of Randomized Control Trials
RCTs are typically large-scale studies that randomly assign individuals to an intervention or control group in order to measure the positive or negative effects of the intervention. RCTs are considered the surest way to avoid making serious mistakes on health, policy, and education issues that can affect the public well being and safety. Some seminal medical and education findings that have been validated by RCTs include vaccines, treatments for hypertension, cholesterol, and various forms of cancer; key educational findings established via RCTs are reduced class sizes in K-3 grades, reading and the role of phonemic awareness, and the impact of high quality early childhood development programs.
Advantages and Disadvantages of RCTs
RCTS provide the highest level of evidence that a treatment works or doesn’t work. By testing outcomes against a randomly selected control group that provides a measure of ongoing, normal conditions in the treatment environment, RCTS create confidence that observed changes in the experimental group are due to the treatment and not to extraneous factors. The greatest disadvantages of RCTs are a) the time and effort involved in their effective implementation and b) dealing with educator or parental resistance and demands for compromises to the RCT design during implementation.
Single Subject Vs. RCT Designs Within Schools
Both types of research design have key roles to play in establishing the validity of potentially effective interventions. Single subject designs are regarded by many as invaluable in measuring and explaining sources of variability in behavior or performance while RCTs focus on controlling such variations statistically. Single subject designs have been used to good effect in identifying and testing the initial efficacy of treatment variables or components that appear promising for possible inclusion within a larger intervention package or approach. Single subject designs are noted for their strong internal validity.
In contrast to single subject designs, RCTs have stronger external validity due to the range of subject characteristics sampled by the random assignment of large numbers of participants to experimental and control conditions which, in turn, allows for broader generalization of treatment outcomes. RCTs also allow the calculation of effect sizes that provide an estimate of the magnitude of the achieved treatment effect. They also provide for subgroup analyses that contrast the performance of clusters of participants that share certain characteristics (e.g. gender, ethnic or racial background, disorder, etc.).
As a general rule, single subject designs are much more acceptable to educators due to their relative lack of complexity, their cost efficiency, the minimal intrusiveness of their implementation, and the fact that they do not require control groups or random assignment. However, while these designs allow study of the daily interactions of treatment and dependent variables, one disadvantage is that some educators object to the withdrawal of treatment for the purpose of establishing causal relationships.
Integrating Single Subject and RCT Designs in a 3-Stage Research and Development Process
Single subject designs are most efficiently utilized in the early or initial stages of an R & D process leading to development of a complex treatment package or intervention in order to (a) identify promising variables for inclusion in the larger package, (b) to test potential variables with diverse participant groups, and (c) to evaluate a small scale, prototype of the larger intervention. RCT group designs can then be used to aggregate treatment components into a larger package and test its efficacy against a control group or condition.
The author and his colleagues have used a three-stage model of this type to develop and test four behavior management packages—one each for the following behavior disorders: Acting Out Behavior, Peer Aggression, Social Withdrawal, and Academic Survival Skills. Stage one involved one year of research within a tightly controlled experimental classroom setting where daily interaction of treatment and dependent variables could be inspected. Stage two involved research within regular classroom and playground settings and generally involved two academic years of research and development. Stage three was focused on field testing in settings across the country in which we trained school personnel in school districts to implement the intervention under supervision of the developers in order to test its efficacy. Typically this model did not allow for testing the effectiveness of the intervention package and required one school year for implementation.
Criteria Used to Classify Evidence-Based Intervention and Practice Approaches
This section reviews the levels of evidence for evidence-based practices that are commonly used in scientific research. There are five evidence levels, each with specific indicators that range from Best Support through Known Risks and side effects associated with the intervention. Several sources are provided for the reader’s convenience at the end of the ppt. along with instructions for accessing the presentation from the Web site of the Institute for Violence and Destructive Behavior at the University of Oregon.