Newsletter
Volume 31 | 2008 | Number 3
Evidence-Based Education: Can We Get There from Here?
By Dr. Ronnie Detrich, Ph.D., The Wing Institute
In the last decade the term “evidence-based” has become ubiquitous in education. This is largely a result of the emphasis on scientifically based instruction in No Child Left Behind (2001); however, it is also the result of a growing movement that began in medicine (Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000) and has spread to other disciplines in recent years. Education and related disciplines have been part of the discussion about evidence-based practices since the mid-1990s when the Society for the Study of School Psychology and various divisions within the American Psychological Association (APA) began developing mechanisms for validating interventions as evidence-based (Chambless et al., 1996; Kratochwill & Stoiber, 2000). In spite of the decade long emphasis on evidence-based education, as is often the case when there is rapid spread of a new term, there is still a great deal of confusion about what is meant by “evidence-based.”
Ultimately, the evidence-based practice movement is not about science but rather it is a consumer protection movement. It is essentially a public policy that emphasizes the benefits to consumers that can be gained with scientifically based interventions. This distinction has been a source of confusion as the evidence-based movement has progressed. Evidence-based practice is more than identifying effective practices through research. Evidence-based practice in medicine has been defined as “the integration of best research evidence with clinical expertise and patient values” (Sackett et al., 2000). APA has adopted a very similar statement as its definition (2005). The evidence-based practice movement is an effort to assure that scientific knowledge informs the practitioner’s decisions about interventions; however, it does not minimize the decision-making responsibility of the individual practitioner. Scientific knowledge functions as a filter in the selection of interventions but clinical expertise and client values also inform the ultimate form of the intervention.
Restricting the discussion, for the moment, to the research evidence, there is no consensus within education about how to best define evidence (Drake, Latimer, Leff, McHugo, & Burns, 2004). Clearly, what counts as evidence is an important matter for behavior analysis. In most of the existing standards for validating interventions as evidence-based, randomized clinical trials have been defined as the “gold standard.” The Institute for Education Science (IES) has clearly established a preference for randomized trials in the standard for the What Works Clearinghouse (What Works Clearinghouse, 2006) and has yet to establish standards for including single subject research in their systematic reviews of the literature. This leaves behavior analysis in a very difficult position given our reliance on single subject designs; we must either expand our research methods to include randomized trials or influence those who establish standards to assure that single subject designs are incorporated into systems for validating interventions as evidence based. Failure to do either of the above choices will likely result in behavior analysis becoming marginalized and consumers would become less likely to benefit from our interventions. Based on current standards, our research is often being left out of the evidence base.
Validating an intervention as evidence-based is more than publishing research demonstrating effectiveness. It requires a systematic review of all published research with a particular intervention or of all interventions in a specific area such as reading or adaptive social behavior. The literature is reviewed against a specific set of standards and any study that does not meet the standards is eliminated from the database. Once the final database is established, studies are reviewed to establish the strength of evidence for each study. It is likely that some studies are better designed and experimentally controlled than others. Once each article has been assigned a strength of evidence rating, the standards can be used to make statements about the evidence-base for a particular intervention. For example, in the What Works Clearinghouse (WWC) standards, there are three levels of rating. If there are two randomized clinical trials supporting an intervention and there are no contradictory studies, then the intervention will be validated as meeting evidence standards. If quasi-experimental designs are used, then the best rating an intervention can receive is meeting evidence standards with reservations regardless of the number of studies that have been reviewed. Even though WWC does consider single subject designs a form of quasi-experimental designs, it has not developed standards for reviewing them and as a consequence, single subject studies are not part of the review process.
One of the major obstacles to a successful evidence-based practice movement is the research to practice gap (Shriver & Watson, 2005), which has been a persistent problem across many disciplines. In addition to identifying the best available evidence for a particular problem, it is necessary to implement the intervention with sufficiently integrity so that the intervention is likely to be effective, and then to evaluate the extent to which the intervention is effective. Detrich, Keyworth, and States (2007) have proposed a roadmap for moving from efficacy research through progress monitoring as a means of assuring that appropriate evidence-based interventions are being selected, implemented with integrity, and the effects are evaluated with an individual consumer. The remainder of this paper will focus on the four components of the roadmap and evaluate the status of behavior analysis with each component.
The first question addressed in the roadmap is “What works?” At this point, the goal of research is to demonstrate a causal relationship. This type of research has been described as efficacy research (Chorpita, 2003). To make the strongest statement possible, it is often necessary to conduct the research under highly controlled conditions with well-trained implementers to minimize the effects of confounding variables. Behavior analysis conducts efficacy research very well. Our single subject methods permit us to make powerful demonstrations of causal relations between independent and dependent variables. This is the most common type of published research and is often reported to be a major contributor to the research to practice gap. Practitioners often view interventions established under efficacy conditions to be difficult to replicate in typical service settings because of a lack of resources, qualified personnel, and personal biases.
Once an intervention has been identified through efficacy research the question then becomes “When does it work?” At this stage of research, typically considered to be effectiveness research (Chorpita, 2003), the primary question shifts to which populations are likely to benefit from an intervention, what conditions are necessary for the intervention to be effective, and what is required to effectively implement the intervention. The primary concern with this type of research is the robustness of the intervention when implemented under less controlled conditions than were present during efficacy research, i.e., will this intervention be effective when implemented in typical service settings? Efficacy and effectiveness research are not mutually exclusive types of research but rather fall on a continuum in which highly controlled, laboratory-based efficacy research falls on one end and research in typical service settings with minimum supervision and guidance from researchers at the other. Some of the questions addressed by effectiveness research could be characterized as questions about external validity. Often these types of questions are best answered by group designs that are well suited for answering actuarial types of questions. In general, behavior analysis has not paid much attention to concerns about external validity. In part, this is the result of our basic epistemology, which places great emphasis on discovering general laws of behavior that apply to all behaving organisms.
Separate from the process of identifying evidence-based interventions is the process of implementing the intervention. The third component of the roadmap is implementation. In this phase, we are concerned with how to make an intervention work in typical service settings. Several scholars (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Schoenwald & Hoagwood, 2001) have commented on the process of implementation and have suggested that there are specific systems variables that are separate from the intervention that must be considered if implementation is to be successful. Failure to attend to these variables will likely compromise the ultimate effectiveness of the intervention. Behavior analysts working in education, with some notable exceptions (Sugai & Horner, 2005), have not written extensively about these systems variables; however, there is much to be learned about systems level implementation from our behavioral colleagues working in organizational behavior management.
Regardless of the strength of evidence for an intervention and the type of research (efficacy or effectiveness) that established the evidence base, no intervention will be effective for everyone. It is necessary to monitor progress to assure that a given individual is benefiting. Progress monitoring is the final component of the roadmap. Behavior analysis has made significant contributions to the science of progress monitoring. In fact, some have argued that the greatest contribution of behavioral approaches is the emphasis on the collection of behavioral data in natural settings (Johnson & Bolstad, 1973). At this level, the power of single subject designs is well recognized and frequently utilized. The response to intervention movement is fundamentally a progress monitoring approach to evaluating the impact of instruction on individual students.
The evidence-based practice movement presents many challenges for behavior analysts and in some instances will require a change in some of our behaviors, most notably adding randomized clinical trials to our methods for evaluating the effectiveness of interventions. In other instances, behavior analysis serves as a model for how practitioners should behave in an evidence-based world. Some have wondered why behavior analysts should become involved in the evidence-based movement because by definition, behavior analysis is evidence-based. Perhaps the best reason for doing so is that Kazdin (2000) reviewed the literature for interventions that were used to address problems of children with mental health issues and found over 500 named interventions. Of those interventions, less than 10% had been evaluated with any experimental research. A large majority of the reviewed interventions were behavioral or cognitive-behavioral. The individuals and families that we serve have a reasonable expectation that they are receiving services that are most likely to be effective. By requiring professionals to select from interventions that have a scientific basis, we are giving our consumers the greatest chance of benefit. This meets the socially important dimension of behavior analysis as outlined by Baer, Wolf, and Risley (1968).
References
- American Psychological Association. (2005, August). Policy Statement on Evidence-Based Practice. Retrieved June 18, 2007, from http://www2.apa.org/practice/ebpstatement.pdf
- Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of applied behavior analysis, 1(1), 91-97.
- Chambless, D. L., Sanderson, W. C., Shoham, V., Johnson, S. B., Pope, K. S., Crits-Christoph, P., et al. (1996). An update on empirically validated therapies. The Clinical Psychologist, 49(2), 5-18.
- Chorpita, B. F. (2003). The Frontier of Evidence-based Practice. In A. E. Kazdin & J. R. Weisz (Eds.), Evidence-based psychotherapies for children and adolescents. (pp. 42–59). New York: Guilford Press.
- Detrich, R., Keyworth, R., & States, J. (2007). A Roadmap to Evidence-Based Education: Building an Evidence-Based Culture. Journal of Evidence-Based Practices for Schools, 8(1), 26–44.
- Drake, R. E., Latimer, E. A., Leff, H. S., McHugo, G. J., & Burns, B. J. (2004). What is Evidence? Child and adolescent psychiatric clinics of North America, 13(4), 717-728.
- Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature (FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
- Johnson, S. M. & Bolstad, O. D. (1973). Methodological Issues in Naturalistic Observation: Some Problems and Solutions for Field Research. In L. A. Hamerlynck, L. C. Handy, & E. J. Mash (Eds.), Behavior change: Methodology, concepts, and practice. (pp. 7–67). Champaign, Ill: Research Press.
- Kazdin, A. E. (2000). Psychotherapy for Children and Adolescents : Directions for Research and Practice. New York: Oxford University Press.
- Kratochwill, T. R. & Stoiber, K. C. (2000). Empirically Supported Interventions and School Psychology: Conceptual and Practical Issues: Part Ii. School Psychology Quarterly, 15, 233–253.
- No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2001).
Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (Eds.). (2000). Evidence-based Medicine: How to Teach and Practice Ebm. Edinburgh, United Kingdom: Churchill Livingstone Inc. - Schoenwald, S. K. & Hoagwood, K. (2001). Effectiveness, Transportability, and Dissemination of Interventions: What Matters When? Psychiatric Services, 5 2(9), 1190–1197.
- Shriver, M. D. & Watson, T. S. (2005). Bridging the Great Divide: Linking Research to Practice in Scholarly Publications. Journal of Evidence-Based Practices for Schools, 6, 5-18.
- Sugai, G. M. & Horner, R. H. (2005). School-wide Positive Behavior Supports: Achieving and Sustaining Effective Learning Environments for All Students. In W. L. Heward, T. E. Heron, N. A. Neef, S. M. Peterson, D. M. Sainato, G. Cartledge, I. I. I. R. Gardner, L. D. Peterson, S. B. Hersh, & J. C. Dardig (Eds.), Focus on behavior analysis in education: Achievements, challenges, and opportunities. (pp. 90–102). Upper Saddle River, N.J: Pearson/Merrill/Prentice Hall.
- What Works Clearinghouse. (2006, September). What Works Clearinghouse Study Design Classification. Retrieved June 18, 2007, from www.whatworks.ed.gov/reviewprocess/studydesignclass.pdf