Newsletter
Volume 31 | 2008 | Number 3
"Evidence-Based Practice:" Improvement or Illusion?
By Dr. Gina Green, BCBA, San Diego State University
Several years ago the field of medicine began to recognize that many medical practitioners were not using treatments for some common conditions that research had shown to be the most effective. That led to concerted efforts to disseminate the best available evidence about treatment effectiveness to medical practitioners, and to train practicing physicians to use research evidence and scientific methods to guide their everyday decision-making. Eventually other professions began to follow medicine’s lead, until evidence-based practice (EBP) became a significant social movement that continues to pick up steam today. Mandates to base practices on research evidence are even appearing in some laws and regulations, including federal education laws in the United States.
At first blush, the EBP movement would appear hold great promise for improving services to people with autism spectrum disorders (ASD). During the 50+ years since autism was first described, hundreds of treatments have been said to be beneficial for people who have that diagnosis. (“Treatment” is used here to mean any intervention, therapy, procedure, or program that may affect the behavior and/or biology of people with ASD). Many of those treatments have been very popular, and large amounts of money, time, and effort have been invested in them. Relatively few, however, have been evaluated in the types of studies that are required to produce credible evidence about treatment effects; that is, controlled experiments in which a treatment is systematically compared to no treatment or another treatment, study participants are selected and assessed carefully, good and bad effects are measured directly and objectively, multiple researchers repeat the experiments and get similar results, and reports of the studies are published in peer-reviewed scientific journals. Some treatments for ASD that have been claimed to be very effective have proved ineffective or harmful when they were evaluated in scientific studies, yet they remain in widespread use, supplanting treatments with proven benefits. It would seem that encouraging or requiring practitioners to use evidence-based methods would reduce harms and wastefulness and result in better outcomes for people with ASD, not to mention large cost savings. That would be the case if “evidence-based” always meant that the methods had proved effective in controlled scientific studies. But that term is now being used to market many practices that have not been evaluated scientifically; in fact, it seems that virtually every practice that is being promoted for people with ASD these days is claimed to be “evidence-based.” This means that in order to make fully informed decisions, it is necessary for consumers, practitioners, and funding agencies to know how to distinguish reliable from unreliable evidence, and treatments that have proved effective in sound scientific studies from those that are based on other types of evidence.
There are a number of protocols for developing evidence-based treatment guidelines. The strongest have several key features: (a) procedures for assembling teams of expert reviewers and methodologists to develop the guidelines; (b) criteria for identifying credible scientific studies published in peer-reviewed professional journals; (c) standards for evaluating the scientific merit of every identified study (i.e., the adequacy of the research design, measurement methods, participant selection and assessment methods, treatment delivery, data analysis methods); (d) procedures for training reviewers to evaluate studies; (e) cross-checks on reviewers’ evaluations by other independent reviewers; (f) procedures for compiling the evaluations of all studies on a particular treatment to determine the strength of the evidence for or against that treatment; (g) procedures for developing practice guidelines based on the scientific evidence, supplemented by expert clinical judgment; (h) peer review of the guidelines. Those protocols define the most credible evidence as the results of controlled experiments in which direct, objective, accurate, and reliable methods were used to measure the effects of a treatment and the extent to which the treatment was delivered as it was intended, with replications by multiple investigators. Treatments for which there is strong scientific evidence of effectiveness are recommended. Those for which there is weaker evidence may be identified as “promising” or “emerging” and worthy of further research, while those for which there is clear evidence of ineffectiveness or negative effects and treatments that have not been evaluated adequately are not recommended. A well-developed protocol with many of the features just listed was used to develop Clinical Practice Guidelines for Young Children with Autism for the New York Department of Health Early Intervention Program in 1999.
The evidence about many popular treatments for ASD falls short of the standards used in the best EBP protocols. Examples include: personal accounts and testimonials of people who say they have ASD or their family members; studies that used indirect or subjective methods to evaluate treatment effects (e.g., questionnaires, checklists, and rating scales completed by parents or other caregivers); uncontrolled, descriptive studies (i.e., those with no control group or condition); assessment reports with no direct testing of treatment methods; studies conducted by the developers or principal promoters of a treatment, their students or close colleagues, with no replications by independent researchers; studies that did not include people with ASD; theories, interpretations, and opinions that have not been tested in controlled studies; and reports that have not been peer reviewed (such as self-published articles, books and journals, many conference presentations, marketing materials, and the like). The effectiveness of treatments that are backed up mainly or exclusively by these forms of evidence is most accurately described as “unknown.” Calling them “evidence-based” creates the illusion that they have the same status as treatments that have been evaluated in multiple controlled experiments.
For behavior analytic practitioners and researchers, the EBP movement poses particular challenges as well as great opportunities. One of the challenges is to make sure that behavior analytic practices are supported by bona fide scientific evidence as defined in good EBP protocols. The principles of behavior analysis are natural laws of behavior that were discovered through careful experimental analyses conducted by many scientists. But it does not follow that every method or technique that is said to be based on those principles is effective, nor that every practice that is said to be behavior analytic is “evidence-based.” There are only a few principles of behavior, but there are many ways to interpret and apply them. Whether any given application is effective is a question that must be addressed in controlled experiments using objective measurement methods. Many behavior analytic techniques and programs that are being promoted for people with ASD have not been evaluated carefully, so there are many questions that need to be addressed by both basic and applied researchers. Another challenge for practitioners of behavior analysis working in ASD is keeping abreast of research on non-behavior analytic interventions in order to comply with ethical mandates to use and recommend methods that have proved most effective in scientific studies. For instance, many interventions for ASD that are described as “biomedical” and “sensory” have not been tested scientifically, or have proved ineffective or harmful in scientific studies. Therefore, practitioners credentialed by the Behavior Analyst Certification Board (BACB) who use or endorse those interventions are not in compliance with the BACB Guidelines for Responsible Conduct.
For the field of behavior analysis, a major challenge is posed by the fact that many EBP protocols limit the definition of scientific research to studies using between-groups research designs with participants assigned randomly to treatment and control groups, where treatment effectiveness is determined by comparing the average scores of those groups statistically (randomized clinical trials). Behavior analytic research methods and many findings from behavior analytic studies are excluded entirely, or minimized. On the other hand, as the evidence from between-groups studies of early intensive behavior analytic intervention for ASD has mounted, so has pressure on funding sources and developers of EBP guidelines to acknowledge the evidence from within-subject, single-case design studies of other behavior analytic interventions. Therefore, the EBP movement may afford an unprecedented opportunity to gain wider acceptance of behavior analytic research methods and findings by professionals from other disciplines and the general public. Behavior analysts must act promptly but wisely to take advantage of this opportunity. Tactics might include developing standards and criteria for evaluating behavior analytic studies that parallel those used to evaluate between-groups studies and are consistent with the science of behavior analysis, working to get behavior analytic research methods included in EBP protocols, and revising EBP guidelines that have excluded behavior analytic research. Some efforts along these lines are underway, but much work remains to be done.
References
- Agency for Healthcare Research and Quality. http://www.ahrq.gov/clinic/epcix.htm
- Carr, J.E, & Firth, A.M. (2005). The Verbal Behavior approach to early and intensive behavioral intervention for autism: A call for additional empirical support. Journal of Early Intensive Behavioral Intervention , 2, 18-27.
- Dawson, G., & Watling, R. (2000). Interventions to facilitate auditory, visual, and motor integration in autism: A review of the evidence. Journal of Autism and Developmental Disorders, 30, 415-421.
- Ellis, C., Singh, N.N., & Ruane, A.L. (1999). Nutritional, dietary, and hormonal treatments for individuals with mental retardation and developmental disabilities. Mental Retardation and Developmental Disabilities Research Reviews, 5, 335-341.
- Evidence-Based Medicine Working Group (1992). Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, 268, 2420-2425.
- Green, G. (1996). Evaluating claims about treatments for autism. In C. Maurice (Ed.), G. Green, & S.C. Luce (Co-eds.), Behavioral intervention for young children with autism: A manual for parents and professionals (pp. 15-28). Austin, TX: PRO-ED.
- Jacobson, J.W., Foxx, R.M., & Mulick, J.A. (2005) Controversial therapies for developmental disabilities. Mahwah, NJ: Erlbaum.
- Johnston, J.M. & Pennypacker, H.S. (1993). Strategies and tactics of behavioral research (2nd Ed.). Hillsdale, NJ: Lawrence Erlbaum.
- Levy, S.E., & Hyman, S.L. (2005). Novel treatments for autistic spectrum disorders. Mental Retardation and Developmental Disabilities Research Reviews, 11, 131-142.
- Malone, R.P., Gratz, S.S., Delaney, M.A., & Hyman, S.B. (2006). Advances in drug treatments for children and adolescents with autism and other pervasive developmental disorders. CNS Drugs, 19, 923-934.
- Matson, J.L., Benavidez, D.A., Compton, L.S., Paclawskyj, T., & Baglio, C. (1996). Behavioral treatment of autistic persons: A review of research from 1980 to the present. Research in Developmental Disabilities, 17, 433-465.
- Metz, B., Mulick, J.A., & Butter, E.M. (2005). Autism: A late-20th-century fad magnet. In J.W. Jacobson, R.M. Foxx, & J.A. Mulick (Eds.), Controversial therapies for developmental disabilities (pp. 237-263). Mahwah, NJ: Erlbaum.
- Myers, S.M., Johnson, C.P, & The Council on Children with Disabilities (2007). Management of children with autism spectrum disorders. Pediatrics, 120 (5), 1162-1182.
- New York State Department of Health Early Intervention Program (1999). Clinical Disorders-- Assessment and Intervention for Young Children (Age 0-3 Years). Health Education Services, P.O. Box 7126, Albany, NY 12224 (1999 Publication No. 4216).
- Romanczyk, R.G., Arnstein, L., Soorya, L.V., & Gillis, J. (2003). The myriad of controversial treatments for autism. In S.O. Lilienfeld, S.J. Lynn, & J.M. Lohr (Eds.), Science and pseudoscience in clinical psychology. (pp. 363-395). New York: The Guilford Press.
- Sidman, M. (1960). Tactics of scientific research. New York: Basic Books.
- Smith, T. (1996). Are other treatments effective? In C. Maurice, G. Green, & S. Luce (Eds.), Behavioral Intervention for young children with autism: A manual for parents and professionals (pp 45-59). Austin, TX: PRO-ED.
- Smith, T. (1999). Outcome of early intervention for children with autism. Clinical Psychology: Science and Practice, 6, 33-49.
- West, S., King, V., Carey, T.S., Lohr, K.N., McKoy, N., Sutton, S.F., & Lux, L. (2002). Systems to rate the strength of scientific evidence (AHQR Publication No. 02-E016). Rockville, MD: Agency for Healthcare Research and Quality. Available from http://www.ahrq.gov/clinic/evrptfiles.htm#strength