Newsletter
Volume 31 | 2008 | Number 3
"Evidence-Based Practice:" Improvement or Illusion?
By Dr. Timothy Slocum, Ph.D., Utah State University
Evidence-based education has been defined as “the integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction” (Whitehurst, 2002). This suggests that in a system of evidence-based education, educators would make decisions based on the accumulated results of relevant scientific research along with their professional experience. It is a strong statement that research results should play a much more central role in education than they have previously and it raises numerous questions about how to build a stronger bridge between empirical evidence and educational practice. One of the key issues in strengthening this bridge is how research results are to be evaluated, summarized, and made available to educational professionals. As the Coalition for Evidence-Based Policy has written, “If practitioners have the tools to identify evidence-based interventions, they may be able to spark major improvements in their schools and, collectively, in American education.” (Coalition for Evidence-Based Policy, 2003, pg. iii).
In the past, supporters of particular educational practices have often abused the phrase “research tells us…” by following it with all manner of opinions and beliefs that have little or no support from legitimate scientific research. Such statements might be paired with a citation of opinion papers or low-quality research motivated and designed for advocacy than for actual science. The problem for educator is to sort through this ‘research’, to judge the rigor (internal validity) of the individual studies, judge the relevance of the participants, intervention, measures, and contexts to a particular practice situation (external validity or generalizability), and to evaluate the larger body of evidence to decide whether the practice has sufficient quality and quantity of research support to warrant implementation. This is a demanding task, even for those with extensive training and experience in research design and interpretation. It is not reasonable to suppose that many educational practitioners have the technical expertise and time to undertake such a task.
A second strategy for making the “best available empirical evidence” accessible to educators is to produce systematic reviews of literature that bring methodological rigor to the process of distilling the research literature and identifying effective practices. This approach is a key component of the movement for evidence-based practice in education. Slavin observed, “A key requirement for evidence-based policy is the existence of scientifically valid and readily interpretable syntheses of research on practical, replicable education programs. Educational policy cannot support the adoption of proven programs if there is no agreement on what they are” (2008, pg. 5). However, the process of planning and conducting systematic reviews of research literature is not a simple, non-controversial process. In addition, the stakes are high; the legitimacy and success of the entire evidence-based education movement depends on the effectiveness of the systematic review process. Reviewers must specify methods by which relevant literature is reviewed, consistent and appropriate standards for scientific quality are applied to individual studies, results are summarized across studies in a meaningful and valid manner, and clear statements about the level of empirical support for the practice are made.
Numerous organizations have begun to produce systematic reviews of educational practices offered as a basis for evidence-based practice, and the number of such organizations is quickly growing. These organizations vary in their funding sources, topics covered, systems for producing reviews, methodological standards for reviews, and other dimensions. Diversity of funding sources and focal topics demonstrate the widespread commitment to the supporting evidence-based practices in education across funding sources and organizations concerned with various facets of education. Diversity in systems and methods for producing reviews entails both strengths and weaknesses. It can be a strength in that research on educational practice is highly diverse and some variation is necessary to produce reviews that are relevant to each topic. For example, the research base on general education reading programs is very different than that on teaching language skills to children with autism – it would make sense that review standards should take these differences into account. In addition, systematic evidence-based practice reviews are a new form of literature review and some amount of variation is necessary for selection of more effective review practices. However, if these variations result in inconsistent recommendations for practice or the recommendations of practices that are ultimately ineffective, the legitimacy of the entire evidence-based education movement can be compromised.
The What Works Clearinghouse is a project of the U.S. Department of Education’s Institute for Educational Sciences (What Works Clearinghouse, n.d.). It has reviewed, or is currently reviewing the topics of (a) beginning reading, (b) early childhood education, (c) elementary school math, (d) middle school math, (e) English language learners, (f) character education, and (g) dropout prevention. The WWC sets the standard for explicit and transparent review process and methodology. It has a standard process for locating primary research, screening and evaluating studies, and deriving ratings of the evidence support for practices. In addition, a protocol is developed to specify important considerations and variables for each topic. WWC produces intervention reports that summarize and evaluate the research related to that intervention and topic reports that compile and integrate information on all reviewed practices within the topic area. Each intervention is rated according to a clearly defined six-point scale from positive effects to negative effects. All reviews of primary research and reports are produced by Clearinghouse staff. WWC reviews place great value on randomized controlled trials (RCTs), and secondary value on quasi-experimental designs (QED). It classifies single-subject research as a type of QED; however, it has not developed systems for evaluating and summarizing results of single-subject research. Therefore such research is not considered in the review process.
The Best Evidence Encyclopedia (BEE; Best Evidence Encyclopedia, n.d.) is funded by a grant from the Institute for Educational Sciences to Johns Hopkins University. It provides summaries of a collection of systematic reviews on math instruction, technology in reading and math, comprehensive school reform, reading for English language learners, and secondary reading. These reviews have been produced by BEE staff and others. To be included on the BEE Web site, reviews must meet explicit criteria including (a) carrying out an exhaustive search for all studies that meet well-justified standards of methodological quality and relevance to the issue being reviewed; (b) present quantitative summaries of effectiveness for achievement outcomes; and (c) focus on studies comparing programs to control groups, with random assignment to conditions or high-quality matching. These criteria are a good deal broader than WWC procedures giving more latitude to the reviewer to define specifics of the review process and more room for judgment about what constitutes high-quality research. The strength of evidence for a program is rated on a five-point scale from strong evidence through no qualifying studies. The criteria for these ratings vary across reports.
The Promising Practices Network (PPN; Promising Practices Network, n.d.) is a project of the RAND Corporation funded by multiple partner organizations. PPN considers a wider range of topics that WWC or BEE spanning the broad outcome areas of: (a) healthy and safe children, (b) children ready for school, (c) children succeeding in school, and (d) strong families. Within these areas, PPN attempts to identify “programs and practices that are proven to improve outcomes for children.” (Promising Practices Network, n.d.) Unlike WWC and BEE, PPN only makes statements about practices that have some level of research support (proven programs or promising programs), they do not list programs that are found to lack research support. A single high-quality study may be sufficient to earn a “proven” rating; however, criteria for weighing information across multiple studies are not explicit. PPN also includes a rating of “screened,” meaning that the practice has been favorably reviewed by “other credible organizations that apply similar evidence criteria.” (Promising Practices Network, n.d.) PPN does not produce topic reports with cohesive overviews or comparisons of the various interventions related to a particular topic, but their Web site allows searches that list all rated programs related to a particular outcome area, indicator, topic, or evidence level.
Other organizations provide evidence-based practice reviews in narrower areas related to particular populations or topics. The National Secondary Transition Technical Assistance Center (NSTTAC; National Secondary Transition Technical Assistance Center, n.d.) provides a good example of this kind of source of evidence-based practices. NSTTAC has produced reviews of practices in five areas related to secondary transition: (a) student focused planning, (b) student development, (c) interagency collaboration, (d) family involvement, and (e) program structure. Center staff review of specific practices within each of these topics; for example, practices such as “teaching functional life skills” and “teaching completing a job application” are reviewed under the topic of student development. These practices are defined at a more general level than the specific programs reviewed by WWC, BEE, and PPN – that is, the practice of “teaching functional life skills” is defined by a particular set of targeted outcomes rather than by a specific instructional program shown to produce these skills. NSTTAC rates the level of evidence for each practice on a four-point scale of strong, moderate, potential, and low. Ratings are based on explicit criteria derived from the indicators of quality research published by Gersten, et al (2005), Horner, et al (2005), and Thompson et al (2005), as well as from previously published literature reviews and meta-analyses. In addition, NSTTAC provides access to relevant evidence-based practice reviews produced by other organizations such as the What Works Transition Research Synthesis Project. NSTTAC is one of the few sources that explicitly recognizes and includes single subject research as an evidence base for practice.
The number of sources of evidence-based practice reviews has grown rapidly and will continue to increase. The National Autism Center will soon publish its National Standards Report, its review of evidence-based treatment approaches for autism (National Autism Center, n.d.). The Council for Exceptional Children is organizing a project to establish evidence standards and develop a process through which the organization will indentify evidence-based practices (Council for Exceptional Children, n.d.). As the number and variety of sources of evidence-based practice reviews grows, professionals will have access to a great deal of information regarding the research base for numerous practices; but they will also be faced with increasing challenges to navigate among sources and evaluate potentially conflicting claims regarding the strength of the evidence-base for specific practices. It is already clear that subtle differences in review procedures rules of evidence can produce substantially different recommendations. No single review system will flawlessly identify effective practices; all will include some rate of false positive and false negative results. A looming challenge is to evaluate the validity of evidence-based practice reviews.
References
- Best Evidence Encyclopedia. (n.d.) Retrieved July 2008 from http://www.bestevidence.org/.
- Coalition for Evidence-Based Policy (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly Guide. Retreived July 2008 from http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf.
- Council for Exceptional Children. (n.d.). Retrieved July 2008 from http://www.cec.sped.org/AM/Template.cfm?Section=Evidence_based_Practice/Template=/TaggedPage
TaggedPageDisplay.cfm/TPLID=24/ContentID=4710. - Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149-164.
- Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, & Wolery, M. (2005). The use of single subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.
- National Autism Center. (n.d.). Retrieved July 2008 from http://www.nationalautismcenter.org/.
- National Secondary Transition Technical Assistance Center. (n.d.). Retrieved July 2008 from http://www.nsttac.org/?FileName=evidence_based_practices.
- Promising Practices Network (n.d.). Retrieved July 2008 from http://www.promisingpractices.net/.
- Slavin, R. E. (2008). What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.
- Thompson, B., Diamond, K. E., McWilliam, R., Snyder, P., & Snyder, S. W. (2005). Evaluating the quality of evidence from correlational research evidence-based practice. Exceptional Children, 71, 181-194.
- What Works Clearninghouse. (n.d.). Retrieved July 2008 from http://ies.ed.gov/ncee/wwc/.
- Whitehurst, G. J. (2002, October). Evidence-Based Education. Paper presented at the Office of Elementary and Secondary Education’s School Achievement and School Accountability Conference, Orlando, FL. Retrieved July 2008 from http://www.ed.gov/nclb/methods/whatworks/eb/edlite-index.html.