Newsletter
Volume 31 | 2008 | Number 3
The Role of Single Subject Research Design in Establishing Evidence-Based Practice Guidelines
By Dr. Susan Wilczynski, BCBA, National Autism Center
Evidence-based practice has become a very popular trend in most fields of health, service provision, and education. Although the term remains the same, the definition and procedures associated with different evidence-based practice guidelines vary. Often, differences in the definition and methodologies employed can make a tremendous difference in reported outcomes. Consumers must be aware of the definitions and methodologies employed in each evidence-based practice guideline to understand the strength and limitations of each evidence-based practice guideline.
During the last three years, the National Autism Center has spearheaded an effort to establish an evidence-based practice guideline that systematically examines the strength of evidence for a broad range of educational and behavioral interventions for individuals with Autism Spectrum Disorders under the age of 22. For the purposes of the National Standards Project, evidence-based practice is defined as the integration of research findings with (a) professional judgment and data-based clinical decision-making, (b) values and preferences of families and students with ASD, and (c) capacity to implement interventions with a high degree of treatment integrity. Each of these components should be given due consideration as treatments are selected and implemented.
The remainder of this presentation focuses predominantly on the methodologies used in the National Standards Project. These methodologies are used to establish the strength of evidence for educational and behavioral treatments so that educators and other practitioners will have a concise resource describing the research findings that should contribute to evidence-based practice. However, this is not to suggest that the remaining components of evidence-based practice should be ignored or undermined.
Researchers have choices to make about the research designs they employ when answering research questions. Single subject research designs readily lend themselves to educational environments and are widely used to examine the effectiveness of educational interventions. It is unfortunate that they have been so widely ignored in the evidence-based practice movement. The National Standards Project involved the review of over 675 empirical articles on the treatment of Autism Spectrum Disorders. The vast majority of these studies relied on single subject research design. Like group research designs, single subject research design can be conducted with sufficient rigor to draw firm conclusions about treatment effectiveness. Similarly, weak group and single-case research designs have appeared in abundance in the scientific literature. Historically, evidence-based practice guidelines have restricted their reviews to studies employing group research designs. Given the fact that group and single subject research design each have advantages and disadvantages as well as strong and weak design, the exclusion of studies employing single subject research design is not warranted.
The National Standards Project established separate criteria for group and single subject research design. Each article was reviewed with respect to five dimensions: Research Design, Dependent Variable, Independent Variable, Participant Ascertainment, and Generalization. Scores on these dimensions were combined to produce a total score, the Scientific Merit Rating Scale score. In addition, all studies were coded to identify treatment effects. Those dimensions of the Scientific Merit Rating Scale and the Treatment Effects rating that directly relate to single-case research design are explored in further detail.
Research Design
The quality of research design that is used to establish a relationship between the independent and dependent variables is given the strongest weighting in the National Standards Project. Research design reflects to the extent to which experimental control was demonstrated in a given study. Studies that received higher ratings demonstrated more direct comparisons of the experimental and control conditions. Without sufficient replication of effects within a study, it is impossible to draw firm conclusions about treatment effects. In addition, in order to determine if differences across conditions exist, a sufficient number of data points must be collected per condition. Further, results must be established across multiple participants. Finally, in order to be most confident that true differences resulting from the application of experimental and control conditions exist, data loss should not occur.
Dependent Variable
The quality of the dependent variable is based on the degree to which accurate and reliable data were collected. Ideally, these data should represent a direct and comprehensive sample of the target behavior possible. Five factors influence the quality of the dependent variable. First, the quality of the dependent variable is influenced by the type of data collected. Specifically, whether the data collected were continuous or discontinuous was evaluated. In addition, the level of interobserver agreement influences the dependent variable rating. Without interobserver agreement estimates, it can not be known if a study reliably measured change across conditions. In addition, sufficient interobserver agreement requires that data are collected across a consider percentage of session and, ideally, that data are collected across multiple conditions. Finally, the quality of the dependent variable can be influenced by the extent to which the dependent variable was operationally defined in the article.
Independent Variable
In recent years, researchers have increasingly recognized the importance of establishing treatment fidelity. The quality of the treatment fidelity describes the degree to which procedural accuracy was established in a study. First, data must be collected to demonstrate that the procedures have been implemented as reported in the study. Second, data need to be collected during an adequate number of sessions. Third, interobserver agreement data for treatment fidelity are ideally reported. Finally, the degree to which the control and experimental conditions are operationally defined can influence our confidence that the treatments could be implemented with a high degree of integrity.
Participant Ascertainment
The quality of participant ascertainment is determined by examining the extent to which well-established diagnostic tools and procedures are used to determine participant eligibility for inclusion in a study. Participant ascertainment is influenced, in part, by the involvement of a qualified professional in the diagnosis and/or confirmation of a diagnosis of an Autism Spectrum Disorder. Further, whether or not evaluators and independent and/or blind to the experimental conditions to which the participants are assigned can influence our confidence.
Generalization
For the purpose of examining the scientific merit of each study with respect to individuals with Autism Spectrum Disorders in the National Standards Project, we defined generalization as the degree to which researchers attempted to objectively demonstrate the spread of treatment effects across time, settings, stimuli, and persons. Separate data were also recorded elsewhere to determine if treatment effects were generalized when establishing a treatment had the strongest level of research support. However, the extent to which objective data were collected to measure if a skill was maintained and/or generalized was the primary focus when establishing the scientific merit for our intended purposes. Of note, measurement of generalization occurred with respect to the spread of treatment effects across stimuli, settings, and/or persons.
The Scientific Merit Rating Scale combines each of the dimensions described here. However, the Scientific Merit Rating Scale score does not reflect the treatment effects; rather, the Scientific Merit Rating Scale score reflects the extent to which the scientific community can draw firm conclusions about treatment effects specific to the Autism Spectrum Disorders population.
Treatment Effects
Once each article was assigned a SMRS score, it was further classified as having: (a) Beneficial Treatment Effects Reported, (b) Unknown Treatment Effects, (c) No Treatment Effects Reported, or (d) Adverse Treatment Effects Reported. Beneficial Treatment Effects are reported when sufficient evidence has been provided to demonstrate a treatment produced a positive outcome. Unknown Treatment Effects are reported when, given the research design and the quality of the data, it is impossible to ascertain what the treatment outcomes mean. No Treatment Effects are reported when sufficient evidence is available to show the absence of beneficial treatment effects is meaningful. Adverse Treatment Effects are reported when sufficient evidence has been provided to indicate a treatment has resulted in greater deficit or harm to participants. When the Scientific Merit Rating Scale and Treatment Effects are combined for each study and the results of all studies for a given treatment are aggregated, the strength of evidence regarding a treatment can be determined. However, these studies may reflect a combination of group and single-case research design. It is noteworthy, however, that more studies employing single subject research designs are required to obtain the same score as do those employing group design. This reflects the need for sufficient representation of participants and issues of validity when single subject research design is used.
All educational and behavioral autism treatment studies could not be included in the National Standards Project. One of the primary reasons a single subject research design study could not be included was because it did not rely on linear graphical presentation of data (e.g., bar graphs may have been used to display data). The advantages and disadvantages of this decision are discussed. Further, the importance of practitioners recognizing decisions like this one impact the final outcomes of evidence-based practice guidelines is considered. For this reason, practitioners should take an active role when utilizing evidence-based practice guidelines. That is, evidence-based practice guidelines should never be used following a ‘cookbook’ method and practitioners should become familiar with information beyond the documents that might supplement their selection of a given treatment. Irrespective of the definitions and methodologies employed, practitioners must vigorously consider multiple sources of data presented in the evidence-based practice guidelines, direct empirical investigations, and the clients whom they serve.