|Measurement of Validity in Behavior-Analytic Research: Procedural Integrity and Procedural Acceptability Reporting in Behavior Analysis in Practice and the Journal of Organizational Behavior Management|
|Sunday, May 29, 2022|
|9:00 AM–9:50 AM |
|Meeting Level 1; Room 153C|
|Area: OBM; Domain: Theory|
|Chair: Nicholas Matey (University of Florida )|
|CE Instructor: Nicholas Matey, M.S.|
In Applied Behavior Analysis (ABA) research, ensuring the validity of the intervention is a critical concern. Thus, reporting various types of validity data (e.g., internal, social) is strongly encouraged by most ABA publications. However, frequency and scope of these reports has not been widely measured in some journals. Procedural integrity, also referred to as treatment integrity, describes how well an independent variable was implemented and represents an index of confidence in the relationship between the independent and dependent variables (i.e., internal validity). Social validity measures, such as procedural acceptability, assess the degree to which the intervention is important to consumers and hints at variables that might lead to lasting change and adoption. The present symposium reports trends in procedural integrity and procedural acceptability reporting within two behavior analytic journals. The three presentations comprise data on 1) treatment integrity reporting in Behavior Analysis in Practice, 2) procedural acceptability reporting in Journal of Organizational Behavior Management (JOBM), and 3) procedural integrity reporting in JOBM. Each presentation will highlight current and historical publication trends, as well as discussion of potential variables contributing to variance in measurement and reporting of various validity data. In addition, future directions for research practices will be discussed.
|Instruction Level: Advanced|
|Keyword(s): procedural acceptability, procedural integrity, social validity, treatment integrity|
|Target Audience: |
Researchers whose projects are related to ABA and subdisciplines (e.g., Organizational Behavior Management; OBM), as well as practitioners, supervisors, and managers who regularly consume behavior analytic research to inform their daily practice.
|Learning Objectives: At the conclusion of the presentation participants will be able to (1) describe common measurement practices for procedural integrity/treatment integrity and procedural acceptability; (2) identify current trends in behavior analytic research regarding measurement of procedural integrity/treatment integrity and procedural acceptability; (3) name potential variables that might deter or enhance regularity in researchers' reporting of integrity and acceptability measures, as well as future areas for explication in behavior analytic research.|
|Treatment Integrity Reporting in Behavior Analysis in Practice 2008-2019|
|LEA JONES (California State University, Sacramento), Galan Falakfarsa (California State University, Sacramento), Denys Brand (California State University, Sacramento), Erik Swanson Godinez (California State University, Sacramento), Deborah Christine Richardson (California State University, Sacramento), ROBBIE HANSON (Lindenwood University), Savannah Velazquez (California State University, Sacramento), Colin Wills (California State University, Sacramento)|
|Abstract: Treatment integrity (TI) is the extent to which procedures are implemented in a manner consistent with their prescribed protocols and is necessary for reaching accurate conclusions regarding the functional relationships between dependent (i.e., behavior) and independent (i.e., environment) variables. Several studies have examined the frequency of TI data reporting in behavior analytic journals. However, no review has included articles from Behavior Analysis in Practice. Thus, the current study reviewed articles in Behavior Analysis in Practice between 2008 and 2019 to assess the frequency of studies reporting TI data. A total of 193 articles consisting of 205 studies met the inclusionary criteria for this review. Ninety-six studies (46.83%) reported TI data, compared to 193 studies (94.15%) that reported interobserver agreement (IOA) data. In addition, 98 studies (47.80%) were considered at high risk for treatment implementation inaccuracies. More research is needed to determine the exact reasons why TI data are not more frequently reported across behavior analytic journals.|
Has Organizational Behavior Management Found Its Heart? An Assessment of Procedural Acceptability Trends in the Journal of Organizational Behavior Management
|JESSICA NASTASI (University of Florida), Davis Simmons (University of Florida), Nicole Gravina (University of Florida)|
Procedural acceptability measures can be used to inform and improve the long-term viability of interventions in Organizational Behavior Management (OBM). However, little is known regarding the use of procedural acceptability assessments across studies employing OBM methodology. In the present review, we evaluated the use of procedural acceptability measures across all articles in the Journal of Organizational Behavior Management (JOBM) for the first decade (1977–1986) and the most recent decade (2010 to 2019). We coded each article for industry, organizational performance problem type, participant type and reported use of procedural acceptability measures. Formal procedural acceptability measures were reported in 20% of articles included from the first decade and 35% of articles included from the most recent decade of JOBM. The use of procedural acceptability measures appears to be on an increasing trend, but the frequency of reported use of acceptability measures differed across industries. Furthermore, most articles included limited information on how acceptability was assessed. Unique considerations for the use of procedural acceptability measures in OBM and recommendations are discussed.
Procedural Integrity Reporting in the Journal of Organizational Behavior Management, 2000-2020
|DANIEL J CYMBAL (Florida Institute of Technology), David A. Wilder (Florida Institute of Technology), Nelmar Jacinto Cruz (Florida Institute of Technology), Ronald J. Clark (Florida Institute of Technology), Grant Michael Ingram (Florida Institute of Technology), Marissa E. Kamlowsky (The University of Kansas)|
In behavior analytic research, procedural integrity refers to the extent to which the independent variable is implemented as described. Collecting and reporting data on procedural integrity is important for assessing the internal validity of a study; it assists in verifying that the independent variable, and not an extraneous variable, is responsible for intervention effects. Previous research suggests that data on procedural integrity are infrequently reported in behavior analytic studies. In organizational behavior management in particular, no recent evaluation of the reporting of data on procedural integrity exists. In the current study, we examined all empirical articles published in the Journal of Organizational Behavior Management (JOBM) from 2000 through 2020 to examine reporting of data on procedural integrity. We found that only 23.7% of studies reported these data. Furthermore, we found that 43.8% of studies appear to be at high risk, meaning they included multiple person-implemented intervention components and no measure of procedural integrity. We conclude by offering some possible reasons as to why the number of JOBM studies reporting these data is so low and by suggesting some ways to increase the collection and reporting of procedural integrity data.