Their paper, “Assessing the Empirical Evidence of the National Adult Treatment Court Best Practice Standards (2nd Edition),” analyzes the claims and sources cited throughout the Incentives, Sanctions, and Service Adjustments (ISSA) section of the Adult Treatment Court Best Practice Standards (2nd edition).
The Adult Treatment Court Best Practice Standards
The National Association of Drug Court Professionals (NADCP; now All Rise) created the Adult Drug Court Best Practice Standards (the “Standards”) in 2013 and 2015 to professionalize the treatment court model and provide guidance on treatment court operations. In 2023, the Standards underwent major revisions, introducing new concepts such as psychosocial and clinical stability. The study’s authors reviewed the empirical evidence underlying the new standards to better understand the research base on which they were developed.
The authors used the evidentiary standard articulated by All Rise as a guide for assessing the research. The 2023 Standards stated, “As before, best practices were defined as services or interventions that have been proven through at least two high-quality experimental or quasi-experimental studies, meta-analyses, or quantitative systematic reviews to improve outcomes in treatment courts, other correctional rehabilitation programs, and/or substance use, mental health, or trauma treatment” (All Rise, 2023, p. 2).
Current Study
The study assessed the strength of the empirical evidence for 174 readings cited in support of 72 claims within the ISSA section. The key questions of the investigation were:
- What is the strength of empirical evidence supporting the claims and guidance provided in the 2nd edition’s ISSA standard?
- Does the strength of the empirical evidence cited in the ISSA standard differ between claims newly introduced in the 2nd edition and those retained from the first edition?
The study evaluated the strength of the evidence using the framework shown in Figure 1 below:
What the Authors Found
- 8.33% of the claims were rated as having strong empirical evidence, and 20.83% of the claims had moderate evidence.
- 30.56% of the claims had limited empirical evidence because only one study provided support for the claim.
- 40.28% had no empirical evidence to support the claim.
- Claims were often rated as having no or limited empirical evidence because: (1) the authors cited research but that research did not directly test the stated claim (i.e., no empirical research exists supporting the claim—instead one must infer a particular conclusion from a study, and that inference may be correct or incorrect), (2) the authors cited non-empirical evidence to support the claim (i.e., deviating from their definition of best practices), or (3) the authors failed to cite relevant research for that claim despite the existence of that research base in the literature (i.e., poor citation practices).
Stakeholders may confidently adopt claims that have moderate or strong support. A list of claims rated as having moderate or strong support can be found here. For claims supported by limited evidence, stakeholders should proceed cautiously, pairing implementation with monitoring to assess local impact and not requiring program adherence for funding.
Drs. Vaske, Baldwin, and Boman also found that 58.14% of claims found solely in the 2nd edition lacked empirical support, compared with 13.79% of claims found in both the 1st and 2nd editions. In contrast, 51.72% of claims found in both editions had moderate or strong support, compared to only 13.95% of claims found solely in the 2nd edition. The implication is that courts should rely on the first edition of Best Practices—rather than the second edition—to implement evidence-informed practices.
Why It Matters and Unintentional Consequences
Treatment courts rely heavily on these national standards to guide their policies, staff training, and program design. These standards help shape decisions that affect participant outcomes, policy, staff training, public safety, and the use of public resources.
Following a thorough review of the claims in the 2nd edition of the ISSA section, the authors identified the presence of ‘spin’ in how research was presented. Spin is defined as “specific reporting that could distort the interpretation of results and mislead readers” (Boutron et al., 2010, p. 2058). The type of spin found was inappropriate extrapolation, where cited research articles did not directly test the claims they supported. Out of the 51 claims with limited or no empirical support, 38 claims (74.51%) need more empirical evaluation—because there are either no studies or only one study supporting a specific claim—to determine if they constitute a best practice for the treatment court field.
The key takeaway from the review is that we still have several unanswered questions about effective behavior modification in treatment courts.
It is recommended that All Rise reevaluate the research cited for claims and reorganize the ISSA section to clearly identify claims supported by research with justice-involved populations, while labeling other claims as “Innovative” or “Emerging” practices that can be implemented under strict monitoring. Additionally, treatment court researchers should prioritize rigorous, context-sensitive studies to test the ISSA claims that lack empirical support.
What Next?
- The CRCP team will develop and disseminate practitioner-friendly products to help guide the application of the research findings from the current study.
- Organizations that specialize in training, technical assistance, and research on the treatment court model are encouraged to help practitioners understand which practices are evidence-based and which are innovative or emerging.
This post is the first step towards discussing the findings. There will be a series of posts, documents, and webinars to help treatment court professionals understand the state of the research and to help guide practice. Additionally, training and technical assistance are available to readers who are interested in learning more about the project. Please reach out to Sheila McCarthy (smccarthy@american.edu), the Program Director of Training and Technical Assistance, or Dr. Jamie Vaske (jborin@american.edu), the study’s first author, with any requests.
The authors have made their article open access and can be read here.
Boutron, I., Dutton, S., Ravaud, P., & Altman, D. G. (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA, 303(20), 2058-2064.
