INSIGHTS
Methodological concerns in psychedelic research: The issues of nonequivalent psychological support and generalizability
Author links open overlay panelRichard J. Zeifman a, Lucas O. Maia b
www.sciencedirect.com/science/article/pii/S0924977X23006867A growing body of clinical trials has presented compelling evidence for psychedelic therapy's therapeutic potential (Mitchell and Anderson, 2023), including psilocybin therapy for major depressive disorder (e.g., Raison et al., 2023; for a meta-analysis, see Perez et al., 2023) and MDMA-assisted psychotherapy for posttraumatic stress disorder (e.g., Mitchell et al., 2021). However, there remain important methodological concerns with psychedelic research that warrant attention. While functional unblinding is a frequently discussed concern (Muthukumaraswamy et al., 2021), the present manuscript focuses on two additional methodological limitations: (1) potential nonequivalence in quantity (frequency and duration) and quality of psychological support (or psychotherapy) within psychedelic trials (threats to internal validity) and (2) issues with generalizability due to potential sample selection bias (a threat to external validity). To enhance the validity of psychedelic research, we highlight and provide recommendations for addressing these concerns.
Psychedelic therapy involves the administration of a pharmacological agent alongside psychological support. To establish the efficacy of a pharmacological agent within placebo-controlled trials, it is essential that the quantity and quality of the psychological support provided are equivalent across conditions. Although psychedelic trials initially assign participants an equivalent quantity of psychological support, several elements may contribute to individuals in the active condition receiving more psychological support. Psychedelic therapy protocols often indicate that patients, therapists, and investigators may initiate additional psychotherapy sessions when necessary. Several MDMA-assisted therapy trials for posttraumatic stress disorder have noted adding additional sessions and indicated that participants in the MDMA condition received a greater number of psychotherapy sessions (e.g., Mitchell et al., 2021). Similarly, a randomized placebo-controlled ayahuasca trial noted that four participants (all in the ayahuasca condition) remained in an inpatient unit for a week following ayahuasca administration (Palhano-Fontes et al., 2019). There is also generally flexibility surrounding the duration of sessions and therapeutic contact. On the dosing day, participants typically remain in the therapeutic setting until the drug's acute effects have worn off. Study protocols often include “check-in” phone calls that allow offering psychological support and are intended to be 5 to 20 min but are permitted to be longer (e.g.,
maps.org/mapp1). Given the psychoactive, emotionally evocative, and sometimes psychologically destabilizing effects of psychedelics, individuals who are administered a psychedelic may receive more psychological support through additional sessions, extended contact on dosing days, and longer check-ins. If this is the case, between-group differences may be attributable to the quantity of psychological support participants receive, rather than the psychedelic substance itself. It is therefore problematic that psychedelic trials generally do not report on the exact frequency or duration of psychological support in each condition, and fail to control for these potential confounds.
There is also lack of clarity as to whether the quality of the therapy provided across treatment conditions in psychedelic trials is equivalent. Within psychotherapy research, measurements of therapist competence (i.e., level of skill shown by the therapist) and adherence (i.e., following a therapy manual) are important for ensuring that the quality of the intervention is equivalent across treatment groups (Waltz et al., 1993). It is therefore problematic that some psychedelic trials do not describe monitoring adherence and that no trials have reported on adherence or competence ratings nor controlled for differences in these ratings across treatment conditions. Additionally, only one psychedelic trial evaluated therapeutic alliance (Murphy et al., 2022), a critical element across therapeutic interventions (Gukasyan and Nayak, 2022).
These potential imbalances in the quantity and quality of psychological support may be especially exacerbated within psychedelic trials due to unblinding of therapists and investigators. Namely, after unblinding, there may be conscious or unconscious bias toward providing greater quantity or quality of psychotherapy to those who were administered a psychedelic. Bias due to imbalance in quantity and quality may be exacerbated in trials that integrate psychedelic administration with evidence-based psychotherapeutic interventions (e.g., cognitive behavior therapy), but nevertheless remain important threats to validity in trials that include psychological support (especially given that such interventions may be less structured and therefore more prone to potential imbalances across conditions).
To address the above concerns, we suggest that future research: (1) ensure participants in each condition receive equivalent quantity of psychological support (e.g., “check-ins” should be psychological support sessions of a standardized duration or be strictly medically focused and completed by a physician not involved in providing psychotherapy) and report deviations in duration or frequency of psychological support; (2) employ tools to rate the quality of psychotherapy or psychological support (i.e., competence and adherence) and therapeutic alliance and report them alongside treatment outcomes; and (3) include therapy quantity and quality as covariates in primary outcome analyses.
Generalizability is essential to establishing the clinical meaningfulness of a trial and its applicability to individuals in need of treatment. It is problematic that most psychedelic trials include numerous psychiatric-related exclusion criteria (often omitted from method and limitation sections) that are closely related to the disorder being studied, including the presence of elevated suicide risk, comorbid psychiatric disorders, or screening positive for borderline personality disorder (e.g., Goodwin et al., 2022). Most trials also include non-specific exclusion criteria, such as excluding participants based on judgment that participants are “incompatible with establishment of rapport or safe exposure to psilocybin” (Goodwin et al., 2022). These non-specific exclusion criteria lack specificity, introduce investigator-specific bias, interfere with replicability (due to their being no specification of how such decisions are made), and further limit the generalizability of study findings. They may also introduce a channeling bias, as researchers might “cherry-pick” participants they suspect are most likely to respond favorably to psychedelic therapy. These biases could further amplify the problem of functional unblinding, since the selected participants might be more prone to expectancy effects and good-subject bias. While these exclusion criteria may be understandable in trials focused on pursuing drug approval and establishing safety and efficacy in homogenous groups, they may nonetheless bias results and limit knowledge surrounding the real-world safety and effectiveness of the intervention, which is especially concerning as governments (e.g., Australia) move toward making such interventions accessible (Kunstler et al., 2023).
To address these concerns, we suggest that future trials (1) reduce the number of exclusion criteria and remove non-specific exclusion criteria; and (2) transparently describe exclusion criteria, participant selection, and how these affect the generalizability of any findings. As research progresses, we recommend (3) conducting effectiveness trials focused on generalizability.
Psychedelic research has shown intriguing preliminary results and a rapidly expanding scope of clinical potential. Given their status as Schedule 1 drugs and their integration of both pharmacotherapy and psychotherapy, psychedelic therapy trials are especially challenging. Early researchers should be commended for their persistence through these regulatory and scientific challenges. However, the potential for imbalance in therapy quantity and quality, and concerns regarding generalizability to a general psychiatric population, are key threats to the validity of past and forthcoming research findings. Attention to, and transparency about, these methodological concerns is paramount. Developing methods to address them will improve the quality of psychedelic research, strengthen confidence in its findings, inform policy-makers' deliberations, and potentially contribute to the development of psychedelic therapy as a mental health intervention.
Declaration of Competing Interest
RJZ is a postdoctoral fellow in the NYU Langone Psychedelic Medicine Research Training program funded by MindMed. MindMed was not involved in the conceptualization, writing, or editing of the present manuscript.
References
Goodwin et al., 2022
G.M. Goodwin, S.T. Aaronson, O. Alvarez, P.C. Arden, A. Baker, J.C. Bennett, C. Bird, R.E. Blom, C. Brennan, D. Brusch, L. Burke, K. Campbell-Coker, R. Carhart- Harris, J. Cattell, A. Daniel, C. DeBattista, B.W. Dunlop, K. Eisen, D. Feifel, E. Malievskaia
Single-dose psilocybin for a treatment-resistant episode of major depression
N. Engl. J. Med., 387 (18) (2022), pp. 1637-1648
View at publisher CrossRefView in ScopusGoogle Scholar
Gukasyan and Nayak, 2022
N. Gukasyan, S.M. Nayak
Psychedelics, placebo effects, and set and setting: insights from common factors theory of psychotherapy
Transcult. Psychiatry, 59 (5) (2022), pp. 652-664
View at publisher CrossRefView in ScopusGoogle Scholar
Kunstler et al., 2023
B.E. Kunstler, L. Smith, C.J. Langmead, D.M. Goodwin, B. Wright, M.A. Hatty
We don't want to run before we walk”: the attitudes of Australian stakeholders towards using psychedelics for mental health conditions
Public Health Res. Pract., 33 (3) (2023), Article 3332321
Google Scholar
Mitchell and Anderson, 2023
J.M. Mitchell, B.T. Anderson
Psychedelic therapies reconsidered: compounds, clinical indications, and cautious optimism
Neuropsychopharmacology (2023)
Advance online publication
Google Scholar
Mitchell et al., 2021
J.M. Mitchell, M. Bogenschutz, A. Lilienstein, C. Harrison, S. Kleiman, K. Parker-Guilbert, M. Ot'alora, W. Garas, C. Paleos, I. Gorman, C. Nicholas, M. Mithoefer, S. Carlin, B. Poulter, A. Mithoefer, S. Quevedo, G. Wells, S.S. Klaire, B. van der Kolk, R. Doblin
MDMA-assisted therapy for severe PTSD: a randomized, double-blind, placebo-controlled phase 3 study
Nat. Med., 27 (2021), pp. 1025-1033, 10.1038/s41591-021-01336-3
View at publisher
This article is free to access.
View in ScopusGoogle Scholar
Mitchell et al., 2021
J.M. Mitchell, G.M. Ot'alora, B. van der Kolk, S. Shannon, M. Bogenschutz, Y. Gelfand, C. Paleos, C.R. Nicholas, S. Quevedo, B. Balliett, S. Hamilton, M. Mithoefer, S. Kleiman, K. Parker-Guilbert, K. Tzarfaty, C. Harrison, A. de Boer, R. Doblin, Yazar-Klosinski, & MAPP2 Study Collaborator Group
MDMA-assisted therapy for moderate to severe PTSD: a randomized, placebo-controlled phase 3 trial
Nat. Med. (2021)
Advance online publication
Google Scholar
Murphy et al., 2022
R. Murphy, H. Kettner, R.J. Zeifman, B. Gibraldi, L. Kaertner, J. Martell, T. Read, A. Murphy-Beiner, M. Baker-Jones, D. Nutt, D. Erritzoe, R. Watts, R.L. Carhart- Harris
Therapeutic alliance and rapport modulate responses to psilocybin assisted therapy for depression
Front. Pharmacol., 12 (2022), Article 788155
View in ScopusGoogle Scholar
Muthukumaraswamy et al., 2021
S.D. Muthukumaraswamy, A. Forsyth, T. Lumley
Blinding and expectancy confounds in psychedelic randomized controlled trials
Expert Rev. Clin. Pharmacol., 14 (9) (2021), pp. 1133-1152
View at publisher CrossRefView in ScopusGoogle Scholar
Palhano-Fontes et al., 2019
F. Palhano-Fontes, D. Barreto, H. Onias, K.C. Andrade, M.M. Novaes, J.A. Pessoa, S.A. Mota-Rolim, F.L. Osório, R. Sanches, R.G. dos Santos, L.F. Tófoli, G.D.O. Silveira, M. Yonamine, J. Riba, F.R. Santos, A.A. Silva-Junior, J.C. Alchieri, N.L. Galvão-Coelho, B. Lobão-Soares, D.B. Araújo
Rapid antidepressant effects of the psychedelic Ayahuasca in treatment-resistant depression: a randomized placebo-controlled trial
Psychol. Med., 49 (4) (2019), pp. 655-663
View at publisher CrossRefView in ScopusGoogle Scholar
Perez et al., 2023
N. Perez, F. Langlest, L. Mallet, M. De Pieri, O. Sentissi, G. Thorens, F. Seragnoli, D. Zullino, M. Kirschner, S. Kaiser, M. Solmi, M. Sabé
Psilocybin-assisted therapy for depression: a systematic review and dose-response meta-analysis of human studies
Eur. Neuropsychopharmacol., 76 (2023), pp. 61-76
View PDFView articleView in ScopusGoogle Scholar
Raison et al., 2023
C.L. Raison, G. Sanacora, J. Woolley, K. Heinzerling, B.W. Dunlop, R.T. Brown, R. Kakar, M. Hassman, R.P. Trivedi, R. Robison, N. Gukasyan, S.M. Nayak, X. Hu, K.C. O'Donnell, B. Klemendi, J. Sloshower, A.D. Penn, E. Bradley, D.F. Kelly, R.R. Griffiths
Single-dose psilocybin treatment for major depressive disorder: a randomized clinical trial
JAMA, 330 (9) (2023), pp. 843-853
View at publisher
This article is free to access.
CrossRefView in ScopusGoogle Scholar
Waltz et al., 1993
J. Waltz, M.E. Addis, K. Koerner, N.S. Jacobson
Testing the integrity of a psychotherapy protocol: assessment of adherence and competence
J. Consult. Clin. Psychol., 61 (4) (1993), p. 620
View in ScopusGoogle Scholar
Cited by (1)
Safety and tolerability of inhaled N,N-Dimethyltryptamine (BMND01 candidate): A phase I clinical trial
2024, European Neuropsychopharmacology