Psychological Interventions for Psychosis
Meta-analytic database of randomized controlled trials on psychosis — filter the evidence and run a full meta-analysis online.
Additional Information
This online meta-analysis tool allows exploring a simplified version of the “Psychosis (Psychotherapy vs. Control)” dataset. The code used to simplify the data for this application can be downloaded here.
Select outcome
Choose which outcome domain to analyze. Only studies reporting that outcome will appear on the next step.
In this step you choose which of three outcome variables to analyze: total, positive, or negative symptoms.
Not all studies in the database report all outcome variables. Once you select an outcome, only studies reporting that outcome appear in the data table. All filters on the next step apply only to those studies. You can return here at any time to select a different outcome.
Select your data
Apply filters to choose trials for your meta-analysis. All trials are selected by default — refine on the right, the table updates automatically, then Run meta-analysis.
Meta-analysis results
Results for the trials you selected. Use the tabs below for pooled effects, plots, outliers, publication bias, risk of bias and moderators.
This tab shows you the results of the meta-analysis conducted for your selected data. In the window below, you can switch from one panel to another to see various results, including the main effect size and heterogeneity, the forest plot, results of outlier analyses, publication bias analyses and moderator analyses.
By default, the web application automatically makes a few decisions concerning the analysis settings. These can be seen below under "Analysis Settings". By default, the application uses a Random-Effects Pooling Model for your meta-analysis. This is nearly always the correct choice when analyzing data in mental health research. In addition, the Restricted Maximum-Likelihood estimator is used by default to calculate the between-study heterogeneity. However, when clicking on the dropdown menu "Pooling Model", you can see that there are various estimators available when assuming a Random-Effects (RE) model. Sometimes, the choice of the between-study heterogeneity estimator can have an impact on the pooled results, and not every estimator is optimal under all circumstances. If you want to learn more about this, you can consult Harrer, Cuijpers, Furukawa & Ebert, 2020. It is also possible to calculate effects using a Fixed-Effect Model; but this model should only be used if you have good reasons for applying it.
Under "Analyzed Moderators", the moderating variables which are currently inspected in meta-regression/subgroup analyses in the "Moderator Analysis" tab are shown. By default, the publication year is used for a meta-regression on your selected data, and the country/region of a study is used as a subgroup analysis. You can easily add more moderator analyses by clicking on the white box. A dropdown will then appear, showing more variables which are available. You can then add those variables by clicking on them. It is also possible to remove variables: again click on the white box and use the backspace ← key.
By default, the application also uses Knapp-Hartung adjustments to calculate the confidence interval for your meta-analysis result. It is possible to disable this method by unchecking the "Use Knapp-Hartung Adjustments" checkbox. You can learn more about this method in Harrer, Cuijpers, Furukawa & Ebert, 2020.
Finally, after you have completed the reconfiguration of your meta-analysis settings, simply click the "Re-run Meta-Analysis" button. This will recalculate the results of your meta-analysis using the new settings you have provided. Please note that the reanalysis may take some time before it is finished.
A random-effects meta-analysis does not assume that every study shares one identical true effect. Instead, the true effects are assumed to be normally distributed. This chart shows that assumed distribution: a bell curve centred on the pooled estimate, whose width is the between-study standard deviation (τ). A wider curve means more heterogeneity between studies.
The thick vertical line marks the pooled overall effect (Hedges’ g). The shaded area under the curve is its 95% confidence interval — the range of plausible values for the average true effect.
The two short solid lines in the tails mark the 95% prediction interval: the range in which the true effect of a future, comparable study would be expected to fall.
The vertical line at 0 is the point of no effect. If the effect and its intervals lie clearly to one side of it, the intervention has a consistent direction of effect.
Individual studies with extreme effects can distort a pooled result and inflate heterogeneity. This is a sensitivity analysis: studies flagged as statistical outliers — those whose 95% confidence interval does not overlap the confidence interval of the pooled effect — are removed, and the meta-analysis is re-run on the remaining studies.
The numbers show the pooled effect and heterogeneity after removing outliers, and the list shows which studies were removed (with their effect sizes).
The chart compares the true-effect distribution with outliers removed (coloured) against the original analysis (grey). If the two are similar, the result is robust to outliers; if they differ markedly, the pooled effect is sensitive to a few extreme studies and should be interpreted with caution.
The Baujat plot helps identify studies that have a disproportionate impact on the meta-analysis.
The horizontal axis shows each study’s contribution to the overall heterogeneity; the vertical axis shows its influence on the pooled effect (how much the result changes when the study is removed).
Studies in the top-right corner are both major sources of heterogeneity and highly influential — these are the most important to inspect when judging how robust your findings are.
A funnel plot plots each study's effect against its precision (standard error). Without bias, studies scatter symmetrically around the pooled effect, forming an inverted funnel — smaller, less precise studies scatter more widely toward the bottom.
Asymmetry (a gap in one bottom corner) can signal publication bias — e.g. small studies with non-significant results going unpublished — though it can also reflect genuine heterogeneity or chance.
The shaded contours mark conventional significance regions (p < .05, .025, .01), which help judge whether any missing studies would fall in non-significant areas.
Egger's regression test quantifies the asymmetry by testing whether the regression intercept differs from zero. A significant intercept (p < .05) indicates asymmetry. Interpret with caution when few studies are included.
A p-curve examines the distribution of the statistically significant p-values (p < .05) across your studies to judge whether they hold evidential value — a genuine effect — or mainly reflect selective reporting / p-hacking.
If a true effect exists, significant p-values cluster near .01 (the curve is right-skewed). With no effect, they are uniform (flat). If results were pushed just past significance, the curve is left-skewed (piling up near .05).
Your observed curve is compared against two references: the null of no effect (flat, 20% per bin) and the null of 33% power.
Evidential value present means the observed curve is significantly right-skewed. Evidential value inadequate means it is flatter than expected even from studies with 33% power. The power estimate approximates the average statistical power of the significant studies.
Subgroup Analysis
About this web app
This application was developed by researchers at the Vrije Universiteit Amsterdam, The Netherlands, the Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany, and the Technical University Munich, Germany. The data are based on a meta-analytic database that was developed by researchers at the Vrije Universiteit Amsterdam in the past 14 years.
This project, led by Prof. Pim Cuijpers and Dr. Eirini Karyotaki, has resulted in a long series of published studies in peer-reviewed journals. Researchers from the Friedrich-Alexander-Universität in Germany, led by Mathias Harrer and Dr. David Ebert (Technical University Munich), have developed the web application and the automated analyses that are conducted online.
About the Metapsy Web Application
This web application serves as an interface to the Metapsy meta-analytic database. The application uses a Shiny server instance to access the database, enable data downloads, and handle computations using state-of-the-art meta-analytic techniques.
The meta (Balduzzi, Rücker & Schwarzer, 2019) and dmetar (Harrer, Cuijpers, Furukawa & Ebert, 2020) R packages are used to perform the meta-analyses.
The metagen function is used internally to perform the meta-analytic pooling. By default, the Restricted Maximum-Likelihood estimator (Viechtbauer, 2005) is used to estimate the between-study heterogeneity/variance τ2 in a random-effects model; REML is also used for meta-regressions. To calculate the Number Needed to Treat (NNT), the method by Furukawa and Leucht (Furukawa & Leucht, 2011) is used, assuming Control Event Rate (CER) of 0.19, which is derived from Cuijpers et al., 2014. For subgroup analyses, the subgroup.analysis.mixed.effects function is used, implementing a mixed-effect model for which results within subgroups are pooled using a random-effects model (inheriting the τ2-estimator specified for the overall analysis), and results between subgroups are compared assuming a fixed-effects model.
To generate forest plots, the forest function in meta is used.
Outlier selection is conducted using the find.outliers function in dmetar. The function implements a simple outlier removal algorithm where all results for which the 95% confidence interval is outside the 95% confidence interval of the pooled effect are removed as outliers. Due to high computational costs, sensitivity analyses based on the “Leave-One-Out” paradigm are not conducted for large meta-analyses (k>50). For smaller meta-analyses, a Baujat plot (Baujat et al., 2002) is created using the baujat function in meta.
Publication bias analyses are conducted using the eggers.test function in dmetar (which is a wrapper for the metabias function in meta). A P-curve is created using the pcurve function in dmetar. Please note that several prerequisites should be considered before P-curves can be interpreted (Simonsohn et al., 2014).
For the risk of bias overview, the rob.summary function in dmetar is used.
The metapsyData R package
Access the Metapsy meta-analytic databases directly in R.
The metapsyData package lets you access the Metapsy meta-analytic psychotherapy databases — including the “Psychosis” database powering this app — directly in your R environment.
Installation
if (!require("devtools"))
install.packages("devtools")
devtools::install_github("metapsy-project/metapsyData")
Usage
Once installed, run getData() to load a database locally. Databases are referenced by their identifier; this app uses "psychosis-psyctr":
library(metapsyData)
library(dplyr)
d <- getData("psychosis-psyctr")
glimpse(d$data)
#> Rows: 206
#> Columns: 55
#> $ study <chr> "Agothor, 2010", "Ago…
#> $ reference <chr> "Aghotor J, Pfueller …
#> $ condition_arm1 <chr> "MCT", "MCT", "CR", "…
#> $ condition_arm2 <chr> "BF", "BF", "TAU1", "…
#> $ multi_arm1 <chr> NA, NA, NA, NA, NA, N…
#> $ multi_arm2 <chr> NA, NA, NA, NA, NA, N…
#> $ comparison <chr> "psy vs ctr", "psy vs…
#> $ outcome_type <chr> "change", "change", "…
#> $ symptom_domain <chr> "total", "positive", …
#> $ instrument <chr> "PANSS_total", "PANSS…
#> $ rating <chr> "interviewer-rated", …
#> $ mean_arm1 <dbl> NA, NA, 71.40, 17.20,…
#> $ sd_arm1 <dbl> NA, NA, 14.72, 4.50, …
#> $ n_arm1 <dbl> NA, NA, 36, 36, 36, 5…
#> $ mean_arm2 <dbl> NA, NA, 73.00, 21.30,…
#> $ sd_arm2 <dbl> NA, NA, 18.05, 9.50, …
#> $ n_arm2 <dbl> NA, NA, 18, 18, 18, 4…
#> $ mean_change_arm1 <dbl> -17.1, -4.5, NA, NA, …
#> $ sd_change_arm1 <dbl> 24.5, 4.8, NA, NA, NA…
#> $ n_change_arm1 <dbl> 16, 16, NA, NA, NA, N…
#> $ mean_change_arm2 <dbl> -11.9, -2.6, NA, NA, …
#> $ sd_change_arm2 <dbl> 19.8, 4.1, NA, NA, NA…
#> $ n_change_arm2 <dbl> 14, 14, NA, NA, NA, N…
#> $ totaln_arm1 <dbl> 16, 16, 36, 36, 36, 5…
#> $ totaln_arm2 <dbl> 14, 14, 18, 18, 18, 5…
#> $ N.randomized._total <dbl> 30, 30, 54, 54, 54, 1…
#> $ year <dbl> 2010, 2010, 2003, 200…
#> $ time <dbl> NA, NA, NA, NA, NA, N…
#> $ time_weeks <dbl> NA, NA, NA, NA, NA, N…
#> $ trial_length <dbl> 4, 4, 10, 10, 10, 26,…
#> $ format <chr> "Group", "Group", "Co…
#> $ n_sessions_arm1 <dbl> NA, NA, NA, NA, NA, N…
#> $ n_sessions_arm2 <dbl> NA, NA, NA, NA, NA, N…
#> $ country <chr> "Germany", "Germany",…
#> $ mean_age <dbl> NA, NA, NA, NA, NA, N…
#> $ percent_women <dbl> NA, NA, NA, NA, NA, N…
#> $ recruitment <chr> "Inpatient", "Inpatie…
#> $ diagnosis <chr> "Schizophrenia Sepect…
#> $ dur_ill_arm1 <dbl> 375, 375, NA, NA, NA,…
#> $ dur_ill_arm2 <dbl> NA, NA, NA, NA, NA, N…
#> $ stage <chr> "Early", "Early", NA,…
#> $ clinical_picture <chr> NA, NA, NA, NA, NA, "…
#> $ other <chr> NA, NA, NA, NA, NA, N…
#> $ Selection.bias..random.sequence.generation <chr> "Low", "Low", "Unclea…
#> $ Selection.bias..allocation.concealment <chr> "Unclear", "Unclear",…
#> $ Reporting.bias..selective.reporting <chr> "High", "High", "High…
#> $ Attrition.bias..incomplete.outcome.data <chr> "Low", "Low", "Unclea…
#> $ Performance.bias..blinding.of.ppts.and.personnel <chr> "High", "High", "High…
#> $ Detection.bias..blinding.of.outcome.assessment <chr> "Low", "Low", "Low", …
#> $ rob <dbl> 0, 0, 0, 0, 0, 0, 0, …
#> $ .id <chr> "Agothor_2010_change_…
#> $ .g <dbl> -0.22545996, -0.41185…
#> $ .g_se <dbl> 0.3671832, 0.3700201,…
#> $ event_arm1 <dbl> NA, NA, NA, NA, NA, N…
#> $ event_arm2 <dbl> NA, NA, NA, NA, NA, N…
The raw data files can also be browsed in the GitHub repository ↗ under data, and the full package documentation is hosted on rdrr.io ↗.
Citation
Run citation("metapsyData") to retrieve the current citation:
Harrer, M., Ebert, D. D., Karyotaki, E., & Cuijpers, P. (2022).
metapsyData: Access the Meta-Analytic Psychotherapy Database.
R package version 0.1.0. DOI: 10.5281/zenodo.5171880.