@fbartos.bsky.social
PhD Candidate | Psychological Methods | UvA Amsterdam | interested in statistics, meta-analysis, and publication bias | once flipped a coin too many times
This week's blog post features "raincloud plots", a relatively recent development in data visualization.
Will the raincloud plot gradually replace the box plot? It just might!
Check out the raincloud plot for the planets in our solar system at
www.jasp-services.com/jasp-for-qua...
Also, this should not be a reason to stop exercising.
1) There are other benefits of exercise
2) Some populations/exercises show benefit
3) There might be wider effects on cognition; however, the literature is too heterogeneous and contaminated with publication bias to be certain
I think that the field needs to clean up the published literature a bit. Additional small studies are not going to move the needle at this point; maybe a couple of large-scale, pre-registered studies might provide more insight?
01.12.2025 16:19 — 👍 10 🔁 2 💬 1 📌 0
We also re-analyzed all of the original meta-analyses individually. Many of them are consistent with publication bias: the evidence for and the degree of the pooled effects decrease once publication bias is adjusted for.
01.12.2025 16:19 — 👍 2 🔁 0 💬 1 📌 0
We run subgroup analyses for each outcome/population/intervention. We found that most results are too heterogeneous to tell (see wide prediction intervals), but some interventions seem to be promising and some have substantive evidence against them. See figures for each outcome.
01.12.2025 16:19 — 👍 1 🔁 0 💬 1 📌 0
First, we found notable publication bias, especially in studies on general cognition and executive function. Importantly, there was extreme between-study heterogeneity (tau ~ 0.3-0.6!). This means that the results were consistent with both large benefit but also large harm.
01.12.2025 16:19 — 👍 3 🔁 0 💬 1 📌 0We were not the only ones to notice, also see @matthewbjane.bsky.social commenting on this when the study came out:
x.com/MatthewBJane...
So, we manually extracted the study-level data from the included meta-analyses and re-evaluated the evidence.
Previous meta-meta-analysis (doi.org/10.1136/bjsp...) indicated consistent benefits of exercise for cognitive benefits across all domains and populations. However, it synthesized meta-analytic estimates and, as such, it could not adjust for publication nor evaluate heterogeneity.
01.12.2025 16:19 — 👍 3 🔁 0 💬 1 📌 0We just preprinted a huge meta-meta-analysis examining the effects of exercise on cognition, memory, and executive function
In short
- 2239 effect sizes
- extreme between-study heterogeneity
- extensive publication bias
- some subgroup/exercise-specific effects
More below (doi.org/10.31234/osf...)
Feb 16-17: JASP Hackathon in Amsterdam!
Now open for applications
Post with details on the JASP website:
jasp-stats.org/2025/11/21/a...
New blogpost on our JRR website!
Check out our new blogpost (link below): we highlight a study for re-analysis on whether CEOs are shaped by their daughters. This could be a potential candidate for a robustness report.
www.journalofrobustnessreports.org/are-ceos-sha...
"Always plot your data" -- an example of using descriptives *before* measuring capability of an instrument. We intend to have regular posts demonstrating how JASP can be useful for quality control and lean six sigma.
www.jasp-services.com/jasp-for-qua...
We just started a website highlighting articles that would make good candidates for a Robustness Report.
🌐 Check out the website: www.journalofrobustnessreports.org
📄 Read our first blog post: www.journalofrobustnessreports.org/inaugural-po...
We built the openESM database:
▶️60 openly available experience sampling datasets (16K+ participants, 740K+ obs.) in one place
▶️Harmonized (meta-)data, fully open-source software
▶️Filter & search all data, simply download via R/Python
Find out more:
🌐 openesmdata.org
📝 doi.org/10.31234/osf...
We developed PublicationBiasBenchmark R package (github.com/FBartos/Publ...) that can be easily extended with new methods and measures. It also automatically generates a webpage with summary reports (fbartos.github.io/PublicationB...). All the raw data, results, and measures are available on OSF.
23.10.2025 16:03 — 👍 2 🔁 0 💬 0 📌 0
Our proposal addresses other issues of current simulation studies (incomparability, irreproducibility...).
We demonstrate the living synthetic benchmark methodology on the publication bias adjustment literature. See how previous simulations use different methods and measures.
To start the process, we suggest
- collecting all published methods and simulations
- evaluating all methods on all simulations
- publishing this set of results as the initial synthetic benchmark
- later research can update this benchmark with new methods and simulations
We want to separate those two steps.
New simulations should be published without new methods. Instead, they should evaluate all existing methods.
New methods should be published without new simulations. Instead, they should be assessed on all existing simulations.
Simulation studies have a conflict of interest problem. The same team:
- develops a new method
- designs a simulation study to evaluate it
However, the new method has to show good performance to get published.
We propose living synthetic benchmarks to address the issue (doi.org/10.48550/arX...).
New blog post! Why companies win by investing in open-source software. With zebra-painted cows, of course.
www.jasp-services.com/why-companie...
We are pleased to have
@fbartos.bsky.social
join us today, Tuesday, September 30th, 11am (EST) to talk about Bayesian Hypothesis testing! This is followed by a workshop on using JASP for statistics around 12:10pm. The zoom is open to public with details in the flyer!
@PsychPrinceton
Simonsohn has now posted a blog response to our recent paper about the poor statistical properties of the P curve. @clintin.bsky.social and I are finishing up a less-technical paper that will serve as a response. But I wanted to address a meta-issue *around* this that may clarify some things. 1/x
25.09.2025 10:07 — 👍 76 🔁 30 💬 2 📌 8> Why are you actively misrepresenting what others are saying all the time?
I'm happy to discuss with you in person if we meet anywhere, but I don't find replying to you online very productive at this point.
> Carter et al are right, and you are wrong
That's pretty much just arguing from authority
I did not say meta-analyses with huge heterogeneity lol. I said under any heterogeneity. Would you consider tau = 0.1-0.2 on Cohen's d scale with an average effect size of 0.2-0.4 huge? I would not. Pretty meaningful result (and probably representative of many meta-analyses), but p-curve fails.
24.09.2025 13:33 — 👍 0 🔁 0 💬 1 📌 0> P-curve does what worse than random effects?
All the simulations I linked shows that p-curve estimates the effect size worse, on averate, than random effects.
Must've been a bug on the platform -- I could not see any responses I sent to the thread but other features worked fine.
24.09.2025 13:29 — 👍 0 🔁 0 💬 1 📌 0For some reason, I cannot reply to Lakens anymore?
Regardless, if anyone is interested in the topic:
- Carter does not say something completely opposite to my claims
- I^2 is not a measure of absolute heterogeneity, Laken's argument strawmans meta-analysis
- p-curve does worse than random effects
It's not completely opposed - they say that they work well only under no heterogeneity. From their and other simulation studies it seems like that a simple random effects model performs better than p-curve even when publication bias is present. As such, I don't see any reason for using the method.
24.09.2025 11:12 — 👍 0 🔁 0 💬 1 📌 0How is it directly opposite to what I'm saying?
Also, glad we got to the late-stage science when you start pulling arguments of authority. Always great debating with you :)
