Rick Carlsson

I am hoping to recruit a graduate student for next year. That person would help conduct research on leadership, individual differences, and methodological skullduggery. Please forward to any potentially interested students.

My Shiny app containing 3530 Open Science blog posts discussing the replication crisis is updated - you can now use the SEARCH box. I fixed it as my new PhD Julia wanted to know who had called open scientists 'Methodological Terrorists' :) shiny.ieis.tue.nl/open_science...

I struggle to think of a statistical "method" that has been more damaging to psychology (and related disciplines) than mediation analysis based on observational data. IMHO all such manuscripts should be desk rejected.

Soft Drink Consumption and Depression Mediated by Gut Microbiome Alterations This cohort study examines the association between soft drink consumption and major depressive disorder diagnosis and severity and whether this association is mediated by changes in the gut microbiota...

This one surely has something on offer for every one: In this cross-sectional mediation analysis, the "effects" of soft drink consumption on depression were "mediated" by abundance of Eggerthela in the gut microbiome.

This was sent to me via dm and now you all got to suffer as well.

What’s in a correlation? Correlation may not imply causation, but let’s just ignore that for a second. Correlations are standardized effect size metrics and as such have some quirks by design. These are benign enough when you...

New blog post! Calculating a correlation is easy enough. But let's say you calculated two of them and they happen to differ. What follows from that? Turns out there are too many moving parts for an easy answer.

www.the100.ci/2025/07/28/w...

First rule of psychology nomenclature is to only use naming established by Freud. This is where psychometrics fail us.

I feel like psychometrics is uniquely bad at naming things. Parallel, congeneric, tau-equivalent, essentially tau-equivalent measures? Configural, metric, scalar, residual invariance? Item difficulty defined so that the higher the difficulty, the easier the item???

All researchers also needs to take full responsibility for their research methods. Not understanding something and just citing an authority figure isn’t good enough. If you don’t have the competence to apply methods, either learn, get people on your team who do, or switch methods.

Agreed. All fields need to take full responsibility for their research methods. Full stop. Likely, most research methods cannot be fully vetted without cross-disciplinary collaborations, but sitting around waiting for discipline x to solve our problems is not an option!

New blog post: Why we should stop using statistical techniques that have not been adequately vetted by experts in psychology daniellakens.blogspot.com/2025/10/why-... where I reflect on how we should check the quality of novel statistical techniques.

Pro tip: When fabricating a dataset, also make several errors in the analysis of the fake data. Then when sleuths point out the errors as well as the fabrication, you can make a more plausible fake dataset when the editor lets you correct the statistical errors "in the interest of fairness".

I think good description and indexing is important on its own. I don’t mind a database of bad theories as long as it’s transparent that it’s just a collection without any claims of validity.

It’s important that the database is correct and exhaustive though!

After a day or two of sleuthing I have realized that a recent (and likely to be very influential) meta-analysis missed 30%-40% of the relevant literature. Do I bother trying to correct the record?

Or submit it as a registered report format systematic review!

I tend to ignore meta-analyses that aren’t systematic reviews as they tend to be haphazard collections of studies at best. If it claims to be a systematic review you should correct and/or submit a registered report with proper searches etc.

I think it’s deliberately not supported.

Psych-DS is (1) spellcheck for your datasets and (2) a pathway to standardizing data in our academic fields that *everyone* can learn.

And it's live RIGHT NOW!

psych-ds.github.io

(This is the announcement post I've been leading up to)

This article notes that "Among 40 papers flagged as having issues, he [i.e., me] found 14 false positives (for example, the model stating that a figure referred to in the text did not appear in the paper, when it did)".

For background, here are the cases that I believe to be false positives.

/0

Join the OSIRIS Reproducibility Network: A Collaborative Effort to Advance Open Science Become part of a dynamic network committed to enhancing research transparency and reproducibility through innovative randomized controlled trials. Contribute your expertise, gain valuable training, an...

OSIRIS (Open Science to Improve Reproducibility in Science) is currently conducting several interventional studies on Open Science practices. We invite interested colleagues to register and join a group that may take part in these studies. More info and registration 👇

That’s a description of how things have been. The idea of FMS as an emerging scientific field will definitely require a more formal approach. I can agree on you on that.

If so, then metascience is not science either. I think we have to separate application of a scientific discipline with the discipline itself.

I will therefore not complain if someone tries to introduce more structure. And so here you go: a decent attempt at methods building from @jamesheathers.bsky.social :
jamesheathers.curve.space

(Could I have a PDF though? </s>) 2/

Wow, this should be super interesting.

Wait, no, that's his book from 2022. Sorry!

Only way to get people to download metadata!

I am sorry that you fail to see the benefit of this paper or this type of work, Brett. Read it first, perhaps.

Lades et al. (2020) ERROR is a bug bounty program for science to systematically detect and report errors in academic publications

We are excited to announce the review of Lades et al. (2020) "Daily emotional well-being during the COVID-19 pandemic". Based on the review by Joop Adema, we find Moderate Errors which do not affect the core conclusions. We recommend the authors seek a correction error.reviews/reviews/lade...

Hey Quant Folks: Does anyone have any recommendations for testing if two correlation matrices are meaningfully and/or statistically significantly different from each other?

NB: In the words of Jeremy Irons: "Please feel free to speak to me as you might to a young child or a golden retriever."

Yes, cheaper and more accessible technology is a way for academics too take control of publishing!

Maybe I should just use fraud to fund it?!

I have even gotten multiple grants rejected on this idea. It’s my mad scientist arc for sure.