Brent W. Roberts

What is the construct validity of all of the most famous and/or most widely used experimental manipulations?

The latest new big thing in biomarker research crashes on the shoals of tiny effect sizes. The median effect sizes in correlational terms are between 0.011 to 0.028. How many millions of dollars have we already spent on this noise mining operation?

Identifying reproducible biomarkers for microbiome association studies requires thousands of sample sizes

Identifying reproducible biomarkers for microbiome association studies requires sample sizes of thousands www.sciencedirect.com/science/arti...

Sand elves are the worst, except for those sand gnomes.

Can you use off the shelf generative LLMs (eg ChatGPT, Claude) to assess personality from brief bits of unconstrained, open-ended text? If so, how well do they do?

Turns out you can, and they do amazingly well. I couldn't believe how well.

Here's our pre-print on it and a 🧵

YouTube video by Alvin Untercover GWAR covers Kansas' "Carry On Wayward Son"

www.youtube.com/watch?v=HJze...
Still one of my GWAR favorites

Again, Universal Basic Income is a good idea.

#AcademicSky #PsychSciSky

Interesting analysis of publication trends pre & post tenure.

#Psychology mirrors natural sciences (and deviates from other social sciences, business, etc, where there are declines post-tenure)

www.pnas.org/doi/epdf/10....

The opening page of the Annual Review of Psychology article by Traci Mann and Andrew Ward (2025), entitled The Self-Control of Eating. https://tinyurl.com/bdnebw79 (open access)

As somebody who studies self-control (but is bad at it), I read this 2025 open-access review on the self-control of eating (tinyurl.com/bdnebw79) by Mann & Ward w/ great personal interest. I highly recommend it, esp. if you study self-control and/or want to deal w/ your eating regulation challenges.

Researchers value null results, but struggle to publish them Survey finds that fear of reputational harm and a lack of support and publication platforms are among respondents’ key concerns.

Scientists overwhelmingly recognize the value of sharing null results, but rarely publish them in the research literature

go.nature.com/450KElr

Reflections on replication It started with a laptop.

New post on how I became involved in the open science movement.
Not by design, but through circumstance—beginning with a case of fraud and an uneasy task. This led to replication work, writing, and a lasting shift in perspective.
rolfzwaan.substack.com/p/reflection...

Are you suggesting that I attrit?

Birth order effects are huge and real…

Replication code as a cornerstone of the credibility revolution 2.0 This opinion piece argues that code sharing is a key yet underutilized component of research transparency and the ongoing “Credibility Revolution 2.0.…

📢 Excited to share my first academic opinion piece—now out #OpenAccess in the Journal of Behavioral and Experimental Economics!
🔗 www.sciencedirect.com/science/arti...

👇Thread: Why replication code is essential for the future of credible science 🧵

Innovative and interactive statistics teaching using Quarto Quarto is an open-source system that is useful for creating scientific teaching resources, particularly statistical resources, in various formats. Quarto is the next-generation version of R Markdown,...

Neat little article on using Quarto and creating course materials: onlinelibrary.wiley.com/doi/full/10..... Resonates nicely with our experiences creating all our workbooks in R Markdown / Quarto.

This is our preprint! 3 mentors, 4 students, 71 meetings, and 2.5 years of work.

We asked: does personality influence preference for non-conventional medicine? According to our meta-analysis, the Big Five does not. But another trait does! 👀 #PsychSciSky

doi.org/10.31234/osf...

4-panel comic. (1) [Person 1 with ponytail flanked by person with short hair and another person speaking into microphone at podium] PERSON 1: In the early 2010s, researchers found that many major scientific results couldn’t be reproduced. (2) PERSON 1: Over a decade into the replication crisis, we wanted to see if today’s studies have become more robust. (3) PERSON 1: Unfortunately, our replication analysis has found exactly the same problems that those 2010s researchers did. (4) [newspaper with image of speakers from previous panels] Headline: Replication Crisis Solved

Replication Crisis

xkcd.com/3117/

But you are resigned to only spreading it to the youth of today rather than changing the minds of those who are already in the field? Don't get me wrong, that is a good thing and I'm grateful for your efforts, but those youths are housed in labs run by those old people who haven't changed....

A helpful way to think about traits (a la Whole Trait Theory) is that they are within-person density distributions. This paper examined how these density distributions varied by different kinds of situations. A really nice way to quantify person-situation interactions. osf.io/preprints/ps...

Most were not publicly shamed but were active researchers who, I assume, went on to do more research. The experience of having your work fail to replicate would surely cause some kind of behavior change, no? Kind of like the parenting practice of "natural consequences".

Despite being psychologists, and supposedly knowing better, most of us try to change others behavior through verbal reasoning (rational arguments) and punishment (public shaming). It would be good to see just how effective or ineffective those efforts were. My vote is for ineffective, btw 2/

Oh, I think it is potentially interesting on a general level. It was, after all, a bit of a quasi experiment. You could compare the targets to those in their co-author network, their students, then random colleagues to see what the changes in adoption levels were like. 1/

I would be curious whether the targets of replication, in particular, changed their behavior. It would be a telling analysis of the effect of the crisis to know whether they went on to do things differently or not.

The enemy of my enemy is my friend???

Thanks for this, though some argue you should never read your own diary...since you are digging around in our histories, I was wondering whether you had any insight into how people's behavior changed? In particular, has anyone followed up on the authors who's work failed to replicate?

📊 Following up on our recent review of forced-choice (FC) measurement research: lnkd.in/edEEiB6h
🧠 Full article here: lnkd.in/eWHf64Y5

Here are a few extra visualizations I had the pleasure of working on (sadly, cut from the final paper due to space) 👇

Eat cake doctor Oz. Carrot cake that is.

Because I know that some of my international friends here might be interested in this: a seminar on how to become a professor in Germany.

www.dhvseminare.de/career_paths...

🚨New paper out in AMPPS! proud to be part of this one, another solid product of @improvingpsych.org✨
We mapped the grey zones of psychology’s research culture through a consensus process. Ever wondered, “Wait... is this a questionable practice?” You’re not alone: doi.org/10.1177/2515... #OpenScience

Western University is offering new scholarships of up to 40k CAD for four years to doctoral students who are studying at, have an offer from, or have had an offer rescinded from one of the top 100 American universities.

Come and study with me or one of my colleagues!