Brent W. Roberts

I think the problem in psych is that if you cleaned the cupboard of the unreplicable and uninformative work you wouldn't be left with much. Exacerbating that is the norm of giving researchers the benefit of the doubt, especially if you like the idea. And, that's what we teach in intro psych...

Journal Article: A Framework For Assessing the Trustworthiness of Scientific Research Findings (via @pnas.org) www.pnas.org/doi/full/10.... @cos.io

What’s a multiverse good for anyway? Julia M. Rohrer, Jessica Hullman, and Andrew Gelman Multiverse analysis has become a fairly popular approach, as indicated by the present special issue on the matter. Here, we take one step back and ask why one would conduct a multiverse analysis in the first place. We discuss various ways in which a multiverse may be employed – as a tool for reflection and critique, as a persuasive tool, as a serious inferential tool – as well as potential problems that arise depending on the specific purpose. For example, it fails as a persuasive tool when researchers disagree about which variations should be included in the analysis, and it fails as a serious inferential tool when the included analyses do not target a coherent estimand. Then, we take yet another step back and ask what the multiverse discourse has been good for and whether any broader lessons can be drawn. Ultimately, we conclude that the multiverse does remain a valuable tool; however, we urge against taking it too seriously.

New preprint! So, what's a multiverse analysis good for anyway?>

With @jessicahullman.bsky.social and @statmodeling.bsky.social

juliarohrer.com/wp-content/u...

A framework for assessing the trustworthiness of scientific research findings1 | PNAS Vigorous debate has erupted over the trustworthiness of scientific research findings in a number of domains. The question “what makes research find...

Our new paper, with colleagues from the Strategic Council of the National Academies, offers an integrative framework of the several components that contribute to making research findings trustworthy including ethics, methodology, transparency, inclusion, assessment, etc

www.pnas.org/doi/10.1073/...

violin plot of results by testosterone measure

violin plot of result by risk pref measure

New meta finds no correlation between testosterone and risk preference, buuuuut testosterone was mainly measured using 2D:4D (nonsense) or saliva (still bad) and we know from Frey et al. 2017 that the risk measures don't tap into a coherent preference.
www.sciencedirect.com/science/arti...

Very happy that our new paper is out in Collabra: Psychology! 😀 Across three studies, we failed to replicate social class-based differences in conformity, which raises further questions about key assumptions of the social cognitive theory of social class.

Gambling with research quality How you get 244 different ways to measure performance on the same test of decision making. And what it means for the reliability of behavioural science

Read my latest post for reflections on reproducibility, research quality and a summary of a great new study which shows how NOT to do it

https://open.substack.com/pub/tomstafford/p/gambling-with-research-quality

And the commentary we wrote about Julia's paper: www.dropbox.com/scl/fi/s90dr...

Here's Rodica's paper: www.dropbox.com/scl/fi/43x28...

Here's Julia's paper that she wrote as an undergrad...
www.dropbox.com/scl/fi/mxcsr...

birth order and personality...

I’m just reporting here…

If you see the contrarian on the road...do to him like you would the Buddha. 3/

The appeal of the contrarian position compensated for the weakness of their position. For example, bopping big pharma in the nose made it feel like a Sampson and Goliath battle. You see the same thing with COVID/masks/vaccines now 2/

For those of us who lived it and through it, I would characterize it as analogous to our reproducibility wars. The boring, but correct data was always in favor of reducing LDL. The contrarians used weak, observational data and anecdote to counter the boring position. 1/

Do conscientious people age more slowly?
Check it out in our work on epigenetic aging and personality differences.
🔍🧬🕰

doi.org/10.1177/0890...

Really cool work!

No One Notices Area Man's Marginal Attempts To Change

No One Notices Area Man's Marginal Attempts To Change

📣Revised preprint by @cas-goos.bsky.social

Measurement reliability, validity, and reporting in psychology still has a long way to go...

We compared original studies w replications, and where possible recalculated reliability & unidimensionality.

Some findings >

osf.io/preprints/ps...

smartphones. Obviously.

Maybe we can have academic hero’s because they are like good BBQ pit masters. Instead of admiring them because they claim to have discovered burnt ends we admire them for reliably turning out excellent burnt ends.

Citations over time for the BART

And for #4 I do have to admit that I gave up on you folks circa 2010 so I have not kept up on the Who's Who of cogneuro, but a cursory citation search shows those tasks are quite widely used to this day--here's the citation history for the BART

#3 kind of says it all and needs no elaboration. I don't think you meant it this way, but what says to me is "I don't care if it is invalid because it is the only way to do something neurosciency."

For #2 I'd go with Eisenberg et al 2019' Nature article showing that experimental inhibitory control tasks don't really predict much of anything and point out that if the task doesn't predict real world behavior it isn't a reflection of the construct of interest and is therefore invalid.

Well, for #1, I'd use your words: "The ability to suppress unwanted or inappropriate actions and impulses (‘response inhibition’) is a crucial component of flexible and goal-directed behavior." (Verbruggen et al., 2019). That's a good definition.

I believe I provoked a lagged set of thoughts on Drew's thoughts....Nice essay

But, Hey, let's not get distracted by these bothersome details, inhibitory control measures are "real" after all and therefore must be valid... 5/

3) the multiple dimensions of inhibition that you could derive from a task were uncorrelated within a task. And, 4) if you bothered to measure multiple inhibitory tasks simultaneously, performance across the tasks was uncorrelated. 4/