Jamie Cummins

Many error remain to be found in clinical trials. Patients deserve reliable results. Kudos to these authors for their persistent work to correct the record.

A thread about being wrong:

5 years ago, we wrote a paper about how how newly enfranchised 16-year-olds vote in Austria. But we were wrong.

This year, @elisabethgraf.bsky.social, @schnizzl.bsky.social, Sylvia Kritzinger and I are setting the record straight: authors.elsevier.com/c/1juT5xRaZk...

"In 2019 we notified journals about serious integrity concerns in 172 clinical trials. Over five years later, only 22 have been retracted. The 135 unretracted trials have 1989 citations in systematic reviews, clinical guidelines, and consensus statements"

[paraphrased]
www.bmj.com/content/390/...

YouTube video by Bennett Institute for Applied Data Science OpenSAFELY in a nutshell

If you'd like to learn more about how OpenSAFELY works - and how we solved the privacy and efficiency challenges, to make national GP data securely accessible - here's a 5 minute video!

www.youtube.com/watch?v=GRjR...

🚨 SynthNet is out 🚨
Researchers propose new constructs and measures faster than anyone can track. We (@anniria.bsky.social @ruben.the100.ci) built a search engine to check what already exists and help identify redundancies; indexing 74,000 scales from ~31,500 instruments in APA PsycTests. 🧵1/3

I was thrilled to have been invited by @sakshighai.bsky.social to speak to folk at LSE on Wednesday about methodological and inferential issues that have cropped up in social science attempts to study large language models!

Congratulations to @simine.com for winning the Einstein Foundation Individual Award! 🎉

A well-deserved recognition for her seminal efforts to improve scientific rigor, which includes instituting detailed checks for errors and computational reproducibility at Psychological Science.

🏆 Individual: @simine.com, psychologist at @unimelb.bsky.social & editor-in-chief of Psychological Science, is recognized for pioneering methodological rigor, reproducibility & collaborative research, driving initiatives such as @improvingpsych.org & the journal Collabra @ucpress.bsky.social. (2/5)

The speaker at the lectern

Title slide

Next: Jack Wilkinson @jdwilko.bsky.social with 'Problematic clinical trials and the threat to evidence synthesis'
Systematic reviews are considered the cornerstone of medicine. But some of the eligible trials that could be included might be problematic. They could get included.
#IRICSydney

I think this is an overly pessimistic take from the @bmj.com.

Sharing data does not inherently increase trust, rather it enables verification which allows for trust calibration.

This example is a win. Serious issues were rapidly detected that would not have been without mandatory data sharing.

With every LLM since GPT-4, I've tried a game: ask it to commit a 20 Questions guess to a cipher, we play 20 Questions, and then we see if what it claims to have been its original choice is consistent with its cipher.

ChatGPT-5.1 Thinking is the first model to do this successfully!

Synchronous Robustness Reports could explore implications of different analytical choices – but they could still suffer from bias. Hardwicke argues that preregistration is crucial to prevent it.

@tomhardwicke.bsky.social

Are methodological and causal inference errors creating a false impression that the gut microbiome causes autism? In this strong analysis, Mitchell, Dahly, and Bishop question the evidence.

They show that triangulation in science requires multiple robust lines of research.

Yes, like a Netflix documentary included IN EVERY SOCIAL PSYCHOLOGY TEXTBOOK

There is a lot of fuss today over whether chatbots can replace human participants in social sciences research when the solution is obvious: ask chatbots to simulate the views of social scientists and survey them on attitudes towards chatbots as substitutes for human subjects.

Delighted to support MU Psych Soc's invited lecture on Forensic Metascience by departmental alum, Dr Jamie Cummins @jamiecummins.bsky.social whose work in this area seeks to enhance rigour & accuracy in scientific reporting.

Sincere thanks to Dr Cummins. #MUPsychologyAt25

The threat of analytic flexibility in using large language models to simulate human data: A call to attention Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research. How...

Super interesting, looking forward to reading this later. You may find this of interest: arxiv.org/abs/2509.13397

Computational Turing Test Reveals Systematic Differences Between Human and AI Language Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior, based on the assumption that they can generate realistic, human-like text. Yet this assumption rem...

LLMs are now widely used in social science as stand-ins for humans—assuming they can produce realistic, human-like text

But... can they? We don’t actually know.

In our new study, we develop a Computational Turing Test.

And our findings are striking:
LLMs may be far less human-like than we think.🧵

It was such an honour and privilege to be back at my alma mater 9 years (!!!) after finishing my undergraduate degree to give a talk as part of psych department's 25 year anniversary!

Lovely to welcome back Dr @jamiecummins.bsky.social for tonight's @mupsychology.bsky.social talk as part of our #MUpsychologyAt25 events @maynoothuniversity.ie

My master thesis file name on my old university's thesis archive site still makes me chuckle.

example #2345432 that nobody really knows what they mean by "AI"

Will AI solve medicine?

This year Demis Hassabis predicted AI could cure all disease in a decade.

But other scientists like Claus Wilke & Derek Lowe say biology is far more complex, or progress will be limited by clinical trials & economics.

In a new 4hr podcast episode of *Hard Drugs*, we answer: Will AI solve medicine?

me with some garden hoses connected in a X -> Z <- Y fashion. If I shut the valve at Z, water from X spills out at Y

I built a DAG diagram with garden hoses for teaching.
Pictured: a collider bias diagram, inspired by a blocked pipe situation I experienced (which I credit with giving me the intuition though it also ruined my belongings in the flooded cellar).

"Traumatized Mr. Incredible" meme with "Data and code available", "After looking at data & code"

The 2011 Presidential Debate where Sean Gallagher loses the election
part 1 #aras25

AI Surrogates and illusions of generalizability in cognitive science Recent advances in artificial intelligence (AI) have generated enthusiasm for using AI simulations of human research participants to generate new know…

Can AI simulations of human research participants advance cognitive science? In @cp-trendscognsci.bsky.social, @lmesseri.bsky.social & I analyze this vision. We show how “AI Surrogates” entrench practices that limit the generalizability of cognitive science while aspiring to do the opposite. 1/

New hobby:

Remaking article abstracts as movie trailers to expose hype and fearmongering.

The threat of analytic flexibility in using large language models to simulate human data: A call to attention Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research.…

"Silicon samples" - using LLMs to generate fake survey responses instead of recruiting humans. Sounds efficient until you realize small model tweaks completely flip your results. Shortcuts in research usually aren't.

Psychologists running empirical studies to rediscover engineering design choices is such a strange genre of papers. By all means, run studies on LLM judgments -- but what else than lexical co-occurence and statistical priors would they be based on??