Randy Ellis

Here's what a Cohen's d = 22 looks like. Totally normal. See it all the time in my own data...

Today in that-didn't-happen: Cohen's d = 22.

Williams et al. (2014) has 145 citations, putting it in top 1% of most cited psych articles.

It is a load-bearing publication in its area, despite having impossible results.

pubpeer.com/publications...

It must be very hard to publish null results Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100,000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science.

I have a new paper. We look at ~all stats articles in political science post-2010 & show that 94% have abstracts that claim to reject a null. Only 2% present only null results. This is hard to explain unless the research process has a filter that only lets rejections through.

Without publication bias, we might not need many replications. With publication bias, 20% to 40% might be justified (but of course, extremely dependent on the assumptions in the simulations!). If the field is a mess, we need a lot of replication studies to clean up!

My colleague Krist Vaessen wrote a new book: “Neomania: How our obsession with innovation is failing science, and how to restore trust”. It's a great analysis how the drive for novelty hinders reliable scientific progress. Open Access, so read it here: books.openbookpublishers.com/10.11647/obp...

YouTube video by Metascience Matters 300+ retractions, image manipulation, and why science should be boring | Metascience Matters #3

Here's my conversation with Mu Yang on Metascience Matters: www.youtube.com/watch?v=E2EK...

We discussed her work as a scientific sleuth, academic incentives for positive data, individual cases she has pursued, and why she loves being a sleuth.

Also on Spotify: open.spotify.com/episode/16R6...

New submission format at SBE:
“Replications as Registered Reports”

link.springer.com/journal/1118...

You can get "in-principle acceptance" before data collection even begins; final paper gets published regardless the results, if the study is conducted rigorously.

#EconSky

Call for metascience grants has a focus on three areas:

🔸️ The impact of artificial intelligence on scientific practice and the research landscape

🔸️ The effective design and leadership of research organisations

🔸️ Scientometrics approaches to understanding research excellence, efficiency and equity

Some discussion about this in a conversation I’ll be releasing in early March, thanks Rasu!

Metascience Matters I'm Randy Ellis, a computational biologist and neuroscientist who cares about metascience, reproducibility, and rigor in science. I started Metascience Matters because I believe science communication ...

YouTube: youtube.com/@metascience...
Spotify: open.spotify.com/show/7coSExb...
Apple: podcasts.apple.com/us/podcast/m...
iHeart: www.iheart.com/podcast/269-...

YouTube video by Metascience Matters Exposomics, Vibration-of-Effects, and the Future of AI in Health | Metascience Matters #1

I started a podcast! Metascience Matters features conversations with metascientists.

Two episodes are live:

Chirag Patel on Exposomics, and Vibration of Effects: youtu.be/RT2nypyb-iM?...

@floriannaudet.bsky.social on Clinical Trials, Registered Reports, and Psychiatry: youtu.be/fn4qtnc99Xo?...

This paper in Management Science has been cited more than 6,000 times. Wall Street execs, top govt officials, and even a former U.S. Vice President have all referenced it. It’s fatally flawed, and the scholarly community refuses to do anything about it.
statmodeling.stat.columbia.edu/2026/01/22/a...

Congratulations!

YouTube video by Richard McElreath Statistical Rethinking 2026 - Lecture B01 - Multilevel Models

Statistical Rethinking 2026 Lecture B01 Multilevel Models is online. This is the first lecture of the "experienced" section, in which we start with multilevel models and venture into vast covariance spaces. Full lecture list still here: github.com/rmcelreath/s...

With some trepidation, I'm putting this out into the world:
gershmanlab.com/textbook.html
It's a textbook called Computational Foundations of Cognitive Neuroscience, which I wrote for my class.

My hope is that this will be a living document, continuously improved as I get feedback.

Data sharing helps avoid “smoking gun” claims of topological milestones Manipulating the topology of electronic bands can realize new states of matter, with possible implications for information technology. A central question is how to tell whether a topological regime ha...

The rarest of sights - a big glossy journal publishing negative replications! Yes, we had to bundle 4 replications into one article AND we had to wait 2 (!!) years in peer review, but here we are:

www.science.org/doi/10.1126/...

Abandon “Abundance” The latest Democratic fad sidelines equality and justice in favor of a focus on cutting red tape. This is not the path forward.

Here’s two, both from the left:

www.currentaffairs.org/news/abandon...

www.levernews.com/abundance-is...

Important. Remember you can’t take science at face value: publication bias is everywhere.

Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology A project to repeat experiments from high-impact papers in cancer biology encountered a series of challenges, many of which were caused by a lack of detail in the original papers.

Have had the same surprise, the Errington paper is a milestone not just for the scale of the project, but they also have a separate paper about issues they encountered running the project, hugely instructive for anyone doing replication studies elifesciences.org/articles/67995

Investigating the replicability of preclinical cancer biology A project to repeat experiments from high-impact papers in cancer biology found that the effects observed in replications were frequently weaker than, or inconsistent with, the effects reported in the...

The other classic ref is Begley and Ellis 2012 (www.nature.com/articles/483... ) which has the same issue, but the ref I always cite now is Errington et al :) elifesciences.org/articles/71601

# Bayesians are to frequentists as vegetarians are to murderers I have close friends who do not eat meat, for moral reasons. It's not that they find meat disgusting. In fact, they find it delicious. Instead they regard meat as murder. And yet we continue to be friends. I myself do not think meat is murder. I regard it in fact as an ordinary and normative part of human society. It's so commonplace. How could it be murder? This sort of moral incompatibility is commonplace. Vegetarians and vegans have to put up with assholes like me all the time. They are surrounded.

When I take train journeys, I sometimes write things. Things that I probably shouldn't publish

To anyone who may have been present at my talk today in which I intimated that you can just, you know, do this: you can just, you know, do this.

Registered reports now

For many social dilemma's in Science (e.g. the slow uptake of diamond open access journals) stronger top down management is necessary. It won't just happen. If scientists will not create this management themselves, someone is going to create it for us.

For comparison, the profits of four publishers (2.64B) amount to 5.58% of the FY2024 NIH budget. Revenues (7.36B) are *15.52%*. I agree with the authors' perspective that funders, governments, and universities should lead efforts to change this. All journals should be diamond open-access.

A table showing profit margins of major publishers. A snippet of text related to this table is below. 1. The four-fold drain 1.1 Money Currently, academic publishing is dominated by profit-oriented, multinational companies for whom scientific knowledge is a commodity to be sold back to the academic community who created it. The dominant four are Elsevier, Springer Nature, Wiley and Taylor & Francis, which collectively generated over US$7.1 billion in revenue from journal publishing in 2024 alone, and over US$12 billion in profits between 2019 and 2024 (Table 1A). Their profit margins have always been over 30% in the last five years, and for the largest publisher (Elsevier) always over 37%. Against many comparators, across many sectors, scientific publishing is one of the most consistently profitable industries (Table S1). These financial arrangements make a substantial difference to science budgets. In 2024, 46% of Elsevier revenues and 53% of Taylor & Francis revenues were generated in North America, meaning that North American researchers were charged over US$2.27 billion by just two for-profit publishers. The Canadian research councils and the US National Science Foundation were allocated US$9.3 billion in that year.

A figure detailing the drain on researcher time. 1. The four-fold drain 1.2 Time The number of papers published each year is growing faster than the scientific workforce, with the number of papers per researcher almost doubling between 1996 and 2022 (Figure 1A). This reflects the fact that publishers’ commercial desire to publish (sell) more material has aligned well with the competitive prestige culture in which publications help secure jobs, grants, promotions, and awards. To the extent that this growth is driven by a pressure for profit, rather than scholarly imperatives, it distorts the way researchers spend their time. The publishing system depends on unpaid reviewer labour, estimated to be over 130 million unpaid hours annually in 2020 alone (9). Researchers have complained about the demands of peer-review for decades, but the scale of the problem is now worse, with editors reporting widespread difficulties recruiting reviewers. The growth in publications involves not only the authors’ time, but that of academic editors and reviewers who are dealing with so many review demands. Even more seriously, the imperative to produce ever more articles reshapes the nature of scientific inquiry. Evidence across multiple fields shows that more papers result in ‘ossification’, not new ideas (10). It may seem paradoxical that more papers can slow progress until one considers how it affects researchers’ time. While rewards remain tied to volume, prestige, and impact of publications, researchers will be nudged away from riskier, local, interdisciplinary, and long-term work. The result is a treadmill of constant activity with limited progress whereas core scholarly practices – such as reading, reflecting and engaging with others’ contributions – is de-prioritized. What looks like productivity often masks intellectual exhaustion built on a demoralizing, narrowing scientific vision.

A table of profit margins across industries. The section of text related to this table is below: 1. The four-fold drain 1.1 Money Currently, academic publishing is dominated by profit-oriented, multinational companies for whom scientific knowledge is a commodity to be sold back to the academic community who created it. The dominant four are Elsevier, Springer Nature, Wiley and Taylor & Francis, which collectively generated over US$7.1 billion in revenue from journal publishing in 2024 alone, and over US$12 billion in profits between 2019 and 2024 (Table 1A). Their profit margins have always been over 30% in the last five years, and for the largest publisher (Elsevier) always over 37%. Against many comparators, across many sectors, scientific publishing is one of the most consistently profitable industries (Table S1). These financial arrangements make a substantial difference to science budgets. In 2024, 46% of Elsevier revenues and 53% of Taylor & Francis revenues were generated in North America, meaning that North American researchers were charged over US$2.27 billion by just two for-profit publishers. The Canadian research councils and the US National Science Foundation were allocated US$9.3 billion in that year.

The costs of inaction are plain: wasted public funds, lost researcher time, compromised scientific integrity and eroded public trust. Today, the system rewards commercial publishers first, and science second. Without bold action from the funders we risk continuing to pour resources into a system that prioritizes profit over the advancement of scientific knowledge.

We wrote the Strain on scientific publishing to highlight the problems of time & trust. With a fantastic group of co-authors, we present The Drain of Scientific Publishing:

a 🧵 1/n

Drain: arxiv.org/abs/2511.04820
Strain: direct.mit.edu/qss/article/...
Oligopoly: direct.mit.edu/qss/article/...

What if NIH had been 40% smaller? Replaying history with less NIH funding shows widespread impacts on drug-linked research

This may be the most important paper ever published about NIH funded research.

www.science.org/doi/10.1126/...

Promotional poster for the LMU Open Science Center Summer School 2025 Public Lecture. The poster features a photo of speaker Prof. Dr. Felix Schönbrodt. The text states the lecture focuses on the replicability crisis and open research initiatives, time and date of lecture being 9:45-10:45 CEST on September 15 2025, and instructions to register online. Logos of LMU Open Science Center, LMU, and Universitätsbibliothek München are displayed at the bottom.

🔍Find out about the replicability crisis across fields & open research initiatives you can implement in your own work!
Attend the “Replicability Crisis” lecture by Prof. Dr. Felix Schönbrodt (@nicebread.bsky.social) on Mon 15 Sept, 9:45-10:45.
👉Register here: www.pretix.osc.lmu.de/lmu-osc/OSSS...

This was an absolutely brilliant talk.

SV: “Journal prestige is here to stay, but we shouldn’t give it away for free.”

This is what I've been saying since 2023 (image below)

"prediction: use of "AI" [...] will come to be broadly associated with cheating, deception, lack of respect for other people, and low quality work that cannot be trusted in important settings"