Tom Hostler

Living the metascience dream (or nightmare) with AI for science What happens when we go from replication crisis to robustness extremes?

AI makes continuous reproducibility and robustness testing trivial. What happens to science under new levels of scrutiny and stress-testing by default?

Some thoughts on how this could play out, informed by watching open science play out over the last decade.

At last an AI tool I can get behind

“Upload an architectural render. Get back what it'll actually look like on a random Tuesday in November.”

antirender.com

A Researcher Made an AI That Completely Breaks the Online Surveys Scientists Rely On We can no longer trust that survey responses are coming from real people.”

We can no longer trust that survey responses are coming from real people.”

Still Not Significant What to do if your p-value is just over the arbitrary threshold for ‘significance’ of p=0.05? You don’t need to play the significance testing game – there are better methods…

Reminds me of this blog post by Matthew Hankins documenting all the ways of discussing non significant findings. mchankins.wordpress.com/2013/04/21/s...

Personal favourites: "flirting with conventional levels of significance (p>0.1)" & "not absolutely significant but very probably so (p>0.05)"

There’s no ‘could’ about doing less research. Colleagues here being told they can’t go for bids that don’t cover overhead (ie all the charity funders)

Other approaches like post-publication review or open peer review are interesting but don't really solve the labour problem. As Melinda says, the main thing that would help is more permanent funded academic jobs where researchers have the time and space to conduct peer reviews, but that's unlikely!

few major peer reviewed journals and everything else just preprinted wouldn't work in a world where peer reviewed articles are the hard currency of promotion in the sciences - although narrative CVs that prioritise a "top 3 papers" approach can help combat a race for quantity over quality.

in peer review at the moment but its very difficult to see a solution. Paying reviewers is antithetical to many who think reviewers need to be unbiased and 'disinterested' (its also unclear how it would work in a world of 10 second AI reviews - would you pay based on quality?). Having a system of a

defending the institution of science from outside influences (e.g. government officials questioning why they couldn't just decide what to fund rather than having academic grant panels). I also found the discussion on the future of peer review interesting: There is undoubtedly a labour problem

purpose was to take the pressure off overworked editorial boards or allow editors to avoid embarrassment when rejecting their peers' work! The idea that peer review is a crucial part of a 'legitimizing' process that separates real scientific knowledge from pseudoscience came later from scientists

115. Melinda Baldwin: A triple history of Nature, scientific journals, and peer review - BJKS Podcast Melinda Baldwin is an associate professor of history at the University of Maryland. We talk about her work studying the history of Nature, scientific journals more broadly, what it means to be a scien...

Really enjoyed this episode of @bjkspod.bsky.social with @mbaldwin.bsky.social talking about the history (and possible future) of peer review: www.buzzsprout.com/1390924/epis...
Interesting to learn just how recent the idea that "research must be peer reviewed" is (70's), and that its original...

Which Kind of Science Reform What hope is there for science reform, if we can't agree on what to reform? Right now, principles are more important than practices.

How can we reform science? I have some ideas. But I am not sure you’ll like them, because they don’t promise much. elevanth.org/blog/2025/07...

Why I'm more optimistic than others about metascience (or metaresearch)

(I wanted to write up some thoughts on #metascience2025 but I just don't have the time so this thread will have to do.)

There’s some interesting ideas around different funding models – e.g. century grants that last for 100 years – that could provide alternatives. Whether radical ideas like this are practical or palatable in the current neoliberal political climate is another question though.

Why is it that our system is designed in such a way that a research career is typically made up of multiple 2-3 year project-focused positions rather than just getting a job for life straight out of your PhD? Is this the best way to make scientific progress?

But what it tends to neglect are labour relations – casualisation, precarity, etc, that may contribute more to researchers’ actual experience of a ‘culture’ at work. What’s interesting is how these ideas relate to metascience’s core agenda of improving scientific progress.

Area | RGS Geography Journal | Wiley Online Library Universities and policymakers increasingly use ‘research culture’ and ‘research environment’ to govern as well as describe research. Both terms help frame who is considered a research actor; how rese...

by @felicitycallard.bsky.social here (doi.org/10.1111/area...) that argues the term is closely linked to the concept of an “organizational culture”, which predominantly focuses on psychological concepts of leadership, employee relationships, institutional social norms etc. These can be important...

From my personal experience, and maybe because of its close links with the REF, I’ve encountered “Research culture” as a bit of a vague concept associated with research strategy buzzwords, consultation events, and organizational PR. There’s an excellent article...

and essential (ditto the damaging gender stereotypes)” and that “improving research isn't just about tools. It's about people, values, and the conditions under which great research gets done”. However, there’s critical opinion about the limitations of actual research culture initiatives.

with @catdavies.bsky.social when she points out the unhelpful stereotypical distinctions, that “research culture has been incorrectly framed as something soft and nice-to-have, with all the gender implications that brings… Metascience on the other hand is construed as objective, robust...

This is an interesting reflection on the relationship between metascience and research culture. There’s some excellent and relevant work that gets done under the banner of “research culture”, especially around things like inclusivity, recognition, and community building and I agree...

as spectator and also participant in various metascience-informed things like prereg, github etc, I can't help but notice a parallel between metascience-as-management (rather than science) and academic-leadership-as-management (rather than academic leadership)

The point isn't about pharma in particular anyway, it's about any industry potentially laundering its evidence and promoting its product through close links with academic research. Big Tobacco funding health research, Big oil funding climate change research etc. Metascience is not immune.

There's entire books written about the flawed research practices and aggressive marketing of Big pharma so I'm not gonna try and convince you in 280 characters. I also can't really understand how you square "it's extremely naive to distrust tech companies" with "you should always be skeptical"

Big pharma and big tech might have *slightly* more influence and money to spend on advertising, PR, media spin, obfuscation of COIs, and other methods of epistemic corruption than a grad student

I'm not :( just living it vicariously through hashtags I'm afraid.. but thanks for the recommendations!

If a drug company funds a study showing that their product is highly effective, we would be rightly skeptical of the results; if tech companies fund #metascience into how AI can improve science should we be equally sceptical?

Chatbots — LLMs — do not know facts and are not designed to be able to accurately answer factual questions. They are designed to find and mimic patterns of words, probabilistically. When they’re “right” it’s because correct things are often written down, so those patterns are frequent. That’s all.

Looking forward to being part of the panel at this online event this afternoon on critical Metascience, I'll be talking about the alignment between open research reforms and academic capitalism. Details in the post below 👇 #metascience

The chapter ends with a very quotable line, in the context of the power of research and scientific knowledge in society overall : "to put it in a nutshell, knowledge is power, and money is knowledge. Society needs to pay more attention to the distribution of authority over research content".