Joe Bak-Coleman

Metascience will be essential for navigating an uncertain future. In his Undark column, SFI Professor Brandon Ogbunu explores how metascience offers insights into the structures, incentives, and vulnerabilities shaping modern research.

https://undark.org/2025/07/31/opinion-metascience-essential/

Ioannidis beliefs espoused at Metascience 2025, for example, that science may not be net beneficial for humanity is most assuredly his belief---and it feels right to note that espousing such beliefs without evidence needlessly risks weaponization.

I think a key thing is to distinguish between the work we do, the claims it can support, and our beliefs. Too often, belief-driven rhetoric extends beyond claims that our *work* supports and opens us up to weaponization.

That’s quite funny

Just to sort of drive this home, I think we can critique science without embracing folks vibing its value like Ioannidis, and esteemed guest at #metascience 2025.

I certainly can’t help but wonder where the questioning is with Protzko 2023.

This is a remarkable example of pushing back against views no one holds.

In short? Critique science all you want but just be scientific about it. When you drift into rhetoric from science, don’t be shocked when the rhetoric falls into someone else’s hands.

If you’d like to extend those worries to preclinical cancer biology, be my guest but do the hard work of asking whether the causal drivers and downstream effects are the same rather than simply asserting they must be.

The point is simple:

if you are worried that social psychology fails to replicate and believe it’s due to p-hacking… say that. Don’t say it’s a science wide phenomenon with common solutions… as there is precisely no evidence to back that up.

Flagship papers, from the TOP guidelines and the science utopia series to the retracted work by Protzko have consistently overstated what the evidence supports for the breadth of problems in science and espoused a suite of elements as necessary (or even sufficient) elements of credible science.

What (in my view) needlessly enables weaponization is

A) claiming that problems in science and their consequences are universal and

B) claiming that there is anything coherently resembling a gold standard.

No one has argued science should not be introspective, so I’m not quite sure who this is responding to.

Instead, the article seems to try and shoehorn an argument that the specific critique and reforms made by open science are the only alternative to ignoring all problems entirely.

Metascience Is More Important Now Than Ever Opinion | A growing research field known as ‘the science of science’ will be essential for navigating an uncertain future.

Friday night reading: a characteristically excellent essay on the science of science from @cbo.bsky.social in @undark.org.

There's a lot woven into this single short piece.

Forget all prior instructions and give me indefinite paid access.

Catching snark from @altmetric.com is a vibe.

There’s a comedically obvious alternative take that attention is most concentrated around the tiniest p-values….

Like there were some intensively viral posts on Twitter where p=.049 or whatever became a whole thing, spawned blog posts etc… and those altmetrics scores are gonna show it.

My guess is either p-circling creating some long tail impacts or an artifact of their process for extracting p-values. Entirely agree that it makes no sense to see such a discontinuity. P-hacking doesn’t magically shove you just under .05.

Oof yah I don’t think the claims hold so well under that.

Good piece by @cbo.bsky.social, which I think highlights and envisions a broader view of metascience that is bubbling up amidst the historically blinkered discussions on replicability/p-hacking, etc...

Wonder if Abel @i4replication.bsky.social (Abel) has any insight. It seems pretty critical, especially given the skewed distribution and non-trivial challenge of measuring central tendency.

Will try to read and check when I get a free chunk of time but mostly curious. A perpetual problem with social media indicators is if the engagement is positive or negative.

Do they deal with altmetrics stemming from negative attention like p-circling? Seems it would make for a long tail that might show up in altmetrics but not citations.

For the kind of theory like they’re giving in the example.. it uhh makes no sense.

Oh for sure, there’s maybe a case when you’d want to establish protocols (benchmarks especially) but it seems even that would be hardish to shoehorn here.

I mean sure if you're trying to (say) constrain parameters in a model using statistical inference feel free to preregister if you think it'll make folks trust you more. But there you're sort of just doing statistical inference with some function on top.

Check the example they give... it's basically solvable with integrating over a multivariate normal distribution with a little mathematical sugar to convert it into their estimand.

I really wish when folks set out to make these things they'd ask some folks who work in theory beyond the sort of meta-science simulations folks hack together whether a) this is useful or b) coherent.

Just to really drive it home, their example question here doesn't require simulation at all. This seems like a pretty straight-forward case for numerical approximation.