Felix Holzmeister

Researchers face incentives to write up their empirical findings in a way that maximizes publication success. We analyze the language of journal articles and its association with statistical significance to explore questionable research practices at the stage of writing up articles, using 140,606 articles from health, biology, psychology, economics, and multidisciplinary journals over 32 years. For most disciplines, a higher share of non-significant main findings is associated with more hedging and negative striking words, fewer positive striking words, and fewer superlatives. We neither find evidence that authors upsell ambiguous results using sensational language nor that ambiguous results are written up less readably. Contrarily, articles with a higher share of statistically significant main findings are written up more sensationally. We find that emphasis on (marginal) statistical significance increases with the share of non-significant main findings, reflecting a dichotomized interpretation of p-values based on arbitrary thresholds. Particularly, p-excuses give the impression of statistical significance when the finding is actually not significant, consistent with the notion of 'spin'. This study provides empirical insights that might help researchers to self-reflect on writing up empirical findings. More training and fewer incentives to sell findings using sensational language and spin can help to improve academic writing.

"p-Excuses"

E.g., p = .056 is “marginally significant” or “close to being significant”

🔹 Most p-excuses are used in psychology.

🔹 The least p-excuses are used in health.

Preprint: osf.io/preprints/me...

#MetaSci #AcademicSky 🧪

#GDRI_rep Update 9: We have been waiting a couple of months to make two announcements. A few journals were in the process of retracting GDRI article. The editors informed the authors and then lawyers got involved. Who knows how long it will take.

Join us for the virtual Replication Games😍🤓 on Nov 13th! (You don't have to be in the UK - join us from wherever you are😉)

🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥

Thought about scientific consensus recently? We have a new DP @i4replication.bsky.social that probes into the famous replication debate between Acemoglu, Johnson & Robinson (AJR) and Albouy - and how experts assess this debate. We find that they disagree. 1/8 www.econstor.eu/bitstream/10...

Come joun us in Münster, it will be awesome😉😎

Excited to finally have my first paper officially published! "How Do Psychology Journals Handle Postpublication Critique?" is now online with AMPPS. Huge thanks to my supervisors @tomhardwicke.bsky.social and @simine.com and co-authors @bethclarke.bsky.social, N Moodie, S Schiavone, and R Thibault ☺️

#GDRI_rep Update 8a: Major updates coming up over the next few weeks.

1st update: We reproduced "Parent–teacher meetings and student outcomes" some time ago. The paper was then retracted by the editors of the Euro Econ Rev.

Additional studies are connected (see diagram)
🧵

📢 Come work with us!

🏛️ Faculty of Economics & Statistics, UIBK @econstatuibk.bsky.social‬ @uniinnsbruck.bsky.social‬

‼️ Three job openings for Full Professor positions

- Data Science

- Microeconomics

- Behavioral Economics

Please share! #datascience #economics

www.uibk.ac.at/de/info-beru...

Open Science Summer School

Are you a PhD-Candidate, Prae- or Post-Doc interested in open research practices and want to spend a few summer days in beautiful #Innsbruck?

👉 Then the #OpenScience Summer School at @uniinnsbruck.bsky.social might be the perfect match for you!
www.uibk.ac.at/en/weiterbil...

A re-analysis of Ciacci's (2024) "Banning the purchase of sex increases cases of rape: Evidence from Sweden" Journal of Population Economics reveals major issues. A year ago, reproducers Adema, Folke, and Rickne found coding errors driving the paper's key results. Let's unpack this in a 🧵

🚨 REPLICATION REPORT UPDATE: One year ago, a tweet by John Holbein alerted me, @ollefolke.bsky.social, and @jopieboy.bsky.social to a paper with a shocking result about Sweden’s law criminalizing the purchase of sex.🧵

American chopper meme. Two men are yelling at each other. YOU SHOULD REPORT THE MARGINAL EFFECTS. I REPORTED THE COEFFICIENT VALUES. WHAT MORE DO YOU WANT? PEOPLE HAVE A HARD TIME INTERPRETING THOSE VALUES. MINUS MEANS LESS AND PLUS MEANS MORE. THAT'S GOOD ENOUGH! INTERPRETING EFFECTS ON THE SCALE OF INTEREST IS NATURAL AND BENEFICIAL.

also

“Try to balance the baseline”: A comment on “parent-teacher meetings and student outcomes: Evidence from a developing country” by Islam (2019) Islam (2019) reports results from a randomized field experiment in Bangladesh that examines the effects of parent-teacher meetings on student test sco…

Great to see our comment now available online in the European Economic Review! (w/ @carlbonander.bsky.social, @nikljako.bsky.social, @flxhlzmstr.bsky.social, Gunther Bensch & Abel Brodeur @i4replication.bsky.social) 🙌

Our proposal for protocolled replicability & robustness assessments is out in Q Open. The main aim is to structure robustness reproductions, but also to extend the scope to external/construct validity as well as pre-specification checks. Comments welcome! academic.oup.com/qopen/advanc...

Do men increase their support for gender equality if they have daughters?

Some papers suggest so!

Join this meta-science project as a co-author, work with SOEP data and help us all learn more!

🔔 Call for Participation 🔔
Lab² is inviting researchers to take part in a multi-analyst study on the effects of having daughters on various outcomes.
Join this metascience project as a co-author and gain the opportunity to work with SOEP data!

Collaborators are looking for participants for a new multi-analyst study using SOEP data. For more information on registration, eligibility criteria, requirements, procedure, and timeline, please visit www.manydaughters.com. Deadline for registration is April 27

#GDRI_rep Update 7: Retraction! Our comment of "Parent–teacher meetings and student outcomes: Evidence from a developing country" has led to the paper being retracted. Our comment is accepted as is.

Short 🧵

When One Size Doesn’t Fit All Many psychological scientists are now calling for a “heterogeneity revolution,” focused on uncovering individual and contextual differences in experimental outcomes.

Many psychological scientists are now calling for a “heterogeneity revolution,” focused on uncovering individual and contextual differences in experimental outcomes.

#GDRI_rep Update 5: We have a new report. We reproduced the paper entitled "Partisan Effects of Information Campaigns in Competitive Authoritarian Elections: Evidence from Bangladesh" by Ahmed, Hodler and Islam published at the Economic Journal. See below for links to report and authors' responses.

Since we started mass reproducing GDRI, we have requested several replication packages. We have not managed to get a single new replication package as of today (excluding updated packages for one article at the request of editors).

In a new @i4replication.bsky.social report we replicate Islam (2019). Main issue: the randomized controlled trial was not randomized. And many other things on top of that. See full report for details. (w/ @carlbonander.bsky.social, @nikljako.bsky.social, @flxhlzmstr.bsky.social, Bensch & Brodeur)

#GDRI_rep Update 4: We have a new report. We reproduced the paper entitled "Parent–teacher meetings and student outcomes" by Islam @ European Economic Review. See below for links to report and author's response.

#GDRI_rep Update 3: The paper "Improving Hygiene and Sanitation through Parental Skill Training" by Asadul Islam, Umair Khalil and Tabassum Rahman was conditionally accepted at Economic Development and Cultural Change. We requested a replication package. It has now been withdrawn

#GDRI_rep Update 1: I4R are now reproducing all published papers that use data from GDRI, or are closely related in other ways. Here is a first update on our work. 🧵

After a long wait, the working paper for the Many-Economists Project: The Sources of Researcher Variation in Economics. We had 146 teams perform the same research three times, each time with less freedom. What source of freedom leads to different choices and results? papers.ssrn.com/sol3/papers....

I4R doing the lord’s work. Hopefully these will be a collective wake up call to be more careful and transparent about complicated studies in applied micro (and elsewhere too)

Disregarding the sadness of this story, it’s been great working with this awesome team! @jrgptrs.bsky.social @fialalenka.bsky.social @jackfitzgerald.bsky.social @flxhlzmstr.bsky.social @nikljako.bsky.social @anderskjelsrud.bsky.social @kotsadam.bsky.social @essieconomist.bsky.social

As a coauthor on the AEJ:AE report, it’s a lot. Oh my god, It’s a lot. In that paper, outcomes are inconsistently handled both in the code and in the field, the paper’s data is connected to a bunch of other experiments, and we find irregularities in the raw survey files.