Essi Kujansuu

GiveWell sought an independent check on the evidence supporting >$1 billion it has granted to its top charities. This fall, Alex Cohen asked whether we could obtain the datasets needed to reproduce the underlying RCTs. We said yes; charging only a modest fee to cover our time.

🧵 + 🐀 + blog post!

#GDRI_rep Update 7: Retraction! Our comment of "Parent–teacher meetings and student outcomes: Evidence from a developing country" has led to the paper being retracted. Our comment is accepted as is.

Short 🧵

I am very happy that the editors took our report seriously. Hope more retractions will follow.

You have to respect a response to a re-response to a response to a report.

Our full report and our codes are here: osf.io/5k4f7/files/.... This OSF page also includes the authors' response, our response to their response and a second authors response to our response to their response.

We thank the authors for taking the time to respond and this back and forth.

#GDRI_rep Update 5: We have a new report. We reproduced the paper entitled "Partisan Effects of Information Campaigns in Competitive Authoritarian Elections: Evidence from Bangladesh" by Ahmed, Hodler and Islam published at the Economic Journal. See below for links to report and authors' responses.

#GDRI_rep Update 3: The paper "Improving Hygiene and Sanitation through Parental Skill Training" by Asadul Islam, Umair Khalil and Tabassum Rahman was conditionally accepted at Economic Development and Cultural Change. We requested a replication package. It has now been withdrawn

#OrphanPapers is apparently a thing now
#EconSky

We have a new report out about the #GDRI_rep. The paper has not been retracted but all three authors have withdrawn their names.

The editor-in-chief at Food Policy (@_Food_Policy) contacted us because an author requested to withdraw their name for: “Determinants and Dynamics of Food Insecurity During COVID-19 in Rural Bangladesh”. We explained the situation and requested a replication package.

We completed a full report on “Food insecurity and mental health of women during COVID-19: Evidence from a developing country” PLOS One and shared it with the authors and journal. Two out of three authors have already requested to withdraw their names.

Here is a list of studies, with links to I4R reports, authors’ responses/statements, and emails sent to the editors and authors (with dates): osf.io/23x64/.

This excel document is being updated every day.

We are now aiming to reproduce about 20 published studies. This number keeps increasing as new GDRI studies are forthcoming or are now conditionally accepted.

#GDRI_rep Update 1: I4R are now reproducing all published papers that use data from GDRI, or are closely related in other ways. Here is a first update on our work. 🧵

I4R doing the lord’s work. Hopefully these will be a collective wake up call to be more careful and transparent about complicated studies in applied micro (and elsewhere too)

While the (quasi-) experimental methods increase credibility of causal results, there are two related concerns that make studies hard to replicate:
1. decline in reporting of null results
2. increase in use of private sector data

a mini thread 🧵based on work with @trfetzer.com

Massive team work in replicating papers and cross checking samples and treatments across multiple papers — what a massive undertaking!
This is evidence that journals *must* require data and code sharing.
I’m wondering if anything in the peer review process could have prevented this

If anything is clear from the last few days, it's surely that Ukraine belongs in the EU

Wow. This is bigger than I thought.
We should be involved in the data collection as much as we are in the data analysis. Well done @i4replication.bsky.social!

Disregarding the sadness of this story, it’s been great working with this awesome team! @jrgptrs.bsky.social @fialalenka.bsky.social @jackfitzgerald.bsky.social @flxhlzmstr.bsky.social @nikljako.bsky.social @anderskjelsrud.bsky.social @kotsadam.bsky.social @essieconomist.bsky.social

The craziest thing about it is that if you want to believe the most “positive” interpretation, you have to agree that they’re bad at RCTs. Messy, don’t follow protocols, etc.

Yet they consistently get the biggest effect sizes!

It’s almost like a proof by contradiction.

This effort was only possible because these journals require data and code repositories. This allowed us to compare data across papers and identify undisclosed connections between 15 studies (e.g., overlapping samples, overlapping interventions, reused treatment assignment).

As a coauthor on the AEJ:AE report, it’s a lot. Oh my god, It’s a lot. In that paper, outcomes are inconsistently handled both in the code and in the field, the paper’s data is connected to a bunch of other experiments, and we find irregularities in the raw survey files.

After being alerted about possible misconduct, the I4R are reproducing published papers that use data from a specific NGO (GDRI). This thread releases the first 2 reports and provides more information about the work and responses/statements from authors journals and journals. 🧵

This is based on our work. We will provide more information on Monday. Stay tuned and thanks Ray and Will for your help!

Sunday 9PM ET / Monday 3AM CET / Monday 1PM Melbourne! See you all then :)

#EconSky

Abel: I am writing in my capacity as Chair of I4R. Last year, I was contacted by a researcher that alerted me of potential scientific misconduct in a set of papers all using data from a specific NGO.🧵

I'm acutely missing the gentle after-vax nudge of try avoid going to the sauna for this one evening. #Finland #Nudge #Sauna #Vaccination

I'm part of the et al., particularly, team MACHINE (AI-led). My suspicion for the finding is that the LLMs are programmed to agree too much -- you notice this sometimes just interacting with them in general.

Those who used to follow me on Twitter know that I tediously made the distinction between a model and an estimation method. In one thread, I considered regressions in a treatment effect context:

Y on 1, D, X
Y on 1, D, X, D*(X - Xbar)

It is common to refer to the first as "OLS."