Jack Fitzgerald

Had a wonderful time organizing the scientific side of the CBS Replication Games! Thank you to the replicators for your hard work!

For those without institutional access to NHB, Nature has provided the following link, from which you can access my Matters Arising for free: rdcu.be/eYab2

27/27

I want to thank the editors of @nathumbehav.nature.com for taking this matter seriously and keeping in touch with me over the past 21 months. Their commitment to open science made this correction possible. 26/x

In addition to the lives at stake, governments spend hundreds of billions each year on counterterrorism. To determine what policies best deter terrorism is to answer a trillion-dollar question. Unfortunately, this study categorically cannot answer that question. 25/x

For example: arrest rates are computed as (arrests/attacks), but countries can have more arrests than attacks. In most country-years, arrest rates are either >100% or a positive number divided by 0. Belgium apparently had a terrorism arrest rate of 16,600% in 2018. 24/x

I don’t have space in the 1200-word limit of a Matters Arising to cover everything wrong I found with this paper. The more you look, the more you find. Many ‘minor’ details that would be worth a comment in their own right are relegated to the Supplementary Material. 23/x

Line graphs displaying time series of the raw number of terrorist attacks over time in each of 28 EU member states. A red box highlights the time series for the United Kingdom. Based on WEA's data.

The original paper explicitly highlights how much terrorism ‘declined’ in the UK after the COVID-19 pandemic. But the decline after 2020 is only that stark because all terrorism-related variables are imputed to 0, without disclosure, after the UK left the EU in 2020. 22/x

This means that the paper’s panel dataset on terrorism (enforcement) in the EU included country-years where the country *wasn’t even yet/anymore part of the EU*, assigning all terrorism-related variables to zero for these country-years without telling anyone. 21/x

Countries not currently in the EU don’t appear in the EU’s TE-SAT reports, WEA's main raw data. This affects four countries in WEA’s data. When a country isn't in a year’s TE-SAT reports, all terrorism variables for that country-year are imputed as 0s without disclosure. 20/x

This isn’t the only variable for which this sort of zero-imputation is done. As I was working, I remember the stark realization I felt after thinking about the countries that were in the EU from 2006-2021, and wondering aloud: “When did Brexit happen again?” 19/x

The paper is effectively simulating full data coverage without telling anyone. Over 57% of the pre-2021 country-years in its data contain average sentence durations of 0, when really the raw data sources contain no data on sentences in these country-years. 18/x

Consider Luxembourg. Average sentence length is never reported in the paper’s raw data for Luxembourg because they experience no terror attacks. The paper imputes these lengths as 0s, falsely implying that the average terrorism convict in Luxembourg gets a 0-year sentence. 17/x

It doesn't stop there. Average sentence lengths are undefined in countries that haven’t sentenced anyone for terrorism, so the paper’s raw data only reports average sentence lengths for a subset of countries. Missing sentence lengths are imputed as 0s without disclosure. 16/x

Image of panel (a) in Table 1 from the Matters Arising.

Trying to reproduce the paper’s main table using the variables actually disclosed in the paper yields estimates that are much smaller than those published in the paper. Almost everything loses statistical significance, and many coefficients are plainly inestimable. 15/x

4x2 array of scatterplots with OLS lines of best fit. In the first row of graphs, the y-axis represents values of NewArrest; the x-axis of the left (right) graph represents values of (IHS) arrest rates. In the second row of graphs, the y-axis represents values of CharSin; the x-axis of the left (right) graph represents values of (IHS) charge rates. In the third row of graphs, the y-axis represents values of ConvictionSin; the x-axis of the left (right) graph represents values of (IHS) convictionrate. In the fourth row of graphs, the y-axis represents values of SentenceSin; the x-axis of the left (right) graph represents values of (IHS) sentence. In the top two graphs, the OLS line of best fit is downward-sloping, whereas it is upward-sloping in the bottom six graphs. No line of best fit in the second column of graphs perfectly intersects all points.

None of these variables can be reconstructed by taking IHS transformations of the underlying rate variables they’re supposedly constructed from. Again, rates of zero somehow get mapped to many different positive values, as do values that are missing due to division by zero. 14/x

Image of the first few observations from WEA's replication dataset. Red boxes highlight variables ConvictionSin, SentenceSin, CharSin, and NewArrest. Source: https://zenodo.org/records/8196717.

WEA's repository includes NewArrest, CharSin, ConvictionSin, and SentenceSin. Though these variables are supposedly constructed from rates that often require division by 0, there are no missing values in any of these variables. These are used to produce WEA’s main table. 13/x

But the observation counts reported in the original paper imply 0 missing observations. This is because the paper’s main table is produced using different variables than the ones disclosed in the paper. 12/x

Here are reported formulas for constructing 3 of WEA’s main independent variables:

• Arrest rate: Arrests/Attacks
• Charge rate: Charges/Arrests
• Conviction rate: Convictions/Charges

For most country-years, at least one of these variables requires division by zero. 11/x

The outcome variable is just the tip of the iceberg. There are irregularities like this in every main variable in the paper. 10/x

This paper had 1 revision round before acceptance. A reviewer raised concerns about how the data transformation affected country-years with zero attacks. After a few extra sentences + citations about the IHS transformation, the reviewer was satisfied and accepted the paper. 9/x

These national time series were printed in the original article (with different formatting). How wasn’t this caught in peer review? Thanks to NHB’s open peer review, we can actually see that *it nearly was*. 8/x

static-content.springer.com/esm/art%3A10...

Displays two scatterplots with OLS lines of best fit. In each graph, the y-axis shows values of DVsin. In the left scatterplot, the x-axis is the raw attack rate (attacks/population in hundreds of thousands), whereas in the right scatterplot, the x-axis is this attack rate transformed using the inverse hyperbolic sine function. In both graphs, a vertical line of dots is visible when x = 0, and the OLS line of best fit slopes downwards. Two red boxes highlight the vertical lines of dots where x  = 0 in each graph.

The 305 post-2006 country-years that experience zero terror attacks also get assigned to 292 different positive values of DVSin. This is impossible, as the IHS of 0 is 0. This implies that the paper’s main outcome cannot possibly be constructed as described in the paper. 7/x

Displays two scatterplots with OLS lines of best fit. In each graph, the y-axis shows values of DVsin. In the left scatterplot, the x-axis is the raw attack rate (attacks/population in hundreds of thousands), whereas in the right scatterplot, the x-axis is this attack rate transformed using the inverse hyperbolic sine function. In both graphs, a vertical line of dots is visible when x = 0, and the OLS line of best fit slopes downwards.

This main outcome is hard-coded in the dataset as ‘DVSin’. Since the dataset contains attack + population counts, I can directly compute per capita attack rates. DVSin is actually significantly *negatively* correlated with per capita attack rates + their IHS transformation. 6/x

Time series of different measures of terrorist attacks over time in each of 28 EU member states. The top panel shows national time series for the raw count of terrorist attacks. The bottom panel shows national time series for 'DVsin'. Based on data from WEA.

The top panel shows terror attacks over time in each country. The bottom panel shows the paper’s main outcome, reported as the inverse hyperbolic sine (IHS) of per capita attack rates. Impossibly, the 9 countries with 0 attacks have positive IHS attack rates that change over time. 5/x

I started working on WEA as part of @i4replication.bsky.social's and NHB's joint initiative to systematically re-examine NHB's articles. I vividly remember the first time I read this paper, when I first wrote in my notes: “This should not be possible.” 4/x

doi.org/10.1038/s41562-023-01807-2

Before the retraction, there was an Editor’s Note warning about this paper's data since Oct 2024. As the reproducer @i4replication.bsky.social was referring to in this post, now that the Matters Arising is out, let's talk about why. 3/x

bsky.app/profile/i4re...

The original paper by Wolfowicz et al. - henceforth WEA - is about the legal determinants of terrorism. Its title encapsulates its findings, reached using panel data on EU member states from 2006-2021. 2/x

www.nature.com/articles/s41562-023-01695-6

My Matters Arising concerning a paper on the legal determinants of terrorism is now out in @nathumbehav.nature.com. The original paper is now retracted. To learn why, read on for a story of irregularities, imputations, and impossible values. 1/x

doi.org/10.1038/s41562-025-02347-7

Pleased to announce that my paper on hypothetical bias in experiments is now published in Experimental Economics! If you're interested in (how we figure out) whether real stakes and incentives matter in experiments, give it a read!

🔗: doi.org/10.1017/eec....

Had a great time presenting my job market paper at the Lindau Nobel Meeting in Economic Sciences! 🔗 : osf.io/d7sqr_v1/

#LINOecon #EconSky