Tim Hannigan

Epidemiology models explain rumour spreading during France’s Great Fear of 1789 - Nature Epidemiological methods are used to show that the Great Fear of 1789, a series of peasant insurrections in rural revolutionary France, was driven by deliberate political action rather than spontaneous...

This paper is causing a stir. My hot takes:

1. In my ideal world, we would be building boatloads of datasets and models like this, leading to a metastudy to synthesize findings.
2. Their use of GPT to structure text is, in fact, a great use of LLMs for research.

www.nature.com/articles/s41...

Screenshot of first page of paper. It is here: https://arxiv.org/pdf/2507.00828 Abstract: Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations

Evaluating topic models (and document clustering methods) is hard. In fact, since our paper critiquing standard evaluation practices four years ago, there hasn't been a good replacement metric

That ends today (we hope)! Our new ACL paper introduces an LLM-based evaluation protocol 🧵

The "Open Academic"

A process model and a set of approaches for social-media-enabled academic openness.

doi.org/10.1016/j.bu...

As in, the over-claiming about AGI is an omnibus technology frame that is based in hype discourses. This is different from the careful use case-affordances-practice bundle that may emerge

Isn’t this a key aspect in the emergence of a new technology? Working out the usecases and affordances, then connecting them with categories of practices? Seen in this light, hype begins as limitless possibilities

Congratulations, this looks like a fantastic paper!

Integrating Generative Artificial Intelligence into Social Science Research: Measurement, Prompting, and Simulation - Thomas Davidson, Daniel Karell, 2025 Generative artificial intelligence (AI) offers new capabilities for analyzing data, creating synthetic media, and simulating realistic social interactions. This...

Attn: the special issue of SMR on Generative AI in Sociology is now online. The whole issue is fire, but the introduction by @thomasdavidson.bsky.social and Danny Karell is a must read, covering prompting, measurement, and simulations. It's a road map for soc sci research in the genAI era. +

Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models Nga Than, Leanne Fan, Tina Law, Laura K. Nelson, and Leslie McCall. Abstract Over the past decade, social scientists have adapted computational methods for qualitative text analysis, with the hope that they can match the accuracy and reliability of hand coding. The emergence of GPT and open-source generative large language models (LLMs) has transformed this process by shifting from programming to engaging with models using natural language, potentially mimicking the in-depth, inductive, and/or iterative process of qualitative analysis. We test the ability of generative LLMs to replicate and augment traditional qualitative coding, experimenting with multiple prompt structures across four closed- and open-source generative LLMs and proposing a workflow for conducting qualitative coding with generative LLMs. We find that LLMs can perform nearly as well as prior supervised machine learning models in accurately matching hand-coding output. Moreover, using generative LLMs as a natural language interlocutor closely replicates traditional qualitative methods, indicating their potential to transform the qualitative research process, despite ongoing challenges.

Figure 1: Researcher-LLM-Researcher workflow. Circular diagram, starting at the top with Box 1: Design Coding Task (Researcher), moving clockwise to Box 2: Initial Text Classifiation (LLMs), going through high agreement to Validation (Researcher) and low agreement to Examine Ambiguous Cases (Researcher) in Box 3, then Box 4: Iterative Coding or Task Refinement (Researcher). Circling back to Box 1.

Our article, with Nga Than, Leanne Fan, @tinalaw.bsky.social and Leslie McCall @stone-lis.bsky.social, proposes a framework (image 2) to (carefully) incorporate LLMs into the qualitative coding process.

But read the entire issue, if it's your jam.

journals.sagepub.com/doi/full/10....

Congrats Carl, this course is a fantastic resource!

Great piece! One of the reasons that botshit can be so insidious is that it accelerates epistemic erosion in organizations and institutions

Generative AI models are skilled in the art of bullshit Large language models are unconcerned with truth because they have no concept of it — and therein lies the danger

The philosopher Harry Frankfurt argued that bullshit was "a greater enemy of the truth than lies are".

But is generative AI-generated botshit even worse than human-generated bullshit? (Yes)

My latest @financialtimes.com here:

on.ft.com/3SikRPK

The Risks of Botshit Botshit — made-up, inaccurate, and untruthful chatbot content that humans uncritically use for tasks — can pose major risks to your business in the form of reputational damage, incorrect decisions, le...

We’ve been tracking this phenomenon as “botshit” - it is a form of epistemic erosion in organizations. It’s increasingly common in practice. Speaks to a lack of oversight with gen AI tools (which can be helpful if used with care)
hbr.org/2024/07/the-...

Also,

www.sciencedirect.com/science/arti...

Good to hear- this is one of the major pitfalls of using an otherwise fantastic language and programming environment. I’ve wasted hours on Python package management

Neat! The cycle of epidemic erosion is an important hazard to try to manage with programs. You might find this article that we wrote last year to be in line with your thinking: www.sciencedirect.com/science/arti...

Also, image courtesy of @toffeeman68.bsky.social

Thanks @jc-marques.bsky.social , much appreciated

Best Article Award: Business Horizons 2024, with Tim Hannigan and Andre Spicer

Here is the research article: doi.org/10.1016/j.bu...

And here is an open-access pre-print version of the paper: dx.doi.org/10.2139/ssrn...

#botshit

Ha! I love it

Eurovision is a ride many Canadians would struggle to comprehend!

What would be really good now would be if computer/data science departments established programs in reading and interpreting text. There could be theories of text, and examples of texts from other time periods and other cultures. Above all, students would learn how to look at texts very closely.

Donald Trump has thrown a massive obstacle onto Pierre Poilievre's path to power — Toronto Star The mercurial American president has given Canada's Conservative leader a lesson in what a difference a week can make in politics, Althia Raj writes.

What a week in Canadian politics!
If you haven’t seen it yet, Althia Raj wrote a great column in this in the Toronto Star apple.news/AeG0-BNg4TjK...

I'm delighted to join the editorial team and begin my role as senior editor at Organization Studies (@orgstudies.bsky.social). I will be focusing on manuscripts surrounding fields, technology & organizing, interpretive applications of computational methods, and organizational wrongdoing.

This is wonderful, thanks for sharing!

Strange times we live in

Thanks Maxim!

Thanks! I'm glad our core argument resonates

I'm pleased to share our new article with Gorgi Krlev, Tim Hannigan (myself) and Andre Spicer in the Academy of Management Annals: What Makes a Good Review Article? Empirical Evidence From Management and Organization Research. journals.aom.org/doi/abs/10.5...

Michael I. Jordan's 2019 essay on this is very good: rethinking AI as social infrastructure rather than something whose telos is necessarily to imitate individual human agents.

The Risks of Botshit Botshit — made-up, inaccurate, and untruthful chatbot content that humans uncritically use for tasks — can pose major risks to your business in the form of reputational damage, incorrect decisions, le...

Great piece! You might like these recent articles with similar arguments. We situate on the practice end, suggesting that LLM output should ontologically be seen as “provisional knowledge”. Without attending to epistemic risks, it can become botshit
hbr.org/2024/07/the-...

doi.org/10.1016/j.bu...

The Risks of Botshit Botshit — made-up, inaccurate, and untruthful chatbot content that humans uncritically use for tasks — can pose major risks to your business in the form of reputational damage, incorrect decisions, le...

Great point! The circumstances of use in organizations matters. We developed a similar argument earlier this year in our HBR piece: hbr.org/2024/07/the-...

Neat tool showing themes of my work (using scholargoggler.com, via @drmaximvoronov.bsky.social)