Dirk Hovy

Table titled “Taxonomy for evaluation of AI in mental health applications,” organized into columns for quality criteria (validity and reliability) and real-world use (implementation and maintenance). Rows distinguish support types: assessment, intervention, and information synthesis. Each cell lists detailed evaluation questions, such as construct and criterion validity, consistency across populations and time, feasibility, effectiveness, usability, acceptability, safety, and unintended consequences, providing a structured framework for assessing AI systems in mental health contexts.

🔎🧩 𝗕𝗲𝘆𝗼𝗻𝗱 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀: 𝗛𝗼𝘄 𝘁𝗼 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗠𝗲𝗻𝘁𝗮𝗹 𝗛𝗲𝗮𝗹𝘁𝗵 𝗔𝗜 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗶𝗯𝗹𝘆
AI for mental health is a high-stakes area: its evaluation needs to meet the highest expectations.

The new preprint 𝘙𝘦𝘴𝘱𝘰𝘯𝘴𝘪𝘣𝘭𝘦 𝘌𝘷𝘢𝘭𝘶𝘢𝘵𝘪𝘰𝘯 𝘰𝘧 𝘈𝘐 𝘧𝘰𝘳 𝘔𝘦𝘯𝘵𝘢𝘭 𝘏𝘦𝘢𝘭𝘵𝘩, written by an interdisciplinary team spanning AI [...]

Honored to give my first keynote at #IRCDL2026 on February 19th.

I’ll talk about how LLMs have shifted from productivity tools to everyday sources of info & personal guidance and what that means for risk, trust, bias, and alignment.

ircdl2026.unimore.it

The image displays the words "Politics & Gender" in yellow text on a green background, with the hashtag "#OpenAccess" below it. A vertical yellow stripe is on the left side.

#OpenAccess from @politicsgenderj.bsky.social -

Male Agency? Analyzing Fatherhood Roles in Swedish Parliamentary Documents, 1993–2021 - https://cup.org/40el36q

- Lena Wängnerud, Elin Naurin, @dirkhovy.bsky.social, Lorenzo Lupo & Oscar Magnusson

#FirstView

#MemoryModay #NLProc
@gattanasio.cc et al. study asks 'Is It Worth the (Environmental) Cost?' analyzing continuous training for language models. Balances benefits, environmental impacts, for responsible use. #AI #Sustainability

arxiv.org/pdf/2210.07365

What are the main issues discussed in a set of documents?

We’ve just released a step-by-step BERTopic tutorial.

We also launch a new page, gathering various NLP tutorials for social scientists.
👉 www.css.cnrs.fr/tutorials-an...

Citation is the foundation of academic promotion. It’s noisy, sure, but its integrity is worth fighting for. Hallucinated citations should be a desk reject.

The second new class I'm teaching is a very experimental graduate level seminar in CSE: "Building Small Language Models". I taught the grad level NLP class last semester (so fun!) but students wanted more—which of these new ideas work, and which work for SLMs? jurgens.people.si.umich.edu/CSE598-004/

🎉 MilaNLP 2025 Wrapped 🎉
Lots of learning, building , sharing, and growing together 🌱

#NLProc

Found and added under data/

I included some test cases on GitHub, will look if I still have the ones we used in the paper.

⏳ Deadline approaching! We’re hiring 2 fully funded postdocs in #NLP.

Join the MilaNLP team and contribute to our upcoming research projects (SALMON & TOLD)

🔗 Details + how to apply: milanlproc.github.io/open_positio...

⏰ Deadline: Jan 31, 2026

If you are curious about the theoretical background, see

Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., & Hovy E. (2013). Learning Whom to Trust With MACE. In: Proceedings of NAACL-HLT. ACL.

aclanthology.org/N13-1132.pdf

And for even more details:

aclanthology.org/Q18-1040.pdf

N/N

I always wanted to revisit it, port it from Java to Python & extend to continuous data, but never found the time.
Last week, I played around with Cursor – and got it all done in ~1 hour. 🤯

If you work with any response data that needs aggregation, give it a try—and let me know what you think!

4/N

MACE estimates:
1. Annotator reliability (who’s consistent?)
2. Item difficulty (which examples spark disagreement?)
3. The most likely aggregate label (the latent “best guess”)

That “side project” ended up powering hundreds of annotation projects over the years.

3/N

However, disagreement isn’t just noise—it’s information. It can mean an item is genuinely hard—or someone wasn’t paying attention. If only you knew whom to trust…

That summer, Taylor Berg-Kirkpatrick, Ashish Vaswani, and I built MACE (Multi-Annotator Competence Estimation).

2/N

🚨(Software) Update:

In my PhD, I had a side project to fix an annoying problem: when you ask 5 people to label the same thing, you often get different answers. But in ML (and lots of other analyses), you still need a single aggregated answer. Using the majority vote is easy–but often wrong.

1/N

New year, new job? If that is your current mantra, check the open postdoc positions with Debora Nozza and me at our lab. Deadline is January 31st.

milanlproc.github.io/open_positio...

🚀 We’re opening 2 fully funded postdoc positions in #NLP!

Join the MilaNLP team and contribute to our upcoming research projects.

🔗 More details: milanlproc.github.io/open_positio...

⏰ Deadline: Jan 31, 2026

Happy to have contributed to this

Countering Hateful and Offensive Speech Online - Open Challenges Flor Miriam Plaza-del-Arco, Debora Nozza, Marco Guerini, Jeffrey Sorensen, Marcos Zampieri. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts.…

#MemoryModay #NLProc Countering Hateful and Offensive Speech Online - Open Challenges" by Plaza-Del-Arco, @debora_nozza, Guerini, Sorensen, Zampieri, 2024 is a tutorial on the challenges and solutions for detecting and mitigating hate speech.

#MemoryModay #NLProc Uma, A. N. et al. examine AI model training in 'Learning from Disagreement: A Survey'. Disagreement-handling methods' performance is shaped by evaluation methods & dataset traits.

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large... Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will follow even the most...

#TBT #NLProc #MachineLearning #SafetyFirst 'Safety-Tuned LLaMAs: Improving LLMs Safety' by Bianchi et al. explores training LLMs for safe refusals, warns of over-tuning.

Come work with @deboranozza.bsky.social, me, and the lab in Milan!

We don't actually trust AI.
We trust the companies behind it.

As Maria Antoniak notes, every "private" chat flows through corporate systems with long histories of data misuse. If we care about AI ethics, we need to name power, not anthropomorphize models.

Research Intern - Computational Social Science | Microsoft Careers Research Interns put inquiry and theory into practice. Alongside fellow doctoral candidates and some of the world's best researchers, Research Interns learn, collaborate, and network for life. Researc...

We're hiring interns in the Computational Social Science group at Microsoft Research NYC!

If you're interested in designing AI‑based systems and understanding their impact at both individual and societal scales, apply here by Jan 9, 2026: apply.careers.microsoft.com/careers/job/...

After I shared “How to professor” last year, some people asked for a similar post on writing. Now I finally got around to typing up our lab's writing workshop slides.
It covers basic advice for research papers and grant applications.
Curious? Read it here: dirkhovy.com/post/2025_11...

Respectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech Flor Miriam Plaza-del-arco, Debora Nozza, Dirk Hovy. The 7th Workshop on Online Abuse and Harms (WOAH). 2023.

#TBT #NLProc 'Respectful or Toxic?' by Plaza-del-Arco, @debora & @dirkhovy.bsky.social (2023) explores zero-shot learning for multilingual hate speech detection. Highlights prompt & model choice for accuracy. #AI #LanguageModels #HateSpeechDetection

#MemoryModay #NLProc 'Leveraging Social Interactions to Detect Misinformation on Social Media' by Fornaciari et al. (2023) uses combined text and network analysis to spot unreliable threads.

The Center for Information Technology Policy at Princeton invites applications for a Postdoctoral Fellow to work with Andy Guess (Politics/SPIA), Brandon Stewart (Sociology), and me (CS).

puwebp.princeton.edu/AcadHire/app...

Please apply before Sunday, the 13th of December!

Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, Bertie Vidgen. Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). 2022.

#MemoryModay #NLProc 'Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models' by @paul-rottger.bsky.social et al. (2022). A suite of tests for 10 languages.