Andrea Piergentili

@bsavoldi.bsky.social presenting our new multilingual benchmark for evaluating LLMs on gender-neutral translation.

Catch our paper at #EMNLP2025
ℹ️ arxiv.org/pdf/2501.09409

#lt2025fbk

LT Highlights @ FBK 2025

🚀 Join us for the LT@FBK day 2025! Discover cutting-edge research and highlights in speech and language technologies from Fondazione Bruno Kessler (FBK)

📅 October 28, 2025
📍FBK, Trento
ℹ️ lt-highlights.fbk.eu

Last but definitely not least: @bsavoldi.bsky.social presenting joint work with @apierg.bsky.social @matteo-negri.bsky.social @luisabentivogli.bsky.social on scalable gender neutral translation evaluation using LLM-as-a-judge at #GITT2025

Agree to Disagree? A Meta-Evaluation of LLM Misgendering Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g., automatically with templatic sentences) and generation-based evaluations (e.g., with automatic heuristics or human validation). However, it has gone unexamined whether these evaluation methods have convergent validity, that is, whether their results align. Therefore, we conduct a systematic meta-evaluation of these methods across three existing datasets for LLM misgendering. We propose a method to transform each dataset to enable parallel probability- and generation-based evaluation. Then, by automatically evaluating a suite of 6 models from 3 families, we find that these methods can disagree with each other at the instance, dataset, and model levels, conflicting on 20.2% of evaluation instances. Finally, with a human evaluation of 2400 LLM generations, we show that misgendering behaviour is complex and goes far beyond pronouns, which automatic evaluations are not currently designed to capture, suggesting essential disagreement with human evaluations. Based on our findings, we provide recommendations for future evaluations of LLM misgendering. Our results are also more widely relevant, as they call into question broader methodological conventions in LLM evaluation, which often assume that different evaluation methods agree.

Super interesting paper by Subramonian et al: "Agree to Disagree? A Meta-Evaluation of LLM Misgendering" arxiv.org/abs/2504.17075
Turns out, misgendering is messier than just pronouns. I'd love to see this analysis extended to grammatical gender languages! #LLM #AI #ethics @fbk-mt.bsky.social

🔍 Stiamo studiando come l'AI viene usata in Italia e per farlo abbiamo costruito un sondaggio!

👉 bit.ly/sondaggio_ai...

(è anonimo, richiede ~10 minuti, e se partecipi o lo fai girare ci aiuti un sacco🙏)

Ci interessa anche raggiungere persone che non si occupano e non sono esperte di AI!

FAMA - a FBK-MT Collection The First Large-Scale Open-Science Speech Foundation Model for English and Italian

🚀 New tech report out! Meet FAMA, our open-science speech foundation model family for both ASR and ST in 🇬🇧 English and 🇮🇹 Italian.

The models are live and ready to try on @hf.co:
🔗 huggingface.co/collections/...

📄 Preprint: arxiv.org/abs/2505.22759

#ASR #ST #OpenScience #MultilingualAI

Will do 🫡

a woman is standing in front of a bookshelf in a bookstore and talking about research . ALT: a woman is standing in front of a bookshelf in a bookstore and talking about research .

👀 Wanted: #Italian or #Dutch native speakers to take a survey on audiovisual translation for a master thesis student: watch a short video, answer some questions, help academic research 😎
⏩ Sharing = nice! ❤️
NL link: ugent.qualtrics.com/jfe/form/SV_...
IT link: ugent.qualtrics.com/jfe/form/SV_...

a man in a suit is making a funny face with the words dreams are expensive behind him ALT: a man in a suit is making a funny face with the words dreams are expensive behind him

💭Dreaming of attending #GITT2025 but need a little extra 💸 boost?
📣 Bursary applications to support participation are now open at tinyurl.com/gitt25
📆 Deadline May 9th
🙏Thanks to our incredible sponsors DCA at Tilburg University tinyurl.com/tudca25 and FLW at Ghent University www.ugent.be/lw/en

📢 Come and join our group!
We offer a fully funded 3-year PhD position:

📔 Automatic translation with large multimodal models: iecs.unitn.it/education/ad...

📍Full details for application: iecs.unitn.it/education/ad...

📅 Deadline May 12, 2025

#NLProc #FBK

An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is pa...

Happy to announce that our paper 'An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation' was accepted at @gitt-workshop.bsky.social ! 🙌

Check it out: arxiv.org/abs/2504.11934 🔥

Co-authors (🫶🏻): @bsavoldi.bsky.social, @matteo-negri.bsky.social, @luisabentivogli.bsky.social

Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally "gaming the metric" during model development rises. This issue is caused by metric interference (Mint)...

Brilliant and necessary work by Pombal et al. about metric interference in MT system development and evaluation: arxiv.org/abs/2503.08327

Are we developing better systems or are we just gaming the metrics? And how do we address this?
Super (m)interesting! 👀

While we look forward to a sunny Geneva, why wait to join the conversation?

We’ve created a starter pack for our #GITT2025 friends!
🕵️ Follow researchers working on gender bias in MT
💬 Stay up to date and dive into the discussion!

All info at sites.google.com/tilburgunive...

BREAKING NEWS: CDC orders mass retraction and revision of submitted research across all science and medicine journals. Banned terms must be scrubbed. Any unpublished manuscript mentioning certain topics, including gender and "LGBT," must be pulled or revised.

BREAKING NEWS: CDC orders mass retraction and revision of submitted research across all science and medicine journals. Banned terms must be scrubbed.

Goes beyond MMWR +other CDC pubs. Applies to research already submitted to top medical journals.

Take a look.
open.substack.com/pub/insideme...

🙌 All members of our group are now on Bluesky! 🙌

You can find all of us in this starter pack 👇

Looking ahead to 2025, my goal is to keep the momentum and build on this year’s lessons: being more intentional about time management, becoming a better collaborator, and and carving out time for deep, focused work.

The rest of the year was spent on testing new things, new collaborations, reading and reviewing papers, and traveling around for conferences. No doubt this has been the year where I learned the most so far, and 99% of the learning happened because I had access to some amazing (and patient) people.

I also developed a demo showcasing gender-neutral translation with LLMs, which I had the chance to present at FBK’s Digital Industry Center Demo Day. Unfortunately the demo is not open to the public for now, but here is a photo of @bsavoldi.bsky.social and me presenting it ✌️

Two key resources enabled the research progress we made this year: GeNTE (2023) and Neo-GATE (2024). They are benchmarks for the conservative and the innovative approach respectively and are both freely available on Hugging Face:

huggingface.co/datasets/FBK...
huggingface.co/datasets/FBK...

My research topic is gender-inclusive MT, and this year we explored two directions: the "conservative" one with gender-neutral translation and the "innovative" one, using neomorphemes (like ə and *, in Italian). I worked on papers published at venues ranging from top conferences to local workshops.

but look i made you some is written in white on a black background ALT: but look i made you some is written in white on a black background

With 2024 wrapping up, and given how little I’ve posted here (or anywhere, really), I thought I’d share a quick recap of my year and finally make some ✨content✨

Our @apierg.bsky.social presenting our #calamita challenges at #CLiCit2024: machine translation and gender-fair generation.

Poster session upcoming, see you there!

For more details:
👉 MagneT: clic2024.ilc.cnr.it/wp-content/u...
👉 GFG: clic2024.ilc.cnr.it/wp-content/u...

Our very own @dennisfucci.bsky.social presenting the challenges of Explainability for Speech Models at #CLiCit2024. If you’re interested, check out the paper 👉 clic2024.ilc.cnr.it/wp-content/u...
#NLProc

Today @luisabentivogli.bsky.social, Dennis Fucci, and @apierg.bsky.social presented a research communication about gender-neutral translation in the morning poster session #CLiCit2024 #NLProc

If you are in Pisa at #CLiCit2024 don't miss the presentation of our last work today at 12 🔥

I’ve created an Italian #NLProc Researcher Starter Pack 🇮🇹

DM me to join if you're not in yet!

go.bsky.app/LHbWLHp

You're in ✌️😎

Hey, if you're in Pisa and interested to connect with other people at #CLiCit2024 check out this starter pack. Raise a hand 👋 and I will add you to this list!

We are happy to announce that our PhD students Dennis Fucci and @apierg.bsky.social along with our head of unit @luisabentivogli.bsky.social, will attend #CLiCit2024 in Pisa! Meet them at the poster presentations during the main conference and the #CALAMITA event!