Michele Papucci

Findings of ACL 2025 Poster Session Today I presented my latest work, toghether with my collegues, at the Findings Poster Session of ACL 2025

Last week I presented my latest work at the Findings Poster Session of ACL 2025 in Vienna!

If you missed it, check it out 💥

michelepapucci.github.io/blog/finding...

Paper Accepted at Findings of ACL 2025 My latest paper "Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors" has been accepted at the Findings of ACL 2025.

michelepapucci.github.io/blog/paper-a...

New blog post about our latest paper, accepted at Findings of ACL.

📣 #clicit2025 paper submission deadline extension: 16/06/2025! 📣

8/ TL:DR;
🚨 State-of-the-art Detectors today are too shallow
📉 A bit of style alignment makes them crumble
🧠 We need stronger benchmarks
🛠 We develop a way to create hard, in-domain texts for making and evaluating the next generation of more robust and reliable MGT Detectors

7/ What about Humans?
Human performance was unaffected: they performed poorly in detecting machine-generated text (around 50% accuracy in a binary task) both before and after our alignment.

6/ We tested a bunch of state-of-the-art detectors:
- 🕵️ Mage
- 🎯 Radar
- 🔍 LLM-DetectAIve
- 👁 Binoculars
- Two domain-specific detectors trained by us: a Linear-SVM and a RoBERTa.
The most robust detector, for our type of attack, was Radar.

5/ We tested two ways of selecting texts for alignment, a random one and a linguistically motivated one. The latter proved better for aligning specific feature distribution of an LLM to the humans', but the former seemed to work better in dropping detector accuracy.

4/ We tested on two domains (News and Abstracts), with two families of models (Llama and Gemma). Detectors run on text generated by the aligned models dropped up to 60% in performance.

3/ Why does it work?
Most detectors rely on shallow stylistic cues—word length, punctuation patterns, and sentence structure. Aligning LLMs to human style shifts the model's writing style towards humans', and Detectors can’t keep up.

2/ We introduce a simple pipeline:
We fine-tune LLMs via Direct Preference Optimization (DPO), using human-written and machine-generated text pairs, marking the former as the preferred. The goal is to shift LLMs' writing style towards humans.

🧵1/ Machine-Generated Text (MGT) detection is failing

Our paper, accepted at Findings of ACL 2025, shows that LLMs can fool generated-text detectors.
arxiv.org/abs/2505.24523

Andrea Pedrotti, Cristiano Ciaccio, @alessiomiaschi.bsky.social @gpucce.bsky.social, Felice Dell'Orletta, Adrea Esuli

Rate Love Hotel - Point and Click Game for InkJam 2024 by heyimfishy for inkJam 2024 Point and click game for InkJam 2024. Find your true love!

itch.io/jam/inkjam-2...
Our submission for the #inkjam is now up and ready to be played and rated!

Let us know what you think of our little ugly game made in a few hours ahahah