Simon Schrodi

Big thanks to our amazing co-authors: Max Argus, Volker Fischer, and @thomasbrox.bsky.social 🙌

Even better, if you're at #ICLR2025 next week:
🖼️ Poster — April 24, 10 a.m. - 12:30 p.m., Hall 3 + Hall 2B (#481)
🎤 Oral — April 24, 4:30 p.m. - 4:42 p.m, Garnet 213–215 (oral session 2B)
☕ Or just catch us over coffee!

Two Effects, One Trigger: On the Modality Gap, Object Bias, and... Contrastive vision-language models (VLMs), like CLIP, have gained popularity for their versatile applicability to various downstream tasks. Despite their successes in some tasks, like zero-shot...

Curious to dive deeper?
📑 Paper: openreview.net/forum?id=uAF...
💻 Code: github.com/lmb-freiburg...
📬 DM me or David (he's not on Bluesky, but you can dm him on other platforms)!

But what is the modality gap good for? Interestingly, we find it affects the model’s entropy suggesting it might not be a bug, but a feature. 👀

Our paper is packed with surprising and insightful findings about both phenomena. Most notably, we show that both effects stem from the information imbalance between image and text modalities and both can be reduced when that imbalance decreases.

In this work, we investigate two undesired properties of CLIP-like models:
- Modality gap: a complete separation of image and text embeddings in the shared embedding space.
- Object bias: a tendency to focus on objects over other semantic aspects like attributes.

David Hoffmann and I will present our joint work, “Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models,” this Thursday as an Oral (top 1.8%) at #ICLR2025!🎉

🧵

Hello world!

Hi Julian, could you please add me? I work on interpretability & data-centric ML for multi-modal models

A starter pack of people working on interpretability / explainability of all kinds, using theoretical and/or empirical approaches.

Reply or DM if you want to be added, and help me reach others!

go.bsky.app/DZv6TSS