Jiaang Li

Feel free to reach out and chat with Xinyi on July 18th in Vancouver at the #ICML

NeurIPS participation in Europe We seek to understand if there is interest in being able to attend NeurIPS in Europe, i.e. without travelling to San Diego, US. In the following, assume that it is possible to present accepted papers ...

Would you present your next NeurIPS paper in Europe instead of traveling to San Diego (US) if this was an option? Søren Hauberg (DTU) and I would love to hear the answer through this poll: (1/6)

Check out our new preprint 𝐓𝐞𝐧𝐬𝐨𝐫𝐆𝐑𝐚𝐃.
We use a robust decomposition of the gradient tensors into low-rank + sparse parts to reduce optimizer memory for Neural Operators by up to 𝟕𝟓%, while matching the performance of Adam, even on turbulent Navier–Stokes (Re 10e5).

PhD student, Jiaang Li and his collaborators, with insights into cultural understanding of vision-language models 👇

Paper title "Cultural Evaluations of Vision-Language Models Have a Lot to Learn from Cultural Theory"

I am excited to announce our latest work 🎉 "Cultural Evaluations of Vision-Language Models Have a Lot to Learn from Cultural Theory". We review recent works on culture in VLMs and argue for deeper grounding in cultural theory to enable more inclusive evaluations.

Paper 🔗: arxiv.org/pdf/2505.22793

Great collaboration with @yfyuan01.bsky.social @wenyan62.bsky.social @aliannejadi.bsky.social @danielhers.bsky.social , Anders Søgaard, Ivan Vulić, Wenxuan Zhang, Paul Liang, Yang Deng, @serge.belongie.com

jaagli/ravenea · Datasets at Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science.

🔗More here:
Project Page: jiaangli.github.io/RAVENEA/
Code: github.com/yfyuan01/RAV...
Dataset: huggingface.co/datasets/jaa...

📊Our experiments demonstrate that even lightweight VLMs, when augmented with culturally relevant retrievals, outperform their non-augmented counterparts and even surpass the next larger model tier, achieving at least a 3.2% improvement in cVQA and 6.2% in cIC.

🛠Culture-Aware Contrastive Learning

We propose Culture-aware Contrastive (CAC) Learning, a supervised learning framework compatible with both CLIP and SigLIP architectures. Fine-tuning with CAC can help models better capture culturally significant content.

📚 Dataset Construction
RAVENEA integrates 1,800+ images, 2,000+ culture-related questions, 500+ human captions, and 10,000+ human-ranked Wikipedia documents to support two key tasks:

🎯Culture-focused Visual Question Answering (cVQA)
📝Culture-informed Image Captioning (cIC)

🚀New Preprint🚀
Can Multimodal Retrieval Enhance Cultural Awareness in Vision-Language Models?

Excited to introduce RAVENEA, a new benchmark aimed at evaluating cultural understanding in VLMs through RAG.
arxiv.org/abs/2505.14462

More details:👇

Super cool! Incidentally, in our previous project, we also found that linear alignment between embedding spaces from two modalities is viable — and the alignment improves as LLMs scale.
bsky.app/profile/jiaa...

Revisiting the Othello World Model Hypothesis Li et al. (2023) used the Othello board game as a test case for the ability of GPT-2 to induce world models, and were followed up by Nanda et al. (2023b). We briefly discuss the original experiments, ...

I won’t be attending #ICLR in person this year😢. But feel free to check our paper ‘Revisiting the Othello World Model Hypothesis’ with Anders Søgaard, accepted at ICLR world models workshop!
Paper link arxiv.org/abs/2503.04421

Thrilled to announce "Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation" is accepted as a Spotlight (5%) at #ICLR2025!

Our model MM-FSS leverages 3D, 2D, & text modalities for robust few-shot 3D segmentation—all without extra labeling cost. 🤩

arxiv.org/pdf/2410.22489

More details👇

Forget just thinking in words.

🔔Our New Preprint:
🚀 New Era of Multimodal Reasoning🚨
🔍 Imagine While Reasoning in Space with MVoT

Multimodal Visualization-of-Thought (MVoT) revolutionizes reasoning by generating visual "thoughts" that transform how AI thinks, reasons, and explains itself.

FGVC12 Workshop is coming to #CVPR 2025 in Nashville!

Are you working on fine-grained visual problems?
This year we have two peer-reviewed paper tracks:
i) 8-page CVPR Workshop proceedings
ii) 4-page non-archival extended abstracts
CALL FOR PAPERS: sites.google.com/view/fgvc12/...

YouTube video by Videnskabernes Selskab VidenSkaber | Min AI forstår mig ikke - professor Serge Belongie

Here’s a short film produced by the Danish Royal Academy of Sciences, showcasing the WineSensed 🍷 project of Þóranna Bender et al. thoranna.github.io/learning_to_...

From San Diego to New York to Copenhagen, wishing you Happy Holidays!🎄

With @neuripsconf.bsky.social right around the corner, we’re excited to be presenting our work soon! Here’s an overview

(1/5)

Belongie Lab Join the conversation

Here’s a starter pack with members of our lab that have joined Bluesky

a panda bear is rolling around in the grass in a zoo enclosure . Alt: a panda bear is rolling around in the grass in a zoo enclosure .

No one can explain stochastic gradient descent better than this panda.

🙋‍♂️

GitHub - jiaangli/VLCA: Do Vision and Language Models Share Concepts? A Vector Space Alignment Study Do Vision and Language Models Share Concepts? A Vector Space Alignment Study - jiaangli/VLCA

Great collaboration with @constanzafierro.bsky.social , @YovaKem_v2, and Anders Søgaard!

👨‍💻 github.com/jiaangli/VLCA
📃 direct.mit.edu/tacl/article...

🚀Take away:

1. Representation spaces of LMs and VMs grow more partially similar with model size.
2. Lower frequency, polysemy, dispersion can be easier to align.
3. Shared concepts between LMs and VMs might extend beyond nouns.

🧵(7/8)
#NLP #NLProc

🌱We then discuss the implications of our finding:
- the LM understanding debate
- the study of emergent properties
- philosophy

🧵(6/8)

🔍We also measure the generalization of the mapping to other POS, and explore the impact of different size of the training data. 👀To investigate the effects of incorporating text signals during vision pretraining, we compare pure vision models against selected CLIP vision encoders.

🧵(5/8)

What factors influence the convergence?
🔍Our experiments show the alignability of LMs
and vision models is sensitive to image and language dispersion, polysemy, and frequency.

🧵(4/8)

The results show a clear trend:
✨LMs converge toward the geometry of visual models as they grow bigger and better.

🧵(3/8)

Mapping vector spaces:
🎯We measure the alignment between vision models and LMs by mapping their vector spaces and evaluating retrieval precision on held-out data.

🧵(2/8)

🤔Do Vision and Language Models Share Concepts? 🚀
We present an empirical evaluation and find that language models partially converge towards representations isomorphic to those of vision models. #EMNLP

📃 direct.mit.edu/tacl/article...