Cohere Labs

“When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs”

Led by: Ammar Khairi, Daniel D'souza, Ye Shen, Julia Kreutzer, Sara Hooker

📜Paper link: arxiv.org/abs/2506.20544

🥢 We then use our new CHOPS method, which selects the best sample in a single call and outperforms Best-of-N with strong reward models while being more efficient.

⚖️ We also introduce X-MBR which leverages crosslingual capabilities for a +12% in winrates from only 5 samples!

🦔We first introduce Hedged Sampling, where we mix deterministic and stochastic decoding methods.

This intervention yields a 7.6% win-rate delta against a single-sample baseline, a +2.6% jump from using a single temperature 📈, with particular benefits for multilingual tasks.

🚀 Scaling inference compute boosts LLM performance, but generating hundreds of samples is expensive.

🍋 Our LLMonade Recipe eliminates these obstacles by "squeezing" maximum value from fewer samples using generalist LLMs-as-judge for cross-lingual, cross-task selection.

Can we improve the performance of LLMs during inference without the need for extensive sampling OR special reward models? 🤔

Our latest work introduces a new inference time scaling recipe that is sample-efficient, multilingual, and suitable for multi-task requirements. 🍋

Can we improve the performance of LLMs during inference without the need for extensive sampling OR special reward models? 🤔

Our latest work introduces a new inference time scaling recipe that is sample-efficient, multilingual, and suitable for multi-task requirements. 🍋

📃: arxiv.org/abs/2506.20544

Led by: Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen H. Bach, Julia Kreutzer

📜 For more details, read the paper: arxiv.org/abs/2505.24119

We also outline future research directions with concrete steps, from evaluation practices over synthetic data generation to cross-lingual generalization.

For instance, we should move beyond reporting average safety scores to improve multilingual safety evaluation.

We found that English-centricity persists across different safety topics.

More importantly, nearly 50% of the papers did not report that they are English only! 😱

😳 There are *ten times* more papers that include English than Chinese, which is the 2nd most-studied language.

Furthermore, non-English languages are usually studied in herds, which limits the possibility for language-specific safety analysis.

We analyzed nearly 300 papers from various *ACL venues.

🚨 We observe a considerable gap between English and non-English safety research.

It’s been two years since cross-lingual jailbreaks were first discovered. How far has the multilingual LLM safety research field advanced? 🤔

📏 Our comprehensive survey reveals that there is still a long way to go.

The Multilingual Divide and Its Impact on Global AI Safety Despite advances in large language model capabilities in recent years, a large gap remains in their capabilities and safety performance for many languages beyond a relatively small handful of globally...

📜 Read the paper: arxiv.org/abs/2505.21344

The Multilingual Divide and Its Impact on Global AI Safety Despite advances in large language model capabilities in recent years, a large gap remains in their capabilities and safety performance for many languages beyond a relatively small handful of globally...

Thanks to all collaborators: Aidan Peppin, Julia Kreutzer, Alice Schoenauer Sebag, Kelly Marchisio, Beyza Ermis, John Dang, Samuel Cahyawijaya, Shivalika Singh, Seraphina Goldfarb-Tarrant, Viraat Aryabumi, Aakanksha, Wei-Yin Ko, Ahmet Üstün, Matthias Gallé, Marzieh Fadaee, Sara Hooker

Here are key recommendations to make AI safer & more equitable for everyone:

🌐 Incentivize the creation of open-access multilingual datasets
🪟 Encourage transparency in model language coverage
🔬 Prioritise resources towards multilingual research

But more work is needed. Policymakers and governance experts play a crucial role in bridging the AI language gap. 🧑‍⚖️ Our paper highlights the need for policy interventions to support multilingual dataset creation, transparency, and research, ensuring AI safety and equity for all.

Our research efforts over the years have worked to address these issues, spanning technical approaches to improving language performance, and building and releasing open datasets covering 101 languages.

Why does the AI Language Gap exist? It’s a vicious cycle of resource biases, global inequities, and limited access to tools. Low-resource languages get left behind, widening the divide. 🫸 🫷

Most AI models are optimized for English and a few high-resource languages, leaving many global communities marginalized. This isn’t just a tech issue—it’s a barrier to global AI safety and cultural representation. 🌍💬⚠️

Over 7000 languages are spoken worldwide 🌐, but AI safety efforts focus on only a fraction of them.

Our latest paper draws on our multi-year efforts with the wider research community to explore why this matters and how we can bridge the AI language gap.

Led by: @juliakreutzer.bsky.social, Eleftheria Briakou, @swetaagrawal.bsky.social, @mziizm.bsky.social, @kocmitom.bsky.social

Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking com...

✅We've distilled our findings into a checklist of actionable recommendations for mLLM research and development. Let's work together to improve mLLM evaluation and unlock their full potential! 💪

📜Paper link: arxiv.org/abs/2504.11829

🧪Through targeted experiments, we demonstrate how MT evaluation techniques can be adapted for mLLMs. We also identify essential components for robust meta-evaluation, ensuring the evaluation methods themselves are rigorously assessed.

🕵️Current mLLM evaluation lacks comprehensiveness, scientific rigor, and consistent adoption. This hinders our ability to truly understand model capabilities and guide development.

🚀🌍The rapid advancement of multilingual large language models (mLLMs) is exciting, but are we evaluating them effectively?

Our new paper explores how we can improve generative evaluations for mLLMs by learning from machine translation (MT) evaluation practices. 🔎

Kaleidoscope: the largest culturally-authentic exam benchmark for VLMs.

Most benchmarks are English-centric or rely on translations, missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLM evaluation.

arxiv.org/abs/2504.07072

...MohammadAmin farahani fard, Silvia Fernandez, María Grandury, Dmitry Abulkhanov, Drishti Sharma, Andre Guarnier De Mitri, Leticia Bossatto Marchezi, Johan Obando-Ceron, Nazar Kohut, Beyza Ermis, Desmond Elliott, Enzo Ferrante, Sara Hooker, Marzieh Fadaee

...Sharad Duwal, Alfonso Amayuelas, Swati Rajwal, Jebish Purbey, Ahmed Ruby, Nicholas Popovič, Marek Suppa, Azmine Toushik Wasi, Ram Mohan Rao Kadiyala, Olga Tsymboi, Maksim Kostritsya, Bardia Soltani Moakhar, Gabriel da Costa Merlin, Otávio Ferracioli Coletti, Maral Jabbari Shiviari,

Led by: Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, Dominik Krzemiński, Jekaterina Novikova, Luísa Shimabucoro, Joseph Marvin Imperial, Rishabh Maheshwary,

📜 Paper: arxiv.org/abs/2504.07072
🌐 Explore the benchmark: cohere.com/research/kal...
📂 Dataset: hf.co/datasets/Coh...