Ai2

our paper on data mixing for LMs is out!

while building Olmo 3, we saw gaps between data mixing literature and real practice

🐠choosing proxy size, # runs, sampling, regression, constraints..
🐟data shifts during LM dev: can we reuse past experiments?

Olmix tackles them all!

Olmix: A framework for data mixing throughout LM development | Ai2 Olmix is a framework for language model data mixing that provides empirically grounded defaults and efficient reuse techniques.

Olmix is for anyone training LMs on diverse data who's tired of guessing at configuration choices—or re-running expensive experiments every time something changes.
✍️ Blog: buff.ly/3ZQ2w6M
💻 Code: buff.ly/9Lhh1uV
📚 Technical report: buff.ly/AHL9ewm

Across 5 domain updates and 64 domains, Olmix produces a mix that's 12% better on our downstream evaluation task suite and 3× more data-efficient than no mixing at all, while being 74% cheaper than recomputing from scratch after each update.

Mixture reuse is our mechanism where we reuse the relative ratios of the unchanged domains and only recompute on the domains that change. Fewer domains to recompute → lower costs as domains are updated throughout development.

A few of our findings:
◉ Mixing costs scale linearly with the # of domains
◉ Log-linear regression predicts mixture performance accurately
◉ Enforcing data repetition constraints can help maintain mix quality in data-constrained settings

OlmixBase emerged from studying a common mixing method pattern where you ↓
1️⃣ Train a bunch of smaller proxy models on different mixes
2️⃣ Fit a regression model to the mixture weights & performance
3️⃣ Return the mix that optimizes predicted performance

Olmix has two components:
✅ OlmixBase: a data mixing method informed by a comprehensive empirical study of key design choices
✅ Mixture reuse: a mechanism for efficiently adjusting your mix as data domains are updated by using info from the previous mix—without starting over

We built Olmix to address two pain points we hit while training Olmo 3:
1️⃣ There's little guidance on how to configure mixing methods—you're left guessing at key choices
2️⃣ Training data evolves constantly, and recomputing mixes from scratch becomes a tax on every update

Data mixing – determining how much web text, code, math, etc., you need for LM development – is a first-order lever on model quality. Introducing Olmix: a framework for configuring mixing methods at the start of dev & efficiently updating as data changes throughout. 🧵

Try AutoDiscovery in AstaLabs today → buff.ly/UaBQLur

We're giving early users 1,000 free Hypothesis Credits to get started.

📚 Learn more in our blog: buff.ly/fHLhXPs

AutoDiscovery is an AI-powered tool that explores structured datasets autonomously—generating hypotheses, designing and running statistical experiments, and surfacing findings that researchers might… Using Asta AutoDiscovery: AI-powered autonomous scientific discovery

Everything AutoDiscovery generates is transparent & reproducible—every hypothesis, statistical analysis, and line of Python code is there for you to inspect.

Science has always had more data than time to explore it. AutoDiscovery helps surface the questions hiding in yours: buff.ly/yGxdpMT

Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing Large Language Models have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing - an endeavor requiring precision, multimodal synthesis, and…

📄 In social science, AutoDiscovery helped economist Sanchaita Hazra find that doctoral-level authors made more edits to AI-generated abstracts, suggesting expertise drives critical engagement with AI.

Her independently verified results were published in a peer-reviewed paper: buff.ly/BJ5Zran

🌊 In marine ecology, Fabio Favoretto at Scripps used AutoDiscovery to explore 20+ years of rocky reef data from the Gulf of California, surfacing cross-trophic productivity relationships that would have taken extensive manual iterations to find.

Read the report: buff.ly/mhgIKD0

👩‍🔬 In oncology, Dr. Kelly Paulson at the Swedish Cancer Institute used AutoDiscovery to explore breast cancer & melanoma datasets, surfacing new hypotheses about immune responses + lymph node spread that weren't part of her team's initial questions.

Read more: buff.ly/9uJFRNL

Researchers across ecology, health, & social science are already using AutoDiscovery to surface findings hiding in their data—from cancer mutation patterns to trophic relationships in marine ecosystems.

Read their stories: buff.ly/xNbB93d

How does it decide what to pursue? Bayesian surprise—a measure of how much the system's beliefs change after seeing evidence. By chasing surprise, AutoDiscovery gravitates toward the unexpected, prioritizing results most likely to represent genuine discoveries. 🔬

Most AI tools for science wait for a research question, then help answer it. AutoDiscovery starts with your data. It generates hypotheses, runs experiments, interprets results, & uses what it learns to keep exploring.

Knowing which questions to ask is often the hardest part of science. Today we're releasing AutoDiscovery in AstaLabs, an AI system that starts with your data and generates its own hypotheses. 🧪

Explore MolmoSpaces & start building:
📝 Blog: buff.ly/nJpBn7E
💻 Demo: buff.ly/s3x8xJo
⬇️ Code: buff.ly/WrpkCxg
📊 Data: buff.ly/FMNHJAI
✍️ Paper: buff.ly/FpYgLDV

All MolmoSpaces assets, scenes, & tools are open + modular, provided in MJCF with USD conversion for cross-simulator portability. Plug in new embodiments, regenerate grasps, & run across MuJoCo, ManiSkill, & NVIDIA Isaac Lab/Sim.

MolmoSpaces supports teleoperation via mobile platforms like Teledex—collect demonstrations right from your phone, compatible with all our embodiment setups, including DROID and CAP; no extra configuration is needed.

📐 MolmoSpaces-Bench is our new benchmark for evaluating generalist policies under systematic, controlled variation. Researchers can vary one factor at a time – from object properties to layouts, task complexity, lighting, dynamics, and instruction phrasing – across thousands of realistic scenes.

MolmoSpaces ships with a massive bank of validated grasps for rigid and articulated objects, loadable directly into environments. An accompanying trajectory-generation pipeline supports reproducible demonstrations and imitation learning at scale.

MolmoSpaces builds on two foundations: Objaverse, one of the largest open collections of 3D objects, and our THOR family of interactive simulation environments, all unified with physics-grounded simulation + validated physical parameters tuned for realistic manipulation. ⚙️

The next wave of AI will act in the physical world, but building robots that generalize across new environments – rather than replaying learned behaviors – requires far more diverse training data than exists today.

That's where MolmoSpaces comes in.

Introducing MolmoSpaces, a large-scale, fully open platform + benchmark for embodied AI research. 🤖

230k+ indoor scenes, 130k+ object models, & 42M annotated robotic grasps—all in one ecosystem.

MolmoSpaces ships with a massive bank of validated grasps for rigid and articulated objects, loadable directly into environments. An accompanying trajectory-generation pipeline supports reproducible demonstrations and imitation learning at scale.

MolmoSpaces builds on two foundations: Objaverse, one of the largest open collections of 3D objects, and our THOR family of interactive simulation environments, all unified with physics-grounded simulation + validated physical parameters tuned for realistic manipulation. ⚙️

The next wave of AI will act in the physical world, but building robots that generalize across new environments – rather than replaying learned behaviors – requires far more diverse training data than exists today.

That's where MolmoSpaces comes in.

Thanks for the feedback. It's not a perfect tool; hallucinations may occur, particularly given the model's small size.