Ai2

GitHub - allenai/olmocr: Toolkit for linearizing PDFs for LLM datasets/training Toolkit for linearizing PDFs for LLM datasets/training - allenai/olmocr

💻 Ready to try it? Dive into the repo & docs: buff.ly/bwd8Xfh
💬 Chat about it in our Discord: buff.ly/DUlWmg3

⚡ Speed boost: up to 3,400 tokens/sec on a single GPU, powered by native FP8 compression and a smarter prompting ↔ retry loop.
🛠️ Reproducibility built‑in: brand‑new trainer code lets you recreate our checkpoints or fine‑tune your own models with just a few commands.

📊 Accuracy upgrade: +3 pts on the public olmOCR‑Bench means cleaner, more reliable text from your noisiest PDFs.

📝 olmOCR v0.2.1 has arrived with new models! Our open‑source OCR engine now reads tougher docs with greater precision—and it’s still 100 % open. 👇

YouTube video by Ai2 Why Ai2? | Nathan Lambert & Kyle Wiggers

www.youtube.com/watch?v=T15z...

We’re committed to cementing Ai2’s role as the trusted non-profit voice in the AI landscape by prioritizing community engagement and championing transparency.

📏 Benchmarks matter.

At Ai2, we’re driving meaningful change by creating robust evaluation tools. These help others cut through hype and focus on what truly enables breakthroughs. 💡

🔬 What sets Ai2 apart: We’re #1 on the @hf.co heatmap for a reason—we’re continually releasing new research to advance the field.

New interview drop! ⚠️ Ai2 Senior Research Scientist @natolambert.bsky.social sat down with @kylelwiggers.bsky.social, Ai2’s new Comms Lead, to talk about what sets Ai2 apart from other frontier AI labs, and our role in the research community. Watch the highlights. 👇

Honduras Strengthens Conservation with EarthRanger Across Protected Areas Honduras is deploying EarthRanger across 75 protected areas to improve wildlife monitoring, support rangers, and advance its zero-deforestation strategy.

📝 Learn more here: buff.ly/yocTk60

🌐 EarthRanger is used by hundreds of teams globally in Latin America—Honduras joins Paraguay, Panama & Mexico in using the platform nationwide. By supercharging Honduras’ work with real‑time intel, Ai2 supports efforts to safeguard natural resources, today and for generations.

It’s worth underscoring: this rollout is also about protecting people. ❤️

With EarthRanger, conservationists can now track the movements of their teams, share locations, and flag threats—adding a layer of safety for those on the frontlines of conservation.

As EarthRanger rolls out across the country, teams are spotting critical patterns like a rise in snake encounters near communities. As habitats shrink, snakes are moving closer to people. Now, teams have the data to raise awareness and reduce risk where it matters most.

The rollout spans 75 land & marine protected areas and backs Honduras’ bold “Zero Deforestation by 2029” pledge—giving conservationists instant views of where their wildlife are, where threats are, and where to act in places such as Puca.

🌎🛡️ On #WorldRangerDay, we’re proud to share that Honduras is expanding the use of @EarthRangerTech – our real‑time wildlife‑protection tech platform – to advance zero‑deforestation and safeguard biodiversity. 🧵

Ai2 is excited to be at #ACL2025 in Vienna, Austria this week. Come say hello, meet the team, and chat about the future of NLP. See you there! 🤝📚

➡️ Link: buff.ly/9GnUFu1

🚀 Ai2’s official subreddit has arrived. Discuss our latest work and happenings in the AI community, from LLMs to bleeding-edge scientific work. We look forward to seeing you there. 👋

issues w preference LM benchmarks:

🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)

led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025

Takeaway: evaluations need context to reflect real‑world use and to ensure models serve all users.

📚 Read more in our blog: buff.ly/tFGX95o
💻 Get the code: buff.ly/QACrWjr
📊 Download the data: buff.ly/h4C28PV

For example, we found that default model answers often align better with users from Western, higher‑income backgrounds—an equity gap that context‑free testing missed. 🌍⚠️

Our fix: contextualized evaluation. Supplying the missing info…
1️⃣ Boosts evaluator agreement
2️⃣ Sometimes completely flips which model “wins”
3️⃣ Leads to more judgments based on content, not style
4️⃣ Exposes biases in default model responses.

We analyzed 3,580 queries randomly sampled from popular language model benchmarks, including Chatbot Arena. We found that underspecification is widely prevalent:
➡️ The vast majority of queries are open-ended (76%)
➡️ Many are also subjective (19%) or incomplete (18%)

When evaluators get these “underspecified” prompts, they have to guess the backstory. The result? Unstable rankings and shaky conclusions about model quality. ⚠️

An LLM prompt like “Is coffee good for you?” feels simple, but a helpful answer depends on who’s asking (e.g., someone who’s pregnant versus a person with high blood pressure). Most benchmarks leave that context out.

In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. 🧵👇

AutoDS shows how AI can turbo‑charge discovery. 🚀

📚 Read more in the blog: buff.ly/O2SXfHv
📝 Check out the paper: buff.ly/jlBxuZJ
💻 Try AutoDS for yourself: buff.ly/fOl9pJG

Evaluated across 21 real-world datasets, AutoDS outperformed competitors by 5-29% at finding discoveries that are surprising to an LLM. In a human study that involved more than 500 hypotheses, 67% of the discoveries made by AutoDS were also surprising to the experts. 📊

Like a tireless researcher, AutoDS spins up its own hypotheses, runs the stats, learns from the outcomes, and then repeats. 🔄 The system can use the results of statistical experiments it generates and conducts to propose new hypotheses.🧑🔬💡

Great science starts with great questions. 🤔✨ Meet AutoDS—an AI that doesn’t just hunt for answers, it decides which questions are worth asking. 🧵