Kyle Lo

thx to all the feedback from OSS community!

our olmOCR lead Jake Poznanski shipped a new model fixing lotta issues + some more optimization for better throughput

have fun converting PDFs!

olmoTrace for connecting model generations to training data won Best Paper for System Demonstrations at #ACL2025!

People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations!

Jul 30, 11:00-12:30 at Hall 4X, board 424.

issues w preference LM benchmarks:

🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)

led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025

lol 'what does skibidi mean' was not on my bingo card for @actinterp.bsky.social‬ workshop #icml2025 😆

presenting olmOCR at the poster session (2:15pm 211 West) for #codeml workshop at #icml2025!
🐟 fully open source OCR, comparable or better than frontier VLMs
🐠 all weights, data, code free & public
🐡 new benchmark of OCR "unit tests" on diverse PDFs & challenging OCR cases

Presenting two posters at ICML over the next two days:
- Both at 11am - 1:30pm
- Both about how to improve pre-training with domains
- Both at stall # E-2600 in East Exhibition Hall A-B (!)

Tomorrow: WebOrganizer w/ @soldaini.net & @kylelo.bsky.social
Thursday: MeCo by @gaotianyu1350.bsky.social

will be at #icml2025, lemme kno if wanna chat about OLMo pretraining data curation, evaluation, data mixing, etc!👋

find us at poster sess on 📅Wed 7/16 @ 11am⏲️ to learn about Web Organizer, distilling web data taxonomies into small models & using them for LM data mixing!

only took few days to descend into madness

shame internet ppl dont kno u as bringer of office cookies

it's still working for me? wat r u seein

back from copenhagen & berkeley travels, now moving into new @ai2.bsky.social office!

thx for summary! im still wondering if it implies what ur saying. the judge is giving a pass because the filters around the model prevent content leakage. couldn’t one also interpret this as open model developers may be expected to provide such filters alongside released weights?

YouTube video by Paramore Paramore - The News [OFFICIAL VIDEO]

🎧

youtu.be/YSFa_wOZPXg?...

thx for organizing! great to meet NLP folks & consume fancy bread 🥖🍞🥐

woahh thx this is clearer than how in presented it 😆

we developed the benchmark independently so no dev/test leakage, and even so, results show olmOCR produces often higher quality output than even proprietary OCR tools & is way cheaper + local as well!

our team will be at #ICML2025, come find me, Jake (our OCR lead!) & @soldni.bsky.social there!

the benchmark works based on thousands of "unit tests"

so instead of fuzzy matching between a model-generated table with a gold reference table,

we define Pass/Fail tests like "the cell to the left of the cell containing 0.001 should contain 1.96"

excited to release our new benchmark for OCR addressing 3 eval challenges:
🐟 coverage of many types of docs (born digital vs old scans, pages w tiny fonts, etc)
🐡 coverage of many different OCR targets (e.g. equations, tables, etc)
🐠 apples-to-apples comparison across systems

we won honorable mention for Best Paper at #CVPR2025 🏆 for Molmo & Pixmo, showing the value of high-quality data for VLMs!

recalling when we released same time as Llama 3.2 😆

huge kudos to Matt Deitke, Chris Clark & Ani Kembhavi for their leadership on this project!

@cvprconference.bsky.social

google down, guess ill go smell flowers or sthn 🤷‍♂️

excited to see this release of 1M public domain & CC zero books, digitized and OCR'd! 👏 big win for open data, congrats to the authors!

arxiv.org/abs/2506.08300

thx for sharing! from your paper:

“the artifacts that they protect, namely model weights and model outputs, are largely not copyrightable, making it unclear whether there is even anything to be licensed.”

dumb question but I thought it’s possible to license non-copyrightable artifacts?

fave part of this work is "ties" ⚖️ cases with many correct answers

"Q: name a color of rainbow", a good reward model should know all answers "green", "blue", ... are correct but w/ no strong color preference

congrats @saumyamalik.bsky.social et al @ai2.bsky.social & friends on RewardBench 2!

latex dummy here - what problem does this solve?

@sarahwiegreffe.bsky.social pls?

reviewers asking for comparison w qwen 3, have yall forgotten model was released a month ago lol 🤪

i think highest bar is having yanai read the draft & say it is good 😆

what we dont see is how many submission attempts, each incurring reviewing costs, until one accepted 🤷‍♂️

Significant

feels literally xkcd.com/882