Joe Barrow

Now, some acknowledgments: this work was made possible thanks to a generous compute grant from Lambda!

And I've got a hosted version of the model that I'll be sharing in a couple days hosted on @modal-labs.bsky.social, which makes it basically free for me to host and scale

GitHub - jbarrow/commonforms: CommonForms dataset and models CommonForms dataset and models. Contribute to jbarrow/commonforms development by creating an account on GitHub.

As part of the paper, I'm working on releasing the dataset and FFDNet models on HuggingFace.

Those will be out in the coming days, you can follow along here: github.com/jbarrow/comm...

🤗Paper: huggingface.co/papers/2509....
arXiv: arxiv.org/abs/2509.16506

Now, just because we filtered for the cleanest forms doesn't mean we got _perfect_ forms. There are still a lot of inconsistencies in how people prepare forms! In future work I'll be looking at mitigating data quality issues like these.

(Note, this doesn't _just_ apply to Acrobat, it's also better than Apple Preview -- neither Acrobat nor Preview even make an attempt at checkboxes, and they're often fooled by any straight, horizontal line. Left: Acrobat, Right: FFDNet)

If we train object detectors to find the form fields on these pages, we get a much cleaner set of forms than if you used Acrobat to automatically prepare your form. (Left: Acrobat, Right: FFDNet).

Step 1 is to filter out for the cleanest forms possible. We start with 8MM PDFs from Common Crawl, and work our way down to ~60k of the cleanest forms we can find. The results is a ~500k page dataset, called CommonForms.

Paper thread of some work I’m *incredibly* proud of, my first single author paper!

Converting a PDF to a fillable form is a hard problem, and a lot of solutions don’t work very well! In CommonForms, I show that you can train models that outperform Adobe Acrobat for <$500! 🧵

Yeah I wonder if that statistic is flipped between the cities (though operated by the same provider — Lyft — I assume?)

No way that 99 out of every 100 riders in Boston have visited more than 27 stations?

Pretty sure you want that number to be lower. :p (my stats for DC ridership)

Horseshoes (and Hand Grenades) - LLM Localization is not Close, but not Close Enough - Joe Barrow TL;DRLarge Multimodal Models (LMMs) can now output bounding boxes when given images as inputs. The results are impressive, but for documents they aren't good enough for real world use, yet. The Probl…

In which I argue that LLM-generated bounding boxes are impressive, but not that useful (yet): notes.penpusher.app/Misc/Horsesh...

Would absolutely love that!

"AI TOPS our stock price"
- Nvidia, today

Ah, yes, that ol' familiar unit of measure "AI TOPS"

Agree, my ideal would be if you could type into an old, cheap, refurb kindle personally.

Here’s a video of a person typing into the Palma: www.reddit.com/r/Onyx_Boox/...

My experience (tablet) is that it’s maybe 10s from pickup to writing — wake up (3s), navigate to apps (2s), open app (3-5s)?

I’ve got one of the older, larger eInk tablets and use it for reading books/papers and taking notes. Battery after several years lasts about a week of average use, longer if I keep WiFi off.

Not necessarily hitting the price point but there are eInk mini tablets (e.g. Boox Palma at around $200) that have Android, no sim (so no phone distractions), and long battery life (thanks to the eInk and being generally underpowered). They accept keyboards, too.

Google Gemini 101 - Object Detection with Vision and Structured Outputs - Joe Barrow - Obsidian Publish This is a missing manual for how to get a simple working prototype up and running with Gemini's vision mode and structured outputs. I'm confident that manual exists elsewhere, but I haven't been able…

I put together a little guide on getting started with Google Gemini -- how to make multimodal calls, get structured outputs, and image bounding boxes to build an object detector.

notes.penpusher.app/Misc/Google+...

Drake meme template. No to: clear, concise prose Yes to: negative vspace

Drake meme template. No to: clear, concise prose Yes to: negative vspace

Holy moly that created an extra half page of space!

A picture of a teapot to the right of a teacup, both in a flat-bottomed basket. The teacup has eta and flowers in it. There are 2 blue bounding boxes on the image, one labeled "Teapot" and one labeled "Teacup" that are over the teapot and teacup respectively.

Gemini 2.0 Flash is pretty good at localization in images. for an LMM (much better than GPT-4o in my experiments).

ML history question: is there an earlier reference to pixel-only in-context (i.e. no fine-tuning) DocVQA performance than the GPT-4 announcement from OpenAI?

Aged white tea, the kind that comes in a compressed disk or ball. My favorite kind of tea, imo they taste naturally quite sweet. Yunnan Sourcing has a bunch!