Rainer Simon

Wonderful - thanks for reaching out. Looks like I can't DM you. is just followed you. Perhaps you'll have to follow me, too, for DMs to work?

Clarification on what our "VLM pipeline" is (thx @miaout.bsky.social): code and utilities that chain together workflows across AI models – incl. the newer vision-capable language models (VLMs). Goal is to extract structured metadata from images.

Have material you'd like me to test? Ping me!

Thanks good point! (All too easy to forget how much jargon slips in...) I'll follow up with a quote-post.

To clarify: #MachinaEmblematic *is not* the enrichment pipeline, but a completely different project. But feel free to ask me about both :-)

Machina Emblematica I am the Machina Emblematica, built to preserve the knowledge of Joachim Camerarius' Symbola et Emblemata and to guide those who seek it.

Thanks, will check it out. But in this case, my question is specifically about RAG systems, like this: machina.rainersimon.io

Working with a small museum/archive/project with digitized images but little metadata?

I'm looking for testers for a VLM pipeline for auto-enrichment (transcription, captions, tags, IIIF). If you share a few sample images, I'll run them through + share results. Would love to hear your feedback!

Danke, schau's mir gerade an - da sind definitiv einige interessante Projekte dabei!

Q to the hivemind: I'm wondering if anyone else around here's been building RAG systems for cultural collections/museums?

I'm curious about what/how you're indexing. Metadata, transcripts/chunks? Images full or segmented? LLM-generated descriptions? What's your queries and response workflow?

Machina Emblematica I am the Machina Emblematica, built to preserve the knowledge of Joachim Camerarius' Symbola et Emblemata and to guide those who seek it.

Source at the Bavarian State Library: www.digitale-sammlungen.de/de/view/bsb1...

Machina Emblematica: machina.rainersimon.io

The Simivulpem, also known as the "half-fox" or "fox-monkey," is a fascinating creature described in the Symbola et Emblemata. This unique animal, found in regions of India, is a hybrid-like being with distinct features: the front part resembling a fox and the back part resembling a monkey. It has a tail similar to that of a cercopithecus, bat-like ears, and a pouch under its belly, akin to a marsupial's crumena, where it carries its young. This emblem symbolizes the protective instincts of parents and the care they provide to their offspring. The Simivulpem's ability to carry its young in a pouch highlights the theme of parental love and protection, ensuring the safety of its young as it moves about. This is further emphasized by other similar creatures mentioned, such as the animal in the Moluccas islands, which also carries its young in pouches, and the marine dog, which can retract its young into its body during danger. The Simivulpem serves as a metaphor for the nurturing and safeguarding role of parents, urging us to appreciate and cherish the care we receive from those who love us. It also reflects the idea of carrying our burdens and responsibilities with us, much like the creature carries its young, reminding us to be mindful of our duties and the support we have from others.

An image (supposedly) of the Simivulpem

Odd find of the day from #MachinaEmblematica: the Simivulpem, a wolf/monkey hybrid.

Story sounded plausible. But I wonder how much is just AI hallucination. Google only gave two hits, both on the Google Books edition of Symbola et Emblemata.

Any early modern creature lore experts who know more? 🙂

Screenshot of the IMMARKUS annotation interface with three historical images—an Oxford bookplate, a medieval-style world map, and an early printed nautical map—displayed side by side, with metadata fields visible on the right.

🎉 IMMARKUS 1.0.5 is out! The latest release of our open-source image annotation tool for humanities, heritage & museum professionals is now available – with bug fixes, productivity enhancements, and improved #IIIF annotation import.

Try it here: immarkus.xmarkus.org
Docs: github.com/rsimon/immar...

An emblem depicting frogs on a pond.

Chat message from the Machina Emblematica explaining how frogs are used as a symbol for arrogance and deceit, that cannot withstand the light of truth.

The #MachinaEmblematica – our mysterious chatbot guide into the 16th century encyclopedia "Symbola et Emblemata" is now reasonably mobile-friendly.

Check it out if you haven't already!

machina.rainersimon.io

Cold and rainy but good fun at the annual Science Day in Leuven where we demonstrated how anyone can easily collect and work creatively with high-quality images streamed from cultural organizations globally. #dvdw2025 #IMMARKUS

We built #MachinaEmblematica as an outreach/engagement tool rather than for research. But as someone with zero knowledge of 16C emblem books (or Latin!) I find it an amazing way to poke around & discover odd new things—far richer than randomly browsing #IIIF images alone.

machina.rainersimon.io

Relevant pages linked to the answer. Although relevance may vary (we didn't yet figure out a threshold to drop less relevant results). But by and large, if the Machina talks about it, the image will be among them.

The Machina Emblematic user interface explaining an image depicting a dragon. Image on the left, text on the right.

@rockettara.bsky.social and I've been teaching the #MachinaEmblematica some new tricks: better image interpretation and conversational context, smarter source retrieval from the Symbola et Emblemata. It’s a bit slower now… we’ll work on that next. 😉

Try it here: machina.rainersimon.io

And a decent Leuchtmarker can be a perfect complement to those! 😀

text-annotator-js – our JavaScript library for text and PDF annotation – is looking for a new name.

Something shorter, nicer and more memorable. (Current favourite: "leuchtmarker" – German for "highlighter pen".) Got ideas? Drop them below!

github.com/recogito/tex...

A screenshot of the Machina Emblematica, giving an explanation of the depicted emblem (a beaver), the accompanying text, and the possible meaning and author's intention.

Our #MachinaEmblematica has a few quirks. But if you ask for specifics of a particular page or emblem, I feel that the multimodal model generates pretty useful responses!

machina.rainersimon.io

Rendered image of the "Machina Emblematica" a fictional arcane mechanism with cogs wheels and pistons.

The chat interface of the Machina Emblematic website, dark background, an open page of the Symbola on the left side, the conversational chat interface on the right.

An artists impression of the Machina's brain, looking like a layered microchip with an organic nervous system. The image is used as a decoration on the splash page.

Side-project announcement! @rockettara.bsky.social and I built a machine that never existed… to read a book from 1590. Meet the #MachinaEmblematica, a chatbot for exploring the Symbola et Emblemata. machina.rainersimon.io

It actually works pretty well on mobile, too! With the caveat that – like Isla de Muerta – the 14th annotation can only be found by those who already knew where it was.

Hint: you have to hit the little marker before "prevented". The rest is buried below the other, shorter annotations above.

Not yet! :-) I'll still hold off for a bit with posting the "solution". But it is a tricky edge case that's, frankly, not at all solved in the current system. (Hint: you *can* find the 14th annotation on a desktop PC if you explore carefully. But on touch, it's pretty much impossible.)

So, the caveat: the demo has 14 annotations total. Can you find them all?

There's an edge case I haven't cracked yet. It makes one particular annotation difficult (and unintuitive) to select. If you spot it (or don’t!), I’d love to hear what you think.

Example text with multiple orange overlapping highlights, illustrating the new design ideas.

Trying a new approach to my overlapping annotations problem:

• No more stacked underlines
• Instead, overlaps get small start markers
• Exact overlaps show a tiny count number
• Color density still hints at overlap

Try it live: dense-annotations.netlify.app

One caveat… see next post 👇

Could be a solution. But I think it may also become hard to navigate quickly with more overlaps – plus also a potential accessibility pitfall. (A11y-wise, underlines should actually be pretty ok I believe.)

Could be a solution. But I think it will also become hard to navigate quickly – plus also a potential accessibility pitfall. (A11y-wise, underlines should actually be pretty ok I believe.)

Here's my main dilemma with overlapping text annotations: Underlines become a mess once you stack more than three. At the same time, there's really no way around them if you want to show where overlap starts and ends.

I guess the Q is how/when to drop underlines without losing too much information?

Thanks! Do you mean their alignment viewer? (E.g. pubs.perseids.org/bodin/). It's a good reference, but I think only uses highlights to visually connect passages – which means it wouldn't have any of the "hard" issues, with overlaps and stacked annotation layers. Or did you think of another tool?

And it's not even because of one of your texts this time 😀 (well, the ever-classic sample text still is a nod back to that time, of course!) But, yes: if we ever manage to come up with a truly great solution for super-dense/-complex annotation, we'd surely have to name that release after you ;-)

A heavily annotated text snippet (the opening paragraph of Homer's Odyssey) with many overlapping orange highlights and underlines, and numeric markers that indicate places that have multiple annotations on the same text range.

Task for today: brainstorming better ways to visualize heavily annotated text. The more I think about it, the harder it gets...
• Partial overlaps → clutter & a11y issues
• Exact overlaps → even worse
• Stacked underlines → work great... until they don't.

Seen elegant solutions? I’d love to chat!