MAGMaR Workshop

EvalAI: Evaluating state of the art in AI EvalAI is an open-source web platform for organizing and participating in challenges to push the state of the art on AI tasks.

🚨 IT'S HERE! 🚨
The Eval Leaderboard is now LIVE! 🏆💻

Our video retrieval collection stumps most pre-trained models. See if you can build a better system!

eval.ai/web/challeng...

ACL 2025 Workshop MAGMaR Welcome to the OpenReview homepage for ACL 2025 Workshop MAGMaR

Submit here: openreview.net/group?id=acl...

🚨 Deadline extension! 🚨

The MAGMaR paper submission date has been extended to May 1, 2025 (archival and non-archival). Show off your multimodal + RAG projects!

The repository is set for manual approval. If you are not approved within a day, please let us know.

The leaderboard will be up by the end of the week. Please feel free to reach out with any questions!

* CLIP features
* SigLIP features
* Whisper ASR transcripts
* PaddleOCR output
* ICDAR OCR output (Etter et al., 2023)
* Florence video captions (test only)

The train queries/judgments are also available, along with the test queries.

hltcoe/MultiVENT2.0 · Datasets at Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We are happy to announce the full release of MultiVENT 2.0 through @hf.co!

huggingface.co/datasets/hlt...

All train and test videos are provided, subset into tar files of 100 videos. Baseline extracted features are provided as well, including...

We now have a Slack channel for the 2025 Workshop on Multimodal Augmented Generation via Multimodal Retrieval!

This channel will serve as the primary communication method between authors, participants, and organizers: join.slack.com/t/magmarshar...

New Workshop on Multimodal Augmented Generation via MultimodAl Retrieval (MAGMaR) to be held at @aclmeeting.bsky.social ACL in Vienna this summer. We have a new shared task that stumps most LLMs - including ones pretrained on our test collection. nlp.jhu.edu/magmar/