Thijs van der Plas

Towards deployment-centric multimodal AI beyond vision and language Multimodal artificial intelligence (AI) integrates diverse types of data via machine learning to improve understanding, prediction, and decision-making across disciplines such as healthcare, science, ...

arXiv: arxiv.org/abs/2504.03603

Towards deployment-centric multimodal AI beyond vision and language Nature Machine Intelligence - Multimodal AI combines different types of data to improve decision-making in fields such as healthcare and engineering, but work so far has focused on vision and...

View-only: rdcu.be/eL0JC

This was a lot of fun to work on -- big thanks to everyone involved and particularly Xianyuan Liu and Haiping Lu for leading this effort!

Many great scientific challenges are both multimodal and multidisciplinary. In our perspective we discuss the additional challenges this brings for developing deployable AI, and provide recommendations to address these challenges early on.

Towards deployment-centric multimodal AI beyond vision and language - Nature Machine Intelligence Multimodal AI combines different types of data to improve decision-making in fields such as healthcare and engineering, but work so far has focused on vision and language models. To make these systems...

Our perspective on deployment-centric, multimodal AI beyond vision and language is now out in Nature Machine Intelligence!

www.nature.com/articles/s42...

GitHub - LTER-LIFE/llm-metadata-harvester: LLM metadata harvester LLM metadata harvester. Contribute to LTER-LIFE/llm-metadata-harvester development by creating an account on GitHub.

Code: github.com/LTER-LIFE/ll... [5/5]

Flexible Metadata Harvesting for Ecology Using Large Language Models Large, open datasets can accelerate ecological research, particularly by enabling researchers to develop new insights by reusing datasets from multiple sources. However, to find the most suitable datasets to combine and integrate, researchers must navigate diverse...

Paper: doi.org/10.1007/978-... [4/5]

We presented our tool at the recent EcoDL workshop at TPDL2025, and our paper and code are now both available open-access at the links below! Big thanks to the rest of the team, and please let us know any thoughts and suggestions as we continue to develop this tool! [3/5]

In a nutshell, we successfully used LLMs to flexibly extract and convert metadata of ecological datasets (e.g., by scraping datasets webpages), with equal accuracy for both structured and unstructured metadata. [2/5]

Can LLMs do our 'digital dishes'? Or in other words, what are tedious, small tasks that researchers often face and could be automated by LLMs? We've worked on one such task: extracting dataset metadata and converting these to a single format to build a dataset knowledge base. [1/5]

Linking remote sensing, citizen science data and AI could transform environmental monitoring In our new paper, published in Ecological Solutions and Evidence, we provided a perspective on how three areas of science – remote sensing, citizen science, and machine learning (a form of AI) – could...

Great read from Michael Pocock: Linking remote sensing, citizen science data and AI could transform environmental monitoring | UK Centre for Ecology & Hydrology www.ceh.ac.uk/news-and-med...

There is still lots of scope for further improvements; if that's of interest please don't hesitate to get in touch!

We combined sentinel-2 images and UKBMS butterfly occurrence records to predict butterfly species presence from satellite data. We developed a soft contrastive loss that acts as a regulariser and improves prediction accuracy.

Published last week at the CVPR FGVC Workshop; our paper on "Predicting butterfly species presence from satellite imagery using soft contrastive regularisation".

PDF (with links to data/code) available here:
openaccess.thecvf.com/content/CVPR...

With @david-alexander.bsky.social

In our latest perspective article, we outline how ML can overcome 4 current obstacles for large-scale, high-resolution monitoring of protected areas.

doi.org/10.1002/2688...

Hope this stimulates the conversation and provides a pathway of how ML research can be applied for monitoring PAs at scale.

🎯 How can we empower scientific discovery in millions of nature photos?

Introducing INQUIRE: A benchmark testing if AI vision-language models can help scientists find biodiversity patterns- from disease symptoms to rare behaviors- hidden in vast image collections.

Thread👇🧵

Hi, I'm using satellite data to predict species biodiversity! Could I be added please :) Thanks!

Mapping the Past Against the Present Past Landscapes, Present Discoveries: A Data Science Approach to Rediscovering Field Systems from Historic Ordnance Survey Maps

Using computer vision and #MapReader software, we analysed the loss of field boundaries across the #PeakDistrict Since the 1950s, the White Peak has seen a 12% reduction-551 km lost from an original 4,814 km. A stark reminder of landscape change 📊🌿 storymaps.arcgis.com/stories/5c89... #maps #GIS