Weaviate

Please join us in welcoming these awesome new joiners today!

Faustas Butkus - Software Engineer 🧑‍💻
Tobias Christiani - Research Engineer ⌨️
Jacob Heier - Enterprise Account Executive 📈
Kamil Tyborowski - Software Engineer 🧑‍💻

We're so excited to have you on the team!

#teamweaviate

Please join us in welcoming three new joiners this week! 🙌

Danielle Washington - Technical Documentation Writer ⌨️
Giorgos Kampitakis - Full Stack Engineer 👨‍💻
Mary Ann Casugod - Product Support Engineer 👩‍💻

Welcome to the team! 💫

#teamweaviate #newjoiners #techjobs

We've hit an exciting milestone at Weaviate: the 100 people mark! 🥳

As a fully remote company, our teams span the globe, and it's incredible to see talented people from all corners of the world unite behind our mission. ✨

Check out our open positions at https://weaviate.io/company/careers

Welcome to these new joiners that start today!

Austin Freels - Director of Customer Success Engineering 👥
Madison Sanders - Enterprise Account Executive 📈
Elisa Seay - Enterprise Account Executive 📈
Nathaniel Ma - Enterprise Account Executive 📈

We're so excited to have you on the team!

👉 Get started here: cloud.google.com/vertex-ai/ge...
📗 Notebook: github.com/GoogleCloudP...
💡 Soon, many more examples!

Exciting News 🚨

Weaviate is spotlighted in Google’s 5-Day GenAI Intensive course hosted by Kaggle!

Explore the whitepaper ‘Embeddings & Vector Stores’ by Anant Nawalgaria & Xiaoqi Ren.

Read the full paper here:
https://www.kaggle.com/whitepaper-embeddings-and-vector-stores

Start your year off with something new✨

Combine @inngest.com's workflow engine with Weaviate's vector search capabilities and create powerful agentic applications that practically run themselves! A big step up for anyone working with large-scale RAG systems.

Check out the blog! lnkd.in/gSW-xHVf

We're so excited to welcome this awesome group of people to the team this week! 🎉

Dyma - Software Engineer Client Libraries
Ivan - Technical Documentation Writer
Brendan - Enterprise Account Executive
Aaron - Enterprise Account Executive
Sean - Business Development Representative

An Overview on RAG Evaluation | Weaviate Learn about new trends in RAG evaluation and the current state of the art.

𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁:
This focuses on how well generated answers are both accurate and relevant. It also notes the shift from focusing mostly on spotting false information (hallucinations) to other metrics like how reasonable and specific the answers are.

An Overview on RAG Evaluation | Weaviate Learn about new trends in RAG evaluation and the current state of the art.

𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁:
This discusses new ways to measure search accuracy using large language models (LLMs) for context precision and recall. It also mentions how human evaluators have traditionally been used to assess recall, precision, and ranking quality (nDCG).

An Overview on RAG Evaluation | Weaviate Learn about new trends in RAG evaluation and the current state of the art.

𝗜𝗻𝗱𝗲𝘅𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁:
This is about measuring how well the search algorithm finds the closest matches to a query. The main question is: When do mistakes from the approximate search start affecting the overall search results?

You can only improve what you measure.

But how do you measure the performance of a RAG pipeline?

To best identify areas of improvement, it is beneficial to evaluate your RAG pipeline component-wise 🔽

Read more on our blog: https://weaviate.io/blog/rag-evaluation

OpenAI's Matryoshka Embeddings in Weaviate | Weaviate How to use OpenAI's embedding models trained with Matryoshka Representation Learning in a vector database like Weaviate

Read more on our blog: https://weaviate.io/blog/openais-matryoshka-embeddings-in-weaviate
Or check out this video: https://www.youtube.com/watch?v=ZvnKlUtMOkQ

OpenAI's Matryoshka Embeddings in Weaviate | Weaviate How to use OpenAI's embedding models trained with Matryoshka Representation Learning in a vector database like Weaviate

With this training procedure, the first dimensions (8, 16, 32, etc.) end up storing much more information than the later dimensions. This means that MRL can be used as a compression mechanism by shortening the embeddings by removing dimensions from the end of the sequence.

OpenAI's Matryoshka Embeddings in Weaviate | Weaviate How to use OpenAI's embedding models trained with Matryoshka Representation Learning in a vector database like Weaviate

If we call a normal training loss function L, the MRL loss function would just be something like: Loss_Total = L(upto 8d) + L(upto 16d) + L(upto 32d) + ... + L(upto 2048d), because of the sum over all nested vectors.

Matryoshka Representation Learning (MRL) is a hierarchical representation learning technique that allows for flexible dimensionality in vector representations by making a key change to the loss function.

Happy New Year from everyone here at Weaviate! 🎉

Thank you all for another year full of innovation, community, and great memories. We appreciate each and every one of you, and cannot wait to see what is in store for 2025!

Cheers to another trip around the sun!! 💚🥂

The top Weaviate video of 2024 was.. 🥁
Open Source RAG with Ollama!

Since the video was published, our open source RAG demo, Verba, has had even more updates, features, and tutorials published. You can try it out here: https://verba.weaviate.io

Video: https://www.youtube.com/watch?v=swKKRdLBhas

What does the future of AI look like in 2025?

We can’t wait to find out! Attendees of the Berlin AI Hack Night gave some insightful responses on how they think AI will improve in 2025, and what excites them most about it.

Join our upcoming events here: https://lu.ma/weaviatecommunityevents

Learn how you can use Weaviate and all of these technologies in our Integration Ecosystem page:
https://buff.ly/4fFzAgF

Check out Recipes, our repository where we share end-to-end notebooks on building applications ranging from RAG to Agentic RAG:
https://buff.ly/41Q8uQN

What a year it has been for Weaviate and the AI ecosystem!

Building AI-Native applications requires a variety of tools. We’ve clustered our technology partners into six categories:
1. Cloud hyperscalers
2. Model providers
3. Data platforms
4. Compute infrastructure
5. LLM frameworks
6. Operations

2024 was a massive year for Weaviate and the AI community! 🚀

Our global roadshow showcased real-world applications of vector databases, driving innovation across industries.

Relive the action:
https://buff.ly/4gKsWaF

There’s more to come in 2025. Keep an eye out for Weaviate in a city near you!

• What is Agentic RAG: https://buff.ly/3UBxmYf
• Verba: https://buff.ly/3Td1SWC
• Late Chunking: https://buff.ly/3zxEzB7
• Advanced RAG Techniques: https://buff.ly/3ZIMPsg
• Choosing the Best Embedding Model: https://buff.ly/3AbdV0R
• Building a Local RAG System: https://buff.ly/3PalZmU

Looking for the most influential AI blogs from this year?

Here's our curated list 📝

(Save for later)

❄️ Happy Holidays from Weaviate!

Whatever you’re celebrating, may it be filled with peace, joy (and maybe a little snow).

As we wrap up 2024, we want to thank our incredible Weaviate Community for making this year so special!

Join us on a journey through this year's highlights and discover why our community continues to inspire us every day.

Blog | Weaviate Blog

Read more about the release on our blog: weaviate.io/blog
Try it on Weaviate Cloud: console.weaviate.cloud
Open Source Release: github.com/weaviate/wea...

Blog | Weaviate Blog

More community ❤️ - Jun Ohtani contributed a Japanese BM25/Hybrid tokenizer to Weaviate!

Since 1.27.0, we’ve also added:
• Weaviate Embeddings - to make vector embedding generation even more seamless!
• Support for Voyage AI's multimodal embeddings

Blog | Weaviate Blog

Under the hood features: our async vector indexing has gotten even more robust and faster. For the adventurous, you can try out the experimental BlockMax WAND indexing for yourself.

Blog | Weaviate Blog

The big highlight is our technical preview of 𝗿𝗼𝗹𝗲-𝗯𝗮𝘀𝗲𝗱 𝗮𝗰𝗰𝗲𝘀𝘀 𝗰𝗼𝗻𝘁𝗿𝗼𝗹. This gives you granular control over who can do what! Set permissions at collection, object, and cluster metadata levels with custom roles or use predefined roles.