Leo Boytsov

A Large-Scale Study of Reranker Relevance Feedback at Inference | Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval

Results? If read tables correctly, there's only very modest boost in both recall & NDCG, which is within 2%. Given that the procedure requires a second retrieval, it does not seem to worth an effort.
🟦
dl.acm.org/doi/abs/10.1...

PRF was not forgotten in the neural IR times, but how does it perform really? Revanth Gangi Reddy & colleagues ran a rather thorough experiment and published it SIGIR.
↩️

It was doc2query before doc2query and, in fact, it improved performance (by a few%) of the IBM Watson QA system that beat human champions in Jeopardy!
↩️
research.ibm.com/publications...

I think this is a problem of completely unsupervised and blind approach of adding terms to the query. If we had some supervision signal to filter out potentially bad terms, this would work out better. In fact, a supervised approach was previously used to add terms to documents!
↩️

Fixing this issue produced a sub-topic in the IR community devoted to fixing this issue and identifying cases where performance degrades substantially in advance. Dozens of approaches were proposed, but I do not think it was successful. Why⁉️
↩️

PRF tends to improve things on average, but has a rather nasty property of tanking outcomes for some queries rather dramatically: When things go wrong (i.e., unlucky unrelated terms are added to the query), they can go very wrong. ↩️

Leo Boytsov on X: "🧵40 years ago the SMART IR system was released. It introduced a few key concepts including vector space interpretation of the retrieval process and the relevance feedback algorithm. I also think it was probably the first open source search engine. ↩️" / X 🧵40 years ago the SMART IR system was released. It introduced a few key concepts including vector space interpretation of the retrieval process and the relevance feedback algorithm. I also think it was probably the first open source search engine. ↩️

PRF is an old technique introduced 40 years ago in the SMART system (arguably the first open-source IR system). ↩️
x.com/srchvrs/stat...

🧵Pseudo-relevance feedback (PRF) (also known as blind feedback) is a technique of first retrieving/re-ranking top-k documents and adding some of their words to the initial query. Then, a second retrieval/ranking stage uses an updated query. ↩️

If you submitted a messy paper, it's pointless to address every little comment and promise fixing it in the final version. 🟦

Instead, think hard about questions you can ask. What is the main misunderstanding? What will you have to do so that a reviewer will accept your work next time. Which concise questions can you ask to avoid misunderstanding in the future? ↩️

🧵 Dear (scientific) authors: I am being in the same boat too. However, if you receive a ton of detailed complaints regarding paper quality, do NOT try to address them during the rebuttal phase. It's just a waste of everybody's time. ↩️

Microsoft Is Having an Incredibly Embarrassing Problem With Its AI Despite investing tens of billions of dollars into OpenAI, Microsoft is still, apparently, competing with its business partner.

@microsoft.com faces an interesting issue that might affect others selling wrappers around ChatGPT and Claude models: users prefer to use ChatGPT directly rather than engage with Microsoft's Copilot.
futurism.com/microsoft-co...

I have bittersweet news to share. | Lysandre Debut I have bittersweet news to share. Yesterday we merged a PR deprecating TensorFlow and Flax support in transformers. Going forward, we're focusing all our efforts on PyTorch to remove a lot of th...

This is a rather blockbuster piece of news: the @hf.co library is dropping support for both Jax and Tensorflow.
www.linkedin.com/posts/lysand...

Humans are creating AGI and you claim that their intelligence is overrated?

Laptop keyboards are close to being unusable. Tremendous productivity hit.

Found a hidden gem on IR evaluation methodology from Microsoft "What Matters in a Measure? A Perspective from Large-Scale Search Evaluation."
dl.acm.org/doi/pdf/10.1...

Parental advice: if you master algebra you will know how to deal with your x-es.
@ccanonne.bsky.social feel free to borrow!

Some people say: A prompt is worth a thousand words! Excuse, but have you seen these ones? They are way longer!

However, unlike many others who see threat in the form of a "terminator-like" super-intelligence, @lawrennd.bsky.social worries about unpredictability of automated decision making by entity that's superior in some ways, inferior in others, but importantly is disconnected from the needs of humans. ⏹️

🧵A fascinating perspective on the nature of intelligence and the history of automation/ (and ahem development of AI). It is also a cautionary story of how to not trust AI too much. ↩️

Thus, it was quite insightful to read a recent blog post by
@netflix
detailing their experience in training foundation RecSys LLMs. It’s an informative read, packed with detailed, behind-the-scenes information.
🟦

Pre-training can be non-trivial. If you represent a set of users or items using fixed IDs, your model will not generalize well to a domain with different set of users or items (although there are some workarounds arxiv.org/abs/2405.03562).
↩️

🧵Although pre-trained Transformer models took NLP by storm, they were less successful for recommender systems (arxiv.org/abs/2306.11114). RecSys is hard:
1. The number of users is high.
2. The number of items is high.
3. A cold-start problem is a hard one.
↩️

Preference Leakage: A Contamination Problem in LLM-as-a-judge Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods in model development. While their combination significantly enhan...

" ... that preference leakage is a pervasive issue that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios. All of these findings imply that preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge. "
arxiv.org/abs/2502.01534

🧵"Through extensive experiments, we empirically confirm the bias of judges towards their related student models caused by preference leakage across multiple LLM baselines and benchmarks. Further analysis suggests ..." ↩️

So you mean specifically GRPO or more like a combination of GRPO and binary rewards?

YouTube video by Vector Podcast Debunking myths of vector search and LLMs with Leo Boytsov, Senior Research Scientist, AWS

We also talked about k-NN search, approaches to learning sparse & dense representations for text retrieval, as well as about integration of vector search into traditional DB systems. 🟦
www.youtube.com/watch?v=gzWE...

🧵It was my great pleasure to appear on the vector podcast with @dmitrykan.bsky.social . We covered the history of our NMSLIB library and how it helped shape the vector search industry! ↩️

They gathered some of the best people. They surely have a fighting chance to reduce the cost of serving and reach profitability within a couple of years.

Ok, please, show us a minimal-viable example of doing so. Do not forget that no everyone knows jax and not everyone is willing to learn it. It is not as widely used as numpy, pytorch, or even tensorflow.