Omar Khattab

Let the Model Write the Prompt | Drew Breunig #dspy #promptengineering #llms #generativeai

Let the Model Write the Prompt Notes from a talk I delivered at the 2025 Data + AI Summit, detailing the problem with prompts in your code and how DSPy can make everything better.

Here's the write up of my Data+AI Summit talk on the perils of prompts in code and how to mitigate them with DSPy. www.dbreunig.com/2025/06/10/l...

Have you heard the news? #MLflow now supports tracking for DSPy optimization workflows—just like it does for #PyTorch training!

Keep reading to see what this means for your #LLM projects… 👇

#opensource #dspy #oss

MLflow Community Meetup | April 23 · Luma Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT! We’re bringing two exciting presentations to the community: 🔹 MLflow + DSPy…

📣 TODAY at 4PM PT - MLflow Community Meetup!

🔗 Register today 👉 lu.ma/mlflow423

Join the global MLflow community for two exciting tech deep dives:
🔹 MLflow + #DSPy Integration
🔹 Cleanlab + #MLflow

🎥 Streaming live on YouTube, LinkedIn, and X
💬 Live Q&A with the presenters

#opensource #oss

MLflow now supports tracking for #DSPy (Community) optimization — just like it does for @pytorch.org training! 🙌

#MLflow is the first to bring full visibility into DSPy’s prompt optimization process. More observability, less guesswork.

Get started today! ➡️ medium.com/@AI-on-Datab...

#opensource

MLflow Monthly Meetup · Luma Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT! We’re bringing two exciting presentations to the community: 🔹 MLflow + DSPy…

Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT! 🗓️

🔹 Explore the new MLflow + #DSPy integration
🔹 Learn how Cleanlab adds trust to AI workflows with MLflow

💬 Live Q&A + demos
📺 Streamed on YouTube, LinkedIn, and X
👉 RSVP: lu.ma/mlflow423

#opensource #mlflow #oss

History - DSPy The framework for programming—rather than prompting—language models.

Nice work! For history:

dspy.ai/api/primitiv...

This was built by a long-time DSPy community member!

Yes there's an evals crisis, but evaluating *models* is not even the right question most of the time

LangProBe from Shangyin Tan, @lakshyaaagrawal.bsky.social, Arnav Singhvi, Liheng Lai, @michaelryan207.bsky.social et al begins to ask what complete *AI systems* we should build & under what settings

🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs!

We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.

It doesn't help that the we in ML often only design abstractions leak all kinds of implementation details. Folks often define ML itself in terms of techniques, not problems!

But it's prematurely abstracting that leads to the bitterness of wasted effort, and not "modularity doesn't work for AI". 2/2

Composition & abstraction are the foundations of CS, but are clearly absent in modern ML.

It's not that they're not crucial for intelligent software. But it takes building many half-working systems to abstract successfully, and it takes good abstractions to have primitives worth composing.

🧵1/2

4) By default, IR methods that use "multiple vectors" (e.g., cross-encoders) are unscalable. It seems like a necessary tradeoff, but the fascinating thing in late interaction is that it's easy to implement in asymptotically sub-linear ways, thanks to pruning.

Hope this was useful!

3) "Multi-vector" makes it sound like these approaches win because they store "more stuff".

But that's not true: if you look at how aggressive ColBERTv2 representations are compressed, it's often ~20 bytes per vector (like 5 floats), which can be smaller than popular uncompressed single vectors!

For dot products, every time you "fix" one query--document pair, you likely break so many other pairs by moving the query and/or document representations.

For ColBERT, you typically *fix* more than you break because you're moving *tokens* in a much smaller (and far more composable!) space.

The problem isn't the vector representation, it's the **learnability of the scoring function**.

A dot product is just very hard to learn. An intuition I learned from Menon et al (2021) is that:

2) More importantly, there's nothing to say you can't store a TON of information in a single vector. And it's easy to use multiple vectors and gain *zero* improvement over a single-vector, e.g. if you replace MaxSim with AvgSim in ColBERT, without any other changes, it doesn't work!

1) If you take ColBERT and force it to use only a constant number of vectors (e.g., 16), it'll barely outperform one vector in the general case.

It's not that you need token-level alignment per se (you don't either!) but you want fine-grained representations, not just *multiple* representations.

Some quick thoughts: On why we gave the ColBERT paradigm the name "late interaction" instead of "multi-vector", a term that emerged later and that has proven to be more intuitive.

**The mechanism is actually not about having multiple vectors at all.** You can see this in four different ways.

🧵1/7

Btw the full general form to export all message templates is:

```
{name: my_adapter.format(p.signature, demos=p.demos, inputs={k: f'{{{k}}}' for k in p.signature.input_fields}) for name, p in your_program.named_predictors()}
```

The default Adapter is dspy.ChatAdapter().

But you can do all customization you mentioned with a custom Adapter:

class MyAdapter(dspy.Adapter):
def format(self, signature, demos, inputs):
return {"role": "user", "content": ...}

def parse(self, signature, completion):
return {....}

Thanks so much, Eric!

The piece you're looking for is the Adapters, but we should make give it nice syntactic sugar, so it feels native.

```
all_messages = {name: my_adapter.format(predictor.signature, demos=predictor.demos, inputs=dict(...)) for name, predictor in program.named_predictors()}
```

Tell me more?

Someone needs to write the book “Modern Machine Learning — the math, the myth, the legend”

literally an average joke

a statistician walks into an error bar

surprisingly, not everyone was mean to him

What do you call LLMs that exhibit reward hacking for translation?

Agents gone ROUGE.

Pytorch on cpu can be bad unfortunately; most of this gain on cpu (see ablations) comes from writing C++

Yup, but they’re related via a constant factor (eg at most 500)