Russ Salakhutdinov

Our approach shows strong generalization and versatility in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney. It also enables easy editing and multi-concept prompt generation.

Prompt engineering for personalized image generation is labor-intensive, requires model-specific tuning, limiting generalization.

PRISM uses VLMs and iterative in-context learning to automatically generate effective, human-readable prompts using only black-box access to image generation models.

New work on automated prompt engineering for personalized text-to-image generation:

PRISM: Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

Paper + Code: kellyyutonghe.github.io/prism/

Carnegie Mellon University at ICLR 2025 CMU researchers are presenting 143 papers at the Thirteenth International Conference on Learning Representations (ICLR 2025), held from April 24 - 28 at the Singapore EXPO. Here is a quick overview of...

blog.ml.cmu.edu/2025/04/23/c...

Copilot Arena: A Platform for Code Figure 1. Copilot Arena is a VSCode extension that collects human preferences of code directly from developers.  As model capabilities improve, large language models (LLMs) are increasingly integra...

blog.ml.cmu.edu/2025/04/09/c...

Llama The open-source AI models you can fine-tune, distill and deploy anywhere. Choose from our collection of models: Llama 4 Maverick and Llama 4 Scout.

www.llama.com

Llama4 models are out! Open sourced! Check them out:

“Native multimodality, mixture-of-experts models, super long context windows, step changes in performance, and unparalleled efficiency. All in easy-to-deploy sizes custom fit for how you want to use it”

With small perturbations (less than 5% of total web page pixels), attackers can execute targeted adversarial goals with up to 67% success rates.

We also find that inference-time compute that is often used to improve model performance can introduce new vulnerabilities and harm robustness.

New work #ICLR2025 on “Dissecting Adversarial Robustness of Multimodal LM Agents” that shows that one can successfully break latest agents that use black-box frontier LLMs, including agents that perform reflection and tree search.

Paper + Code + Data: chenwu.io/attack-agent/

GenAI Summit 2025 #GenAIUCSD25

Excited to be at the GenAI Summit at UCSD!

I'll be sharing our latest work on VisualWebArena, inference-time tree search, and Internet-scale training of LLM Agents.

genaisummit2025.ucsd.edu

4/4 Llama 3.1 70B agents successfully complete 16.7% of tasks on 150k websites. Agents trained on human-annotated data from Mind2Web and WebLINX struggle to generalize to real-world websites. Adding synthetic data significantly improves generalization.

With B Trabucco, G Sigurdsson, R Piramuthu

3/4 Language models perform competitively with human annotators, achieving:
- 97% accuracy in detecting and filtering harmful content
- 89% success rate in generating feasible tasks
- 82% accuracy in judging successful task completions

2/4 The pipeline follows a three-step process:
- LLM generates tasks for 150k websites
- LLM agents complete these tasks and produce trajectories
- LLM reviews the trajectories and evaluates their success

1/4 New Work on InSTA: A pipeline for Internet-scale training of web agents across 150k diverse websites without human annotations.

Paper + Code: data-for-agents.github.io
Environment: github.com/data-for-age...

3/3 Joint work with Tiffani Min, Yue Wu, Jimin Sun, Max Kaufmann, Fahim Tajwar, Yonatan Bisk

2/3 Offline-collected state transitions are evaluated using PRMs to determine optimal intervention timing, creating labeled trajectories for training the helper model.

This minimizes costly intervention calls during training while leveraging PRMs to enhance robustness to off-policy data.

1/3 New work on Self-Regulation and Requesting Interventions: Enabling agents with a limited intervention budget to decide when to seek help:

Paper: soyeonm.github.io/self_reg/

We develop an offline framework that trains a helper policy to request interventions by combining LLM-based PRMs with RL

Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem Figure 1: Training models to optimize test-time compute and learn "how to discover" correct responses, as opposed to the traditional learning paradigm of learning "what answer" to output. The major...

blog.ml.cmu.edu/2025/01/08/o...

Inductive biases of neural network modularity in spatial navigation TL;DR: The brain may have evolved a modular architecture for daily tasks, with circuits featuring functionally specialized modules that match the task structure. We hypothesize that this architecture ...

blog.ml.cmu.edu/2025/01/02/i...

🌲 Ruslan Salakhutdinov (@rsalakhu.bsky.social) from CMU (@scsatcmu.bsky.social) opened the workshop with a talk on Tree Search for Language Model Agents.

Timestamp 36:20 in neurips.cc/virtual/2024...

📎 arxiv.org/abs/2407.01476

#NeurIPS2024 #AdaptiveFoundationModels

🎉 Had fun at #NeurIPS2024 Workshop on #AdaptiveFoundationModels!

🚀 Speakers: @rsalakhu.bsky.social @sedielem.bsky.social Kate Saenko, Matthias Bethge / @vishaalurao.bsky.social Minjoon Seo, Bing Liu, Tianqi Chen

🌐Posters: adaptive-foundation-models.org/papers

🎬 neurips.cc/virtual/2024...

🧵Recap!

With my amazing students and collaborators at @neuripsconf.bsky.social in Vancouver!

ScribeAgent: Fine-Tuning Open-Source LLMs for Enhanced Web Navigation TL;DR: LLM web agents are designed to predict a sequence of actions to complete a user-specified task. Most existing agents are built on top of general-purpose, proprietary models like GPT-4 and rely ...

blog.ml.cmu.edu/2024/12/06/s...

Carnegie Mellon University at NeurIPS 2024 – Machine Learning Blog | ML@CMU | Carnegie Mellon University

Carnegie Mellon University is proud to present 194 papers at the 38th conference on Neural Information Processing Systems (NeurIPS 2024)

blog.ml.cmu.edu/2024/12/02/c...

2/2 Our findings show that even when unlearning a single fact, current methods either fail to properly unlearn with high recall or end up unlearning many other irrelevant facts.

Paper: arxiv.org/abs/2410.15153
Code+Dataset: github.com/wrh14/deep_u...

joint work R Wu, C Yadav, K Chaudhuri.

Evaluating Deep Unlearning in Large Language Models Machine unlearning is a key requirement of many data protection regulations such as GDPR. Prior work on unlearning has mostly considered superficial unlearning tasks where a single or a few related pi...

1/2 New work on Evaluating Deep Unlearning in Large Language Models.

Paper: arxiv.org/abs/2410.15153

Unlearning specific facts in LLMs is challenging because the facts in LLMs can be deduced from each other. This work proposes a framework for deep unlearning of facts that are interrelated.

What is happening?! Who is this? 😆

Hello BlueSky