Vineeth Yeevani

YouTube video by Tesla This Robot Sucks

Is anyone else baffled by this? Why build this if you have the Optimus robot? Based on the teleoperation + current behavior cloning paradigm, it should be exceedingly trivial to learn the repetitive sequence of motions to handle cleaning this car. What am i missing?

m.youtube.com/watch?v=vVFX...

FSD not great in SF.
1. Doesn’t follow taxi + bus lanes
2. Doesn’t understand streets have a lane that’s sometimes parking sometimes driving
3. Always messes up left turns onto van ness
4. Always have to intervene for Valencia/market intersection

Also, the technique makes robotics more accessible while trading understandability. People find that useful in other tools (llm, gen image models). This makes me think this will be useful also.

I was wrong about this. My change of heart was inspired by dexgen (zhaohengyin.github.io/dexteritygen/). They use BC for course motion, RL for fine grained. Instant policy could just do both in a single shot. Instant policy would be great for tele-op

Deepseek r1 proves Yann LeCun’s thesis that models need physical grounding
Deepseek: good prior + easy verification = 🔥.
Good prior = video
easy verification = robotics in physical world

If time travel existed, wouldn’t we have met future us by now? Maybe it’s a beacon—you can only travel back to when it was first turned on. No paradox, just a switch we haven’t flipped yet.

Goal: vlm capable of planning + action
Problem: no data
Solution:
Bootstrap intelligence from vlm
1. Start with off the shelf vlm
2. Collect rollouts from code as policy from vlm for a set of tasks
3. GRPO over rollouts
4. Goto 2
5. Offline RL over vlm for direct obs->act

Let’s revisit code as policy in the deepseek-r1 era

Why is instant policy better than using point cloud matching/alignment approaches? Can we not just sprinkle genAI on everything without making a clear cut case for why scaling a method with DL is better than classic approach?

This btw is just regular old RL + curriculum learning. However, unlike control RL where we learn on the test set, here we have diff envs from train and test time.

Recipe.
1. Start with weak base + problems that range from really simple to really hard
3. Sort problems by how well model does on them
4. Pick 75% problems model can do, 25% it can’t. RL with GRPO. Use 10x rollouts + high temp on 25%
5. Repeat step 3 till all problems solved

Been seeing “competence threshold” take hold as an idea. High level: for a model to benefit from RL, it has to have some minimal competence. My view is, competence threshold is a function of the number of rollouts per question. Larger the number of rollouts, less competent the base model has to be.

Rerun is the best ml logger I’ve used. Miles better than wandb and tensorboard.

Turns out problem was too low resolution. Going higher res fixed the problem.

On second thought, rectified flow doesn’t help with this problem. Need to look for something different

The samples look like the mean of the input. Suspicion that the similar examples result in a nullcline in velocity when t -> 1. Rectified flow should fix this. But, I hate the multi stage process of reflow.

Anyone know how to improve sample diversity with flow matching when examples are really similar?

Why write clean code for ml? Took me ten min to convert a video diffusion training library to flow matching.

Pre-LN transformer layers don’t work for perceivers with fixed encodings

Declaring the death of model scaling is premature. Regardless of whether model scaling will continue, industry leaders’ flip flopping on this issue shows the folly of trusting their forecasts. They are not significantly better informed than the rest of us, and their narratives are heavily influenced by their vested interests. Inference scaling is real, and there is a lot of low-hanging fruit, which could lead to rapid capability increases in the short term. But in general, capability improvements from inference scaling will likely be both unpredictable and unevenly distributed among domains. The connection between capability improvements and AI’s social or economic impacts is extremely weak. The bottlenecks for impact are the pace of product development and the rate of adoption, not AI capabilities.

New AI Snake Oil essay: Last month the AI industry's narrative suddenly flipped — model scaling is dead, but "inference scaling" is taking over. This has left people outside AI confused. What changed? Is AI capability progress slowing? We look at the evidence. 🧵 www.aisnakeoil.com/p/is-ai-prog...

Keeping functionality in modular libraries helps everyone doesn’t have to rewrite everything. Least cohesion of modules is a great rule of thumb.

Next scale prediction demonstrates how to explicitly simulate CNNs using transformers.

That’s the best board I’ve ever see

A self driving car dataset has examples going left around a tree and right around a tree. The algorithm averages the two and goes straight into the tree. Hallucination.

The reward signal here would be if the user accepts some task completion or not. You’re fine tuning to improve the success rate of the agents taking into account the multi-turn decision making nature of agents.

I agree that you can do a lot with the foundation. But you could always do more. Offline RL techniques can make this happen. A good number of them make the mdp assumption.

My assumption here is that most ppl working on this are fine tuning to make agents. They have some reward dataset. Under POMDP, they could use a PPO library that works well in sequential decision making. With this, you avoid writing any alg code.

Engineering/product driven. Many existing decision making algorithms are based on POMDPs. It’s faster to use existing libraries to sketch promising agent use cases.

GitHub - vyeevani/easy-transformer-masks: Really flexible masks for transformers Really flexible masks for transformers. Contribute to vyeevani/easy-transformer-masks development by creating an account on GitHub.

It’s hard to make transformer masks for nested sequences - essential when working with perceivers or robotics. Wrote a library to help github.com/vyeevani/eas...

They still have a decision making casual structure - the future can’t influence the past.