Marcel Hussing

Agreed and that is basically what section 4.1 in the paper I linked says. There is nuance to it for course but that part really needs to be step 1. However it is unclear how to start this discussion and our proposal is to turn the discussion into research.

Can we hop in general? A discussion of benchmark selection and design using the Hopper environment Empirical, benchmark-driven testing is a fundamental paradigm in the current RL community. While using off-the-shelf benchmarks in reinforcement learning (RL) research is a common practice, this choic...

What is a better benchmark? @cvoelcker.bsky.social and I wrote finding the frame paper about this in the hopes of starting a discussion around Benchmark selection. Benchmarks are often chosen via "we did what others did" where the starting point was picked arbitrarily. arxiv.org/abs/2410.08870

RL event at tacos el Gordo?

Yea that flavor is so good. I think my favorite has to be cabeza though. We should go sometime

Holy shit it's so good, I said the same thing. Which one's your favorite?

Getting ready to leave for NeurIPS tomorrow morning. ✈️ Let me know if you're around and wanna hang out!

I've always been confused by this term. The model isn't obtaining new knowledge. You are just exposing a different part of an already learned distribution. Maybe something like contextual generalization would make more sense to me.

You know what would be funny? If it comes back and the reviews aren't out yet.

a man wearing a suit and tie is standing in a field of yellow flowers ALT: a man wearing a suit and tie is standing in a field of yellow flowers

Me all day today

It's mind-boggling to my how many of the papers I reviewed don't cite a single paper that is older than like 2020. It's like we try to collectively forget what people did in the past so we can publish more.

Am at Columbia today giving a talk about (in part) this work. Have a few hours to kill afterwards. If anyone is around there and wants to chat, DM me.

Not sure there is a single good source. Maybe we should write one @cvoelcker.bsky.social

I don't necessarily think it's dull but one would need a conference where work like that can be published. Only TMLR comes to mind to some extent.

The cynic in me wants to say "because the paper needs to confuse the reviewer to get accepted" but I would of course never say that.

I also think it's not that they don't work but there were a lot of entangled problems that over the years have been addressed. I'm convinced that many of these things need to be restudied with our new algorithmic/architectural insights that simply make learning stable.

Yea 😂 we spent a lot of time on getting the exponents on the ridge regression small to avoid an explosion down the line but that worked out only semi well. 😅 I do think it's probably possible to get much smaller exponents but I suspect that will require a fundamentally different approach.

This should of course say quantizing Q-values 🤦

This was a fun collaboration between theory and practice with the theory group at Penn.

👩‍🎓👨‍🎓
@ericeaton.bsky.social
@mkearnsphilly.bsky.social
@aaroth.bsky.social
@sikatasengupta.bsky.social
@optimistsinc.bsky.social

(6/6)

We also empirically evaluate the algorithms. We first demonstrate that the sample complexity bounds are not representative of average case performance. Then, we derive insights for deep RL with discrete action spaces.

💡 Quantizing actions leads to agreement across policies! (5/6)

We build two objects near a reference point and apply randomized rounding. In ridge, the reference is the convex minimizer. With a Rademacher argument and uniform gradient convergence, this yields replicability. Also, the algorithm is replicable even if not fully accurate. (4/6)

In this work, we show that we can get replicability guarantees even in function approximation settings with RL. The idea is to ensure replicability of ridge regression and uncentered covariance estimation first. Then, use these tools in common approaches that solve linear MDPs. (3/6)

is motivated by the fact that in deep RL, variation from randomness can lead to drastically different solutions when executing the same algorithm twice. An algorithm is formally replicable, if (whp) it produces identical outcomes. I.e., run your algorithm twice and get the same policy twice. (2/6)

Replicable Reinforcement Learning with Linear Function Approximation Replication of experimental results has been a challenge faced by many scientific disciplines, including the field of machine learning. Recent work on the theory of machine learning has formalized rep...

I think I posted about it before but never with a thread. We recently put a new preprint on arxiv.

📖 Replicable Reinforcement Learning with Linear Function Approximation

🔗 arxiv.org/abs/2509.08660

In this paper, we study formal replicability in RL with linear function approximation. The... (1/6)

A large chunk of CS theory is ordering alphabetically, some even order randomly. Without any common standard, ideas like these are just gonna disadvantage people.

This one is so accurate it hurts my soul

Happy guy sad guy meme with sad text: USE PPO AND TUNE HYPERPARAMETER FOR WEEKS and happy text: USE REPPO AND GET A POLICY

I have been told I need to get more modern in my paper promotion! github.com/cvoelcker/reppo / arxiv.org/abs/2507.11019 @marcelhussing.bsky.social

Super stoked for the New York RL workshop tomorrow. Will be presenting 2 orals:
* Replicable Reinforcement Learning with Linear Function Approximation
* Relative Entropy Pathwise Policy Optimization

We already posted about the 2nd one (below), I'll get to talking about the first one in a bit here.

arxiv.org/abs/2207.04136 we always wondered how to discover the factored structure if not given. It's an intriguing question for which I have a few ideas but so far too little time.

(Maybe) unpopular opinion: There should not be *any* new experiments in a rebuttal. A rebuttal is for clarifications and incorrect statements in a review. You should not be allowed to add new content at that point. Either your paper is done or it isn't. It should not be written during rebuttals.

New ChatGPT data just dropped