Training your own agent models with reinforcement learning is now practical: the science has been figured out, teams with agents in production already have a harness, evals and real traces to build an RL environment, and a stable recipe (GRPO + masked importance sampling + score centering + top-p sampling with mask replay) makes async RL work even across heterogeneous GPUs, clouds and quantization levels.
Searchable transcript of Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal — AI Engineer (20:51). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] >> I'm Jason. I'm from Modal, and today I'm talking about scaling agent RL across the globe. But first, I think a lot of AI engineers view models as kind of like static backends that that they kind of swap out like as the new models come out. They'll maybe test the models, run evals, do AV AB tests, and kind of like rebuild their harness around whatever model is cheapest, but also highest quality for for their use cases.
00:49 And the claim I want to make for this talk is that this is actually a losing strategy when you end up having to compete on like not only the agent economics, but also the experience of the agent and like these two competing priorities of like offering differentiated value. And I think this is what we hear from our customers who do fine tuning. It's like the first the first area where I think fine tuning took off is is people who had agent deployments who need reliable SLOs.
01:24 Like like they they need reliability SLOs. If they if they need those, then they need to own their own inference. And the open source models out of the box were not always capable of supporting their deployments without, you know, degrading in in quality. And I think fine tuning, especially, you know, after these deployments became more mature, I think was the only path to owning your own inference.
01:50 But more importantly, kind of like pushing yourself up this curve in terms of latency, quality, [clears throat] and cost all together. Like if you if you do fine tuning, you like own all these yourself kind of. Um, there were other use cases though. I think one we heard which was interesting was proprietary data sets that their agents actually needed to reason about and understand and take action on, but uh these these data sets were like not present in any pre-training set.
02:18 And so they're like out of the box, sometimes they would they would build agents on that had to deal with this type of data and Claude GBT just like would not work at all uh for these tasks. And so they actually just had no other alternative but to fine tune open source models. And lastly, I think like there's an emerging attitude around fine tuning, which is um in order to build differentiated value with agents, uh like more and more people are deciding to vertically integrate across all the different levels of the
02:52 stack. Um, these are this is most common in industries where people are already competing on like some metric of task intelligence per dollar uh like coding. But I think um what we should see as the industry matures outside of just coding is like we should see this trend kind of continue elsewhere. Um, so I think that's that's why I think a lot of AI engineers have not really been fully bought in because the the latest model comes out and it's better than any model they could fine tune from the last generation, but I
03:25 think there's like a lot of different reasons people do fine tuning and in the long haul, I think uh yeah, this this is only going to become more common. So you may have heard of fine tuning in the context of RL or SFT and the question is why is this talk about RL and not SFT? Um, I think the idea that I want to communicate here is that uh if you're if you're doing SFT, you're basically training on traces recorded uh during production.
03:55 Um, but the problem is that agents in particular have this like branching structure uh based on the number of turns that they run. Um so in production, if you train on this particular trace, uh later on when you deploy that model, it may see like a completely different branch in the kind of like session like the the tree of possible sessions. And uh this is quite bad.
04:21 SFT is quite brittle in in this regard. So kind of like why RL has emerged as this like dominant paradigm of fine tuning is that um in RL the model actually explores all of these branches uh during training time. At least that's kind of the hope and when people talk about RL like generalizing better, that's often what they mean. I think one reason why this tradeoff has been hard for people is that they view like post training and fine tuning in RL as difficult.
04:56 Um but I think my claim here is actually that if you have an agent deployment in production, you probably already have the ingredients you need. Um so if you're in production, you have an agent harness that that's deployed somewhere and hopefully you have evals of some kind. Maybe calling it evals is not necessarily required. It could be AB tests, it could be um basically any any way you do change management for this harness, for this deployment.
05:21 Um but those two together give you the the tools you need to basically build out something that you can consider an environment, an RL environment. Um and why production traces are important for this is that you need to kind of get high quality tasks to train on in this environment from somewhere. And the best place often to do that is to synthetically generate these tasks from actual live production traces of things that your users are are doing um with with your product and yeah kind of forming uh finding examples
05:55 where it's failing, mining those and creating tasks around them so that fine tuning actually gives you some chance of improving your product in those areas. And this is basically all you need to do RL. But then the next part part about it being hard is that okay once just because you have an agent deployment doesn't mean that your team may be capable or maybe that's how you feel.
06:18 Um and probably because RL has a reputation of just being difficult generally. Um but I think over the last couple of at least over the last year over the last several months uh like we have kind of figured out the science of of RL. And that's what I'm going to basically talk about today is that uh a particular RL recipe that we've had a lot of success with um that I think simplifies and stabilizes a lot of what you know was difficult before.
06:51 But first okay just to kind of recap how how does reinforcement learning work? Uh it's it's this loop. Mechanically this is this is what's happening. You have a bunch of uh inference GPUs that are running your sampler aka the model that you're training. Um and those roll outs you you basically generate roll outs in your environment uh with your harness.
07:12 Those roll outs get collected and fed to a trainer which is consuming all this data and updating the model uh with you know gradient descent. And then uh that updated model you you broadcast those new weights to all the samplers and um over time you the roll outs you're generating are from this like continuously improving model. Uh so you're you're you're kind of like going back and forth between them and iterating and and that's how the model improves.
07:41 Uh so it turns out that like the math of RL prefers this to be done sequentially. So first you do your roll outs then you train and update your model then you you take that model update send it back to your roll out side so that the new roll outs you generate are from that new version of the model. Uh but the problem is that this wastes a ton of idle compute.
08:01 Um there are ways to like share the GPUs between the two but I think uh especially with agents um what's much better is that if you actually are continuously doing both in parallel, asynchronously. Um this is better because you can like pack your GPUs, get high utilization from them, and uh not waste money. But the downside is that um your rollouts that you generated were very likely from an old copy of the model.
08:28 And this is what people refer to as staleness in asynchronous RL. Basically uh rollouts were generated generated by V1. Uh the current model that you're training is V3. Are those rollouts still useful for V3 if the model's gotten better since then? Um and this is like one of the reasons why it was unstable. Another reason why it was unstable is actually purely at the systems level.
08:50 There's a lot of details that can show up as differences in between what the trainer thinks it's training and what the samplers think we're training. Uh basically at all these different levels of the stack, there can be these like small differences in behavior. And that means that like in in the math of RL, I think the the idea is that both both sides of this think they're doing the same thing.
09:15 They think they're learning from the same data, training like the same distribution. Um but actually it's like slightly different. And so the the solution that I'll present here is accounting for these differences um broadly with like kind of the simplest set of fixes that will correct for this mismatch without drowning out the signal of the the training loop.
09:38 So yeah, what's the recipe? Here it is. Uh we start from GRPO. This is uh what DeepSeek introduced for DeepSeek R1 and since then has become kind of a cornerstone of of doing RL on language models. Uh we add something called masked importance sampling. This is correcting for token-wise mismatch. And this mismatch is not just the mismatch from all the systems level stuff that I pointed out.
10:01 Uh it's also from the staleness as well. So if an old version of the model would have sampled a different token here, uh the ratio between the can be can be very different and then uh this is a mismatch that we want to correct for somehow. And and uh I won't go too much into details here about how these mechanisms work, but uh I will you know reference them.
10:21 Um and then the second is score centering. So even though we're correcting for each token-wise mismatch, what ends up happening is over the course of training whatever mismatch is still remaining accumulates and it kind of accumulates in the the weights themselves. Like the latest version of the model is slightly different than uh what it should be even though we've like corrected for the most obvious differences.
10:47 Um and so this score centering approach is it basically removes this drift over the course of training uh not just at each token or at each step. Um it it kind of like builds up a term that accumulates with the drift and removes it. Um and then the last thing is top P sampling uh with mask replay. So top P is is one of the parameters used when you sample from a language model.
11:08 Basically says only consider the top P of the of the distribution, P being like 0.98 0.95. Basically only the tokens that have probability mass greater than 95 uh or greater than 5%. Um and so it just removes a bunch of very rare likely bad tokens that can just never be sampled from. And so if your sampler decides to only sample from these tokens, but your trainer decides unluckily to sample one of these rare tokens, this mass mismatch can be basically catastrophic if you don't correct for it.
11:40 So we correct for it. Um it turns out actually in the math of score centering that if you're doing top P sampling like this, that actually the the technique we we use to do score centering uh is exact instead of approximate cuz if you don't do top P, it's actually only an approximate correction for drift. So all this is to say that uh with this recipe uh things get much much better and much more stable.
12:02 Um and this is a this basically just an ablation that that we did on Qwen 3.6 35B of the MoE. Um it's trained on the code environments with evals on Stable Bench Pro and the hard 51 subset. Um, basically what this shows is an ablation of these different techniques, uh, putting them all together showing like much more stable training, uh, than any of uh, the alternatives that we've tested here.
12:32 Um, so yeah, I think the that's all well and good, but I guess the question is like how far can we push this recipe? Does it actually scale? Does it Does it work for broad problems? Uh, before I talk about that, I think I want to talk briefly about why we care about RL at Modal and like some things about doing RL on Modal are different. I kind of want to just like get into that a little bit first.
12:53 Um, so yeah, first why Modal for agent RL in particular? Well, we do training. We just released uh, clusters as GA uh, last week I think or a week before. Um, we also do inference really well. We have bunches of endpoints. Uh, we've We've done a a lot of the hard work around doing LLM inference at scale. Um, and the environments you need they run in Modal sandboxes.
13:12 So like just naively we have all of the kind of infra primitive layers like or infra primitives that you'd want to be able to compose into like a full production agent RL training system. The problem The only problem actually, uh, cuz all that sounds great, but the only problem is that our capacity model's very different compared to other people. Um, the reason for that is we aggregate tons of capacity across many different providers.
13:39 Here there's three pictured, but in reality it's like 15 or 20 at this point. Um, we We run on all the hyperscalers, but we also run out a bunch of Neo Clouds. Um, and we handle all of the kind of like GPU health and reliability concerns. Uh, but this is great, but uh, it's great for reliability. It's great for kind of like fast and flexible scheduling.
14:00 It gets us around compute shortages. Uh, but it also fragments our compute pool. So if you want to schedule large training clusters, it's actually quite difficult in this like multi-provider multi-region fleet. Uh, we've engineered it so that we can schedule clusters pretty quickly up to like four nodes or eight nodes, uh but once you get past that to 16 nodes or 32 nodes, uh it gets pretty painful.
14:24 It takes a really long time to find like a contiguous block of all these nodes. Um and in RL this is particularly bad because in RL we generally have especially for agent RL, we have like two two inference GPUs for every training GPU. So two thirds ish of your cluster is actually just stuck doing inference. Um so if you're scheduling an eight node cluster max you're probably only using two or three of those nodes for for the trainer side and most of it's just spent doing inference.
14:54 Um it would be really great if we could move these inference GPUs out of the cluster. And one reason why that we think we can do this is because uh during RL the weight updates the actual like gradients that you're applying to your model end up being incredibly sparse and there's two reasons for that. But um yeah essentially if you if you take the model at step T and subtract the the new model T minus T plus one from it or sorry take the model at T plus one subtract T uh you get this delta and it turns out that even
15:26 though the full model weights at each step are very dense and very large uh the updates themselves are incredibly sparse. 100 X or 1,000 X uh as sparse. So you can actually just feed this to off the shelf compression and it turns out that the amount of data you need to communicate from the yeah from the trainer to the rollout engines goes down by a factor of 1,000.
15:50 This means that you can actually move your inference GPUs outside of the cluster. And so actually for the for the run that I mentioned uh of Clean 3.6 we ran the training cluster. I think it it landed somewhere in GCP but the inference GPUs actually ran all across our fleet and all the different hyperscalers and Neo Clouds that we run across. Um and it turns out with this like sparse delta compression technique you can do this.
16:13 Uh it works well and it does not introduce like bad performance. Um. Not only that, we we actually fragment our roll out fleet across different GPU types. So again, because the we have fragmentation due to all the different providers we run on, uh but we also have fragmentation because we have a lot of different GPUs that different customers want to use.
16:36 Uh it'd be really great if we could leverage as many of them as possible for our RL runs and that's that's what we've done. Um. Each of these GPU types they they increase the mismatch potentially. So if the trainer is running in V 200 uh but your roll outs are running in H 100 or A 100, there are hardware level differences in the the kernels, the math you can run that implement the model and so this like increases more and more mismatch which traditionally should cause your training runs to destabilize.
17:04 Um. This is just kind of like a graph to show uh how many requests are being processed by each GPU type in the run we did over the course of training. So actually the recipe that I mentioned uh is capable of handling this amount of mismatch that we throw at it, which uh yeah, I think is like kind of unprecedented. I think this just goes to show how much we've figured out the science of RL over the past year or two.
17:33 Um. Basically this shows that you have up to like four or five times as much mismatch uh but the training reward and gradient norm everything is is training as stably as it was uh in the homogeneous case which is basically running training on B 200s and inference on B 200s. And kind of a bonus like a side effect of this is that uh the roll outs that we run for this full training run are actually 15% cheaper.
18:03 And that's because the the demand for the high end GPUs like B 300s uh means that the price is just generally higher. But if you can actually run your roll outs cheaply on on cheaper hardware that there's less demand for, like the overall cost of doing this run is is significantly lower. This is actually without any inference tuning for each GPU type.
18:21 We're just doing like the basic off-the-shelf inference recipe for each GPU. If you actually decided to uh you know optimize each individual deployment for each GPU type with for example different levels of quantization in the model um you could actually probably make this significantly more cheap. We just uh this is not the focus of this work but uh but yeah.
18:44 Oh, I did also maybe I forgot to mention but in this case there are cases where like for example H200 we have two different pools of H200s. That's because we run two different uh quantizations of the model uh on H200s. So we have one H200 pool doing it at BF16 which is like the the safest way to do it but we have another running at FP8 uh and again like the the correction that we do can like drown out the mismatch that's introduced by even excessive quantization of the model and still learn from these quantized
19:13 rollouts. Um so yeah, I think the the point I'm trying to make here is that um like we've released this. There's a blog post that where we go into like all of the technical detail. Uh we'll be releasing the source code for it very soon. Uh it's based on Miles and SGLang the the reinforcement learning framework and the inference engine. And um I think the the the takeaway is that um I do believe that if you want to be competitive in a world where a bunch of different competitors are running agent stacks you do need to
19:43 to some degree own the full stack yourself or at least like build competency in every area of the stack. Um I think a lot of work traditionally ends at this agent harness but actually um the vertical integration across all of this is just extremely important in a in a hyper competitive uh industry. We see this again in certain industries like coding um but I think uh we we also see it in other industries and it's only going to keep growing I think.
20:13 Um and of course like if you're building agent deployments you own everything up the stack as well and vertically integrating across all of this is is important uh for maintaining kind of competency and and competitive edge. Um, yeah, you should own your own agent in RL. And it's actually possible now because the science has been figured out and there are very good tools and very good infrastructure that lets you do that.
20:37 Um, that's all I got. So, thanks. We're We're working on hard problems. Come visit our booth and and talk to us.