Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI Transcript, AI Summary & Key Points

00:12 Uh, welcome. Please grab a seat if you haven't already. My name is Vam. We have Jung Yao, Droo, and Dex. Before anything else, uh, a huge thank you to Sean and the AI engineers worldfare for having us. This is generally um a great room of AI and we're really really happy to be here. Um if you haven't been able to get set up yet, no worries. Uh we're going to have ample time to get it to to it later.

00:37 Um Junga will give us a brief background on the basics behind auto research. Okay. Um in case any of you still don't know uh auto research is this is just a quick brief. So auto research is basically a loop of a try and errors. You give a language model your code and a replayable evaluation script. It will keep trying ideas and test them against the eval and it keeps the ideas that improve the score and discard everything else.

01:24 So some example metrics includes accuracy, speed uh and cost. Similar loop has been exist uh in academic for a while like two years but it really got viral in this year when Kapasi uh posted about it in his model training scripts and turned out works really well for him. In just a few weeks, it just the use case of auto research spread from training to everything like a backend engineering, prompt and agent engineering, speed up model inference and it also quickly show up in production.

02:12 For example, it reduce the latency of Shopify's CI/CD pipeline and improve the browser use the company's browser use agent. In April, our experimental research agent Aiden even won OpenAI's hiring challenge uh called the prime golf uh which we will tell you a bit more soon and uh I also have a talk on Wednesday specifically about it. Um so okay auto research can be really powerful but it's also hard to master with a normal coding agent you basically steer um agent every single step but for auto research it's like the

03:02 agent is running in a loop on its own so it's much bigger multiplier on your prompt and your setup pointed at the right eval you basically let it boil the ocean for you, but otherwise you're just burning through all your RM tokens. Um so hopefully this workshop will give you a good starting point to learn about auto research and um apply that in your own work.

03:31 Okay, we'll come back to Bay. Thanks Janelle. Um so here's how we'll spend the next two hours. Um, Dex will take you inside parameter golf, uh, the challenge that we entered our agent into. Um, what that challenge was actually all about and how our agent ended up as the most influential contributor. Um, then it's your turn. Uh, we'll get you set up and kick off an auto research run on your laptop.

04:00 While your auto research is running in the background, we'll break down the basics of an auto research loop and the pieces that actually shape what an agent does. Um, and we'll go deeper and do a guided demo where you're able to run, steer, and inspect what your agent is doing live. And we'll finish with a fun part, which is a live competition. Um, you'll take everything that you learned today, point auto research at a game that we've all played before, and go head-to-head to see whose agent can get the highest score

04:34 uh, with the winner walking away with a really cool prize. a little bit more on who we are. Um, so we've been working on auto research for quite some time now. Before WICO, uh, we released aid in April 2024, which was an autonomous ML engineering agent. The basic idea was give it a machine learning problem, let it write code, run experiments, read the results, and keep improving.

05:04 When OpenAI built MLE Bench, aid was one of the first agents they evaluated independently and on that benchmark it scored about four times the medal rate of the next best agent. More recently, as we discussed, we pointed our agent to parameter golf and across more than a thousand research it researchers, it became the number one contributor. I'm going to hand it off to Dex who's going to tell us more about the competition.

05:39 Hello everyone, my name is Dex. Uh so I'm going to talk about the paramet golf. So paramet golf is open as uh model craft challenge. The goal is to train the best language model you can under two hard constraints. uh once the four artifacts including model weights plus training code has to fit in 16 megabytes and the training has to finish within 10 minutes.

06:04 Submissions are scored by bits per bpb on the fine wide validation set and the low is better. Openi provided a baseline data set and evaluation scripts so participants can for the ripple improve the model and submit their result through GitHub. So everything is open sourced. They also put up a million dollar in computer credits and reviewed every submission.

06:32 During the 44 day competition, there were,6 researchers entered and filed 248 PR roughly about uh 250k of GPU compute burn. The final lead board is short. Only 47 PRs merge as records from 31 contributors. And for this competition, we build Aiden, a self-improving multi-aging system based on the cloud and GPT model families that run indefinitely to help clamp on a single metric.

07:05 In permit golf, Aiden automated the four research loop. It pulls public page papers from archive, strong implementations from GitHub and its own part logs. Then propose new ideas, tests them, audit them for legality, keep the ones that work, write the implementation and run the parents. Once the results, clear the quality gates, it will create and submit a PR for review.

07:38 And the results are remarkable. Aiden landed seven mer leaderboard records. That's twice the next best human. It also had the highest OPR H index in the whole competition which means Aiden didn't just produce accept records. It published reusable work that other participants kept building on 435 downstream PR stated and built on Aiden's work and it got there on a fraction of the budget from just 4% of the community's GPU compute.

08:08 It got 15% of total merged records. That's 3.7 times more cost effective than the field average. And this wraps up a how Aiden becomes the number one contributor to the open herring challenge. Thank you. Cool. So, we're actually going to run auto research on a version of the same parameter golf challenge that Dex just walked you through. Uh, for future reference, we will just call this mini PG, which stands for mini parameter golf.

08:44 Um, as Dex discussed, the real parameter golf competition was heavy. Every single experiment was training a language model on a massive data set, which is fine when the competition is over four weeks, but in a workshop like today, you'd never actually be able to see your auto research run. So, think of mini PG as a scale model of the challenge. uh you're still going to train a real language model, score it on the same fixed evaluator, and drive down the same single number valp, which is essentially measuring how well

09:17 the model predicts text it hasn't seen. But what we did is we shrank this in scale. We're using a tiny data set instead of fine web and training an eval that only takes a few minutes. So you'll actually be able to get a loop going and see several steel several real steps live. So before I get to the instructions on the next slide, just a little bit of orientation.

09:41 Um you're going to run auto research yourself on mini parameter golf. You're going to sign up for two platforms if you haven't already. One is we our auto research agent that runs a search and the other is modal uh the compute your experiments train and evaluate on. Free credits. Thank you modal. Um then you will clone our parameter golf repo uh and hand it to your coding agent.

10:04 tell it to kick off a run. It handles the setup and it'll start the loop. Once it's going and it's running in the background, uh you can kick it off and we will check back in in around 15 or 20 minutes to take you inside the details. Instructions are here. Um sign up for WO on dashboard.ai. Um modal for moto modto.com-signup. Uh there's a code to claim your modal credits.

10:32 Um, if you already have a modal account, you can go directly to modal.com/crediteds and enter your code there. Um, and once you're done with those two steps, uh, clone the repo shown here, um, and tell your agent to set up mini PG from the readme and kick off a run. If you run into any issues or have any questions, feel free to scan the QR code and enter it there and we'll be monitoring that or uh, raise your hand and we'll come to you.

10:59 check back in in around 105 minutes. Thanks, guys. >> Uh, all right. So, sorry about the internet problem. I think it's being worked on, but until then, I think it would be good to maybe go through uh the anatomy of an auto research run. And we're going to be doing this in the context of uh WCO as the auto research system and mini PG as the task. But really, this this kind of contract that we have here is fairly general.

11:25 Most auto research systems are going to have you ask very similar questions and uh think about the same things. Okay. So the first thing is the source file. Uh this is what the agent actually has to modify. This defines the search space and where it can make changes. So this can be a single file, it can be it can be multiple files. In the case of minip it's uh this one file called solution.py and solution.py PI essentially defines the model config and the uh uh the training dynamics that's going to go on while the

12:01 model is training. And then the second thing and in my opinion this is probably the most crucial part of an auto research system is the evaluation. And so you want to have a very simple way of allowing your auto research system to evaluate every solution that it generates. Um, and you want to have it output a single scalar metric. And you want the eval to be fast, you want it to be cheap, you want it to have low noise.

12:32 Uh, because this essentially leads to your ability to run more steps if you have a fixed budget or a fixed time constraint. Um, and if there's low noise, then both you as well as the agent are more confident as you're kind of hill climbing on, uh, whatever task you're, uh, autoresarching on. And so, uh, one thing to be aware of though is agents love to reward hack.

12:58 So, they love to find holes in your evaluation protocol, holes in your benchmark. And so, you really want to figure out how to make your eval as robust as possible. The next two things are the metric and the goal. So you really want to ask yourself two questions. Um what do I want to improve and is higher or lower better? Um and in our case for mini PG, you want to minimize the validation BP.

13:26 So you essentially want to find a model that is better at predicting on unseen text. The next thing is the number of steps you actually run the auto research system for. This is a bit tricky because it's difficult to know a priority how many steps you should give your your system. We've chosen 20 in this case just because we want to give your agents enough time to kind of explore different ideas and you can you can see what the agent is actually exploring, what works, what doesn't.

13:58 Um but really more ideas that gets tried uh generally leads to more breakthroughs. And so uh this goes back to what I was saying earlier where if you have a fast eval um or if you have a cheap eval then when you're under some kind of time constraint or cost constraint, you can run more steps. Um and so uh again the eval is is always super important to uh consider.

14:30 All right. So you also want to bound the uh time for which your eval runs for uh because a lot of things can go wrong. So loops can stall, hardware can fail, uh sometimes the agent will produce really bad code that causes the entire system to hang. And so you want to bound the the actual evaluation running. Um in our case, we set it to be 210 seconds.

14:53 And the reason for that is we give 60 seconds for uh training the model, 60 seconds for testing the model on unseen text and then 90 seconds which is a buffer for you know modal to spin up the GPUs for training, spin it up for um uh for the testing and any other kinds of uh overheads that exist. And the last thing is the guidance that you give the agent at the start of your optimization.

15:26 Uh so this is how you steer the agent from the start. And you really want to think about it as you know you've probably run experiments before. You've compounded a bunch of domain expertise and human knowledge. And you might have some ideas about things that work, things that might be interesting to try and maybe things that don't work. And so this is where you can provide that kind of guidance.

15:49 Um in the case of WICO, you can simply point it to a markdown file, have all of your information and context in there. Um and the agent is able to use this and uh it gives the agent the best chance at uh at finding success. Um, in the case of mini PG, this can be uh some hard constraints or different ideas that seem to work quite well like uh focusing on compression uh different quantization methods uh maybe even weight sharing across different blocks in the model.

16:31 Um we are just going to demo how to get started on auto research for you considering the internet issues. Um before we get to that a little bit more about what we think um makes auto production grade auto research actually production grade. Um the first is structure. So uh we have a treeb brace search uh where the agent shows you every attempt it's run so that ideas as duv had mentioned can sort of compound on each other instead of starting from scratch.

17:03 The second is observability. Um you can sort of open any attempt read the exact code change behind its score so you can see why the agent is taking the approach it is. And the third thing is steerability. So you have a sense of control. You can inject your own ideas and knowledge while the loop is running and have it try new directions in case it's stuck.

17:29 A little bit more about the treebased search. Um we will shortly demo this on our auto research dashboard, but um treebased auto research essentially treats the problem as a structured optimization process over code. Uh it starts with the root of the tree. As you can see the score was 0.5 um which is just the baseline program and a measured score. Every node after that route is a modified version of the code run against your eval with a score attached.

17:56 You can think of this tree as an experiment graph. As you can see the search builds on what works. We're finding good candidates and pruning bad ones. So when a branch improves a score it builds on those ideas and goes deeper next. which means that you can inspect the exact changes your auto research system is making and build off of those. So the next idea we talked about was observability.

18:25 Um because you can see every one of the nodes in this treebased search, you can open any one of them and see exactly what changed. You can see your metric move in real time correlating to a movement correlating the movement to a reason. You can hover over over a node, get a summary of what changed, and you can open it and read the exact diff. So when the metric jumps, you can read the change that caused it.

18:50 And typically, these runs go for hours across hundreds of experiments. So once you can see what it's doing, the next natural step is to be able to change what it does. This is essentially when you stop watching and start to take over. Um, if I have an idea I want to try in plain English, um, I take the idea, it opens a new branch and you can see it try this new idea and have new nodes develop and a new curve.

19:20 I can also put a budget to each node so it spends compute where I want. And this is the part that made it usable on real problems because you might have context and intuition that an agent doesn't. At least for now, we believe it's important to find ways to fold human judgment into the system whenever it's necessary. I want to ground that point uh steerability in something real.

19:42 Um because parameter because in parameter golf steerability was a big part of why we did well. Um this is an actual breakthrough from the competition. It started with our agent uh which is the line in pink. um for five days from April 15th to April 19th. Uh it kept trying new things, um new attention, compression, a richer stack, but the score essentially plateaued.

20:08 There was a human in the community uh that shipped a clever new tokenizer, which is the blue line. Um, and within eight hours of emerging that change, our system picked it up, fused it with the pieces it had already been working on, and shipped the biggest jump in weeks. Running auto research on something this hard is less of a set it and forget it.

20:30 Uh, it's a collaboration where we steer the agent. We pull in ideas from the people around us. And steerability is essentially what makes that loop possible. It's how real breakthroughs actually happen. Um, I'm going to hand it over to Dex and Jungao to talk a little bit more about how to get auto research set up. Um, we'll essentially mimic the process that you guys were trying to do on your laptops.

20:56 >> Okay. Um, so what we are going to do here is we just walk you guys through the high level idea like how how does it happen? We just use our mobile hotspot. Um so we will be freestyling a little bit here. Uh hope you don't mind. Um so yeah can you just uh yeah okay maybe we can also walk through the whole doc being fed to uh claude. So >> yeah. So basically uh so you can go into the rebel and some like simple instruction like start the wo raw based on the readme and it should be like if the internet is good I think

21:44 it should start the uh default wle run and uh we can also go over via the like the how is the wle command working. Yeah, this case you have a source file. The solution is basically the training script of mini PG. I assume like a eval command is I think Z also mentioned this is additional guidance or prompt that will guide the whole uh search process and the metric is a validation BPB.

22:18 Uh this is like a hold out um uh loss like a loss only hold out set. Uh go is minimize and others are just configs we just mentioned. Oh, sorry. Eval commander is actually the the bash command that uh you're going to run the uh uh run the eval and eventually it will output a uh score like it print a score to the terminal and it's going to be passed by the agent and that's the key uh signal that will drive the whole tri and error loops.

23:05 >> Yeah. Um in meanwhile I can also show the dashboard. So if you go into the W dashboard um and I think there I did some wrong and you you basically see a lot of things and uh after the the command like auto booted by the agent um I think then you you will see the romping popped up and then you can try out the features we provide. Yeah. Um, do we have a run just to get started?

23:45 >> Yeah, I think it's just starting. >> Okay, still running saying um yeah, Claud is working. Uh question from the audience while you guys are doing that. Um one, is it possible to turn up the brightness on the dashboard or switch to light mode? >> Start again. >> I'm I'm right here. Can you turn the light mode on the dashboard? It's a little hard to see.

24:23 >> I see. Light mode. Forgot. Where is it? Is there uh second question is can auto research apply to RL? What is the scalar metric when there are multiple things to optimize? >> What is the >> can auto research apply to RL? What is the scalar metric when there are multiple things to optimize? >> I see. Yeah. Um it can definitely be applied to IO like apply to IO training scripts.

25:08 Um I think there is a um derived a project called sobound uh speedr runner that's actually exactly doing it like for post training aquin model uh for playing a game and uh yeah in that project you can apply auto research to the uh IR IR training scripts And um the second question you have a multiple objectives to optimize for. Um the idea is you eventually just combine them to a single metric like you either apply different weights on those different metrics or you treat some of the matrix as constraint and optimize

26:01 for one of them. For example, if you're doing things like fraud detection, you might hold the uh accuracy and trying to optimize the precision. Um and uh for parameter golf exactly you hold the model size as a constraint and you optimize for the accuracy of the system. Yeah, we can see the um agent already generated uh one node. So the zero zero node is basically the root node the baseline solution and 3.93 means the validation BPB attached to that solution and uh you can see the wos auto research already generated one

26:52 node here um um here we also have a summary of the uh ideas it's tried And uh seems like already getting improvement. Oh yeah, you can also >> also open the code and see what's actually been changed. >> Yeah. Do we have a stability? >> Okay. Uh for stability you can basically tell the agent for example say like uh explore the quantization method. Is it connected?

27:36 >> Yeah. Uhization method do not change the not just change the model config because we don't like just want to do the hyperparame tuning. We also want to try like more aggressive method. >> I don't think it's connected. I think clock didn't follow the instruction. Yeah, basically we also support you bring your claw code agent to our dashboard. We call it a bridged uh session.

28:26 Um so basically yeah you can you can see the dashboard the tree structure the metric and the uh clock code at the same time and then you can tear clock code okay this is the uh the direction I want to steer the whole uh search process to >> checked in with some of you I think internet might be a little better uh some of you guys are using hotspot I know there's some modal payment issues.

28:58 We're going to move on to the final part which is the competition aspect. Uh you don't need modal for this and originally this was designed to be a 30inut competition but we will give you guys until the end of the day um a little bit more on what this is. You guys have all played Flappy Bird. Um we're going to do a Flappy Wo Cup. So essentially, everyone hopefully has played Flappy Bird.

29:29 You tap to flap. You dodge the pipes and you fly as far as you can. And your final score is dependent on how far you fly. We have turned this into an auto research problem. So instead of playing it, there's a controller called should flap that decides every frame whether the bird should flap. Your job is to optimize that controller. The metric is the average score and you want that as high as possible with the highest score winning.

30:04 With all games, there's a twist. Uh we're not just running this on Flappy Bird. Uh we're running this across four different worlds. So the first world is normal, which is the classic game. The second is moving gap where the pipe gap drifts up and down. The third world has wind pushing the bird around. And the fourth world is variable which mixes all the conditions.

30:32 Your score is the average across all four of these worlds. So you can't just focus on one. Um a controller that does well in a normal game but dies in the wind won't win. And the leaderboard scores you on a hidden validation set that you never see. So you can't obviously overfit on the public game. A few rules uh mostly so it's fair and it doesn't break.

30:57 Uh you only and we will share the GitHub repo for this shortly. Uh you only edit the controller which is a controller.py. You keep you should keep should flap signature exactly as it is. So you can't touch the game or the evaluator. No imports, external libraries, file IO. So we're keeping this self-contained. And in the spirit of keeping it fair, uh, no parallel compute allowed.

31:23 So you would be using WICO for auto research. Um, everyone is defaulted to the same model, Opus 4.8. So it's a level playing field. And remember, the local score that you see on your laptop isn't going to match the leaderboard because of our hidden test set. Some tips. Um, you'll get about 50 steps on WCO. Uh, actually this is with the 30 minute time time frame.

31:53 So, uh, I guess you can run unlimited steps until the end of the day. Um, can start from the baseline. uh the baseline already works. So improve the code rather than rewriting the controller. Uh think about making small focused edits. Um things like tuning the safety margin, the velocity keyboard, the bottom buffer, adjusting the behavior per world and um maybe implement some of the features we talked about today.

32:24 Whether it's inspecting nodes that scored well, building on them, bringing in ideas from outside, um, and trying several directions. This is the instructions for the game. Um, by the way, you guys are playing for a Nintendo Switch. Uh the way that we're going to determine who wins is for your team name, put in your email and by the end of the day, the highest score uh we will reach out to the person that that is.

32:59 In the meantime, we're going to continue answering questions from our Slido or uh please raise your hand if you have any questions. Okay, we're just about to wrap up here. Um scores look like they're coming in. I don't know who Fern Nog is, but a little suspicious because there's only one node and so we will be inspecting your runs to ensure that there's no cheating going on.

33:24 Um, [laughter] nice try. Uh, we will be opening this competition uh till midnight tonight. um we will find a way to get to you can put in your email or we're also we also have your emails uh from the runs that you guys are doing or you can just contact us on our website. But um feel free feel free to keep going at it. And um thank you all for coming. >> [music]