DigitalOcean Droplet is a virtual compute instance in DigitalOcean’s Core Cloud offering. The service provides CPU and GPU instances for running applications and workloads; in the cited use, a Droplet hosts a model and runs an optimization agent.
Optuna is an open-source Python framework for automatic hyperparameter optimization, particularly for machine-learning and deep-learning workflows. It defines an objective function, uses a trial object to suggest parameter values, evaluates the resulting objective, and repeats the process through a study; in the described DigitalOcean agent, it applies this optimization loop to serving configurations, scores each trial, and adjusts the next recipe. Its eager search spaces support Python conditionals and loops, while its optimization algorithms can prune unpromising trials and run searches in parallel across threads or processes. Optuna is framework-agnostic and can be installed with pip; the site states that Python 3.10 or newer is supported.
tuner.py is DigitalOcean's optimization agent for finding vLLM serving configurations for a given model. It uses Bayesian optimization through Optuna to propose vLLM and Kubernetes settings, deploy trials on inference hardware, measure throughput under latency targets, and retain the best configuration or restore the previous setup. It produces an HTML results report and an optimized YAML serving recipe; the videos describe experiments involving GPT-OSS and DeepSeek R1 Distill, as well as planned extensions for disaggregated prefill and decode serving.
vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.
Searchable transcript of Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean — AI Engineer (05:33). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 All right. Hello everyone. So, my name is Rohan and this is a quick demo on the optimization agent that what has been recently built. So, a quick intro into what the purpose is and what the goal is of this agent and then I'll get into a brief demo and results that from a previous run that we just did. So, what the goal is here is to build an agent that helps find the best VLM serving config for any given model.
00:32 The purpose is to reduce the time it takes to manually find the best configs and the VLM serving recipe and if we can automate this and have it run in the background and have the agent find the recipes and the optimized parameters faster then it will save a lot of time. So, that's the goal of the agent and a little bit more into how the agent actually does this.
00:53 So, the strategy that the agent uses is called Bayesian optimization and Bayesian optimization is a technique that's used to find the most optimal numbers in any given situation. So, in our situation it's to find the optimal numbers in these VLM recipe. It uses mathematical steps that essentially minimizes the search space. So, by minimizing the search space it reduces a lot of the time that it takes in finding the optimized numbers in this recipe during the run.
01:27 So, BO Bayesian optimization is a strategy that is used but Optuna is the tool that uses a strategy in the search loop. So, Optuna uses Bayesian optimization. It feeds it the current recipe, the starting recipe. Bayesian optimization runs on that and over many iterations or many trials it eventually finds the optimized recipe and Optuna basically takes each recipe after every trial, assesses it, gets a score, uses a strategy to change the recipe, improve and then it reruns it within the uh search loop um until the
02:02 trials is done. So let's go a little bit into how we do this. So um currently in a droplet here and this droplet for example has GPT OSS 120B model uh loaded up. So what we can do is uh we can look at the current um recipe. So what we see here is four regimes, right? That we're testing it on. So there's chat short, there's chat long, there's rag and batch classify.
02:31 Uh they have their own set of ISO OSL tokens that we're feeding it through. Uh all these um can be changed depending on what uh we want to see. But essentially this is the starting config, starting recipe that you would get when you download the weights of the model. Um so moving on, once you have the model downloaded in your droplet and you are ready to run this agent, you paste in this command uh which essentially tells you you know where the where where the weights are coming from, um activate a virtual environment
03:05 and have um uh or not have but run the script called tuner.py. Uh this script essentially tells you or it it has a couple of things you have to tell it. So you you want to give it the name of the study. You want to tell it the starting YAML file which we just saw above here. You want to tell it the hardware that it's running on and you want to tell it the number of trials that you want.
03:27 So uh in this example we have 24. Um briefly I'll show you another example of a past run that was run uh that had only nine trials. So um anyway uh this is the command that's that's run. Uh typically these commands take um a couple hours to run. I think for the GPT OSS it took about four or five hours and then for the other model it took a couple of hours as well.
03:54 So we can go into the HTML report that's generated after the uh agent finishes its work. So this one is for the GPT-OSS model that I mentioned. Uh what what we see in the HTML report is which trial number was the best trial out of the total of how many trials. So in this case it was 50 trials. Uh you get so at the end of the run you get an optimized YAML file which includes all the numbers that we currently look for in the BLM serving recipe.
04:24 So in this case we found or the agent found that 256 was the optimized number so on and so forth. And the way that it's getting the accuracy is the amount of good put tokens that's being generated. So overall we found that from the first config we had an improvement of 17.5% in terms of tokens generated. And this one as I mentioned is for GPT-OSS. We have another HTML report here which is for the DeepSeek R1 Distill model.
04:50 In here we found that or in here we had nine trials that was run and out of the nine the ninth one was the best. Uh we found that token throughput doubled and ITL was improved by almost half aka it was reduced by almost half. So this is a quick intro and brief demo on the commands you run and what the output looks like. In in in general you get a HTML report that tells you um how each trial went and then you also get um a YAML file that has the optimized recipe at the end of the day.
05:24 So hopefully that was helpful. Thank you.