Gimlet Labs

Gimlet Cloud is Gimlet Labs' serverless inference platform for AI agents and multi-agent systems. It can run agentic pipelines with custom logic, data sources, chained models, and non-model stages such as search; the platform manages scheduling, orchestration, and optimization. Its multi-silicon approach splits inference workloads into fine-grained pieces and maps them to suitable hardware, including Nvidia, AMD, AWS Trainium, Cerebras, D-Matrix, and Intel Gaudi systems. Gimlet describes this heterogeneous execution model as achieving 3–10× faster inference than homogeneous systems. The company also develops kforge, a multi-agent system that generates and checks optimized low-level kernels from PyTorch for CUDA, ROCm, and Metal backends.

KForge is an AI-assisted kernel-generation tool from Gimlet Labs that autonomously produces optimized low-level kernels from PyTorch for heterogeneous hardware. Its multi-agent system uses shared memory to explore kernel designs, enforce correctness checks, and identify fast implementations across CUDA, ROCm, and Metal, supporting both training and inference without requiring manual kernel authoring or changes to PyTorch. The system is described as using hardware and workload profiles to search optimization designs; in the cited video, Gimlet Labs reported a 33% end-to-end speedup for GPT-OSS 12B on AMD hardware in under a week, building on existing optimized kernels.

FriendliAI is an inference cloud and API for running frontier open-weight and custom AI models in production, with support for long-context inference, real-time streaming, tool calling, and agent workloads. Its stack combines custom GPU kernels, continuous batching, parallel inference, prefix caching with cache-aware routing, and a GPU-, CPU-, and NVMe-based KV-cache hierarchy for repetitive or long agent inputs. It also uses hybrid speculative decoding that adapts to live traffic, alongside multi-cloud scaling and autoscaling. The service provides model APIs and one-click deployment for models from Hugging Face, as well as deployment support for proprietary or fine-tuned models. FriendliAI advertises 99.99% uptime SLAs and reports performance claims including up to 7x faster output token speed and lower inference costs.

Nebius is a full-stack AI cloud infrastructure company offering purpose-built infrastructure for AI training and inference. Its platform provides non-virtualized GPU and InfiniBand capacity, scalable storage, self-service cluster provisioning, built-in MLOps tooling, and serverless and managed inference. Nebius also operates Nebius Token Factory for model serving; the company describes its infrastructure as covering the path from training through inference, with flexible consumption options and support for workloads ranging from experiments to global-scale deployments.

Agents That Own Their Inference is a workshop resource for building and operating production AI agents on dedicated GPUs. Its Jupyter notebooks, Kubernetes manifests, load-testing utilities, evaluation code, and Terraform setup guide participants through running a vLLM OpenAI-compatible endpoint and measuring inference behavior, including prefill and decode, time to first token, token generation time, KV-cache capacity, batching, queueing, and saturation. The workshop establishes a baseline, changes serving configurations, reruns realistic load tests, and evaluates output quality alongside throughput and latency. Exercises cover BF16-to-FP8 quantization, speculative decoding with a draft model, vLLM serving-policy changes, dense versus mixture-of-experts behavior, and deploying an agent on the measured inference endpoint. It can use an Akamai-hosted workshop namespace or a participant's Kubernetes cluster with an NVIDIA GPU; the repository is licensed under Apache-2.0.