Gimlet Cloud

Gimlet Labs is an applied research lab and AI-infrastructure company building systems for serving AI workloads. Its Gimlet Cloud provides serverless inference for AI agents, supporting multi-agent systems, custom logic and data sources, model chains with non-model stages such as search, and platform-managed scheduling, orchestration and optimization. The company describes a multi-silicon inference architecture that divides workloads into fine-grained pieces and maps them to suitable hardware, including GPUs and specialized accelerators. Its research also covers workload partitioning, heterogeneous-hardware compilation, dynamic datacenter scheduling, hybrid edge/cloud execution and cost-aware optimization. KForge is its multi-agent system for generating optimized low-level kernels directly from PyTorch, with correctness checks and support for CUDA, ROCm and Metal backends.

Modular Cloud is Modular's cloud layer for distributed AI inference, built on the company's unified stack of MAX serving and modeling APIs and Mojo GPU kernels. It provides hosted inference endpoints for open or custom models, with deployments available in Modular's managed cloud or in a customer's VPC; self-hosted MAX and Mojo containers can also run on customer infrastructure. The service is presented as a tunable system in which users can adjust variables across MAX and Mojo and examine performance and cost trade-offs in a cloud lab. Its hardware-agnostic stack targets GPUs, CPUs, and other accelerators, with OpenAI-compatible APIs, custom-kernel support, workload tuning, and observability. Modular describes shared endpoints, dedicated endpoints, and custom-model deployments with per-token or per-minute pricing.

Google Distributed Cloud is Google's on-premises and distributed cloud platform for running compute, networking, storage, and related infrastructure at a customer's site. It supports deploying trained AI models, Gemini, and AI agents while keeping data within the customer's environment.

FriendliAI is an inference cloud and API for running frontier open-weight and custom AI models in production, with support for long-context inference, real-time streaming, tool calling, and agent workloads. Its stack combines custom GPU kernels, continuous batching, parallel inference, prefix caching with cache-aware routing, and a GPU-, CPU-, and NVMe-based KV-cache hierarchy for repetitive or long agent inputs. It also uses hybrid speculative decoding that adapts to live traffic, alongside multi-cloud scaling and autoscaling. The service provides model APIs and one-click deployment for models from Hugging Face, as well as deployment support for proprietary or fine-tuned models. FriendliAI advertises 99.99% uptime SLAs and reports performance claims including up to 7x faster output token speed and lower inference costs.

Agents That Own Their Inference is a workshop resource for building and operating production AI agents on dedicated GPUs. Its Jupyter notebooks, Kubernetes manifests, load-testing utilities, evaluation code, and Terraform setup guide participants through running a vLLM OpenAI-compatible endpoint and measuring inference behavior, including prefill and decode, time to first token, token generation time, KV-cache capacity, batching, queueing, and saturation. The workshop establishes a baseline, changes serving configurations, reruns realistic load tests, and evaluates output quality alongside throughput and latency. Exercises cover BF16-to-FP8 quantization, speculative decoding with a draft model, vLLM serving-policy changes, dense versus mixture-of-experts behavior, and deploying an agent on the measured inference endpoint. It can use an Akamai-hosted workshop namespace or a participant's Kubernetes cluster with an NVIDIA GPU; the repository is licensed under Apache-2.0.