InferenceX

Agents That Own Their Inference is a workshop resource for building and operating production AI agents on dedicated GPUs. Its Jupyter notebooks, Kubernetes manifests, load-testing utilities, evaluation code, and Terraform setup guide participants through running a vLLM OpenAI-compatible endpoint and measuring inference behavior, including prefill and decode, time to first token, token generation time, KV-cache capacity, batching, queueing, and saturation. The workshop establishes a baseline, changes serving configurations, reruns realistic load tests, and evaluates output quality alongside throughput and latency. Exercises cover BF16-to-FP8 quantization, speculative decoding with a draft model, vLLM serving-policy changes, dense versus mixture-of-experts behavior, and deploying an agent on the measured inference endpoint. It can use an Akamai-hosted workshop namespace or a participant's Kubernetes cluster with an NVIDIA GPU; the repository is licensed under Apache-2.0.

DeepInfer is DeepSeek's open-source, high-performance inference framework for offline and large-scale inference of DeepSeek models. It provides optimized kernels and model and sampler runtimes, attention data and context parallelism, mixture-of-experts expert parallelism with EPLB, speculative decoding through DSpark and confidence-scheduled verification, bounded replay for decoder prefill, asynchronous CPU batch scheduling, batch-invariant inference, deterministic sampling, and structured outputs constrained by regular expressions, JSON Schema, or EBNF grammars. The repository currently documents support for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It runs on Linux with Python 3.12 and requires CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The release does not include a KV-cache system, prefill/decode disaggregation, or load balancing across data-parallel replicas needed for a complete inference service. DeepInfer is released under the Apache License 2.0.

Gimlet Cloud is Gimlet Labs' serverless inference platform for AI agents and multi-agent systems. It can run agentic pipelines with custom logic, data sources, chained models, and non-model stages such as search; the platform manages scheduling, orchestration, and optimization. Its multi-silicon approach splits inference workloads into fine-grained pieces and maps them to suitable hardware, including Nvidia, AMD, AWS Trainium, Cerebras, D-Matrix, and Intel Gaudi systems. Gimlet describes this heterogeneous execution model as achieving 3–10× faster inference than homogeneous systems. The company also develops kforge, a multi-agent system that generates and checks optimized low-level kernels from PyTorch for CUDA, ROCm, and Metal backends.

A managed inference service from Red Hat for running enterprise AI inference workloads without directly selecting or managing GPUs. It provides usage-based scaling while abstracting underlying GPU capacity.

Baseten is an inference platform and company that provides tools to deploy, serve, and scale open-source and custom AI models in production. It offers managed infrastructure and tooling for model hosting, APIs, and inference at scale.