Admission control layer for self-hosted LLM inference serving coding agents
Add an ingress decision layer in front of a self-hosted vLLM or SGLang engine that admits, defers or rejects each request before it takes any GPU memory. Coding agents fan out into dozens of LLM calls with retries in seconds, and every layer of a standard stack (agent, gateway, router, engine) only notices overload after saying yes, so capacity is consumed by requests that then miss latency targets. The layer reads both the request (input length, tenant, agent session, deadline) and live engine state (KV cache usage, queue age, cache hits, replica load) — a view no existing layer has.
online B2B Deep Tech / Infrastructure
From AI Engineer — Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP at 00:16
Problem: Overload buries the requests you needed first: a burst far above capacity makes the engine wait, swap or error, missing SLOs and wasting memory, because every admission decision today happens only once the request is already on the GPU
For: Teams hosting their own LLM on their own GPUs for privacy or cost reasons whose main traffic is coding agents or other high-fanout automated callers
Products from this video
Examples
- OpenAI's November 8, 2023 outage: routing servers ran out of memory under heavy traffic and failed health checks; the fix was cutting incoming traffic and ramping it back up slowly — graceful load shedding
Other takes on AI security infrastructure
- Agentic consent governance layer for autonomous AI systems
- Enterprise agentic last-mile identity and access control layer
- Identity-based security infrastructure for multi-agent AI systems
- Governed identity and policy control plane for AI agent actions
- AI-agent security and control infrastructure for enterprises
- A guardrail tool that enforces strict test-driven development and other engineering rules for AI coding agents