AI product Open source
Tracely is a trace-native CI/CD platform for AI agents that turns production failures into regression tests. It ingests agent traces over OTLP, promotes agent and conversation metadata to indexed fields, applies structural checks and LLM-based evaluators, and clusters related failures into issues. A failing trace can be promoted to a hermetic regression case containing its input, tool outputs, and model outputs as fixtures; the case can then be replayed deterministically and offline in CI, with a fail-to-pass contract and a gate that returns a non-zero status and can block a pull request. The platform also provides trace exploration, multi-turn scenarios, alerts, trends, judge calibration, an SDK and CLI, a GitHub Action, and an MCP endpoint. It can be self-hosted with Docker Compose or deployed with the documented Railway and Kubernetes configurations. The repository is licensed under MIT.
1 use taken from transcripts — each links to the moment in the video.
Turns failed AI agent runs into regression tests you can run before merging changes. It grades production traces, groups similar failures into issues, and freezes bad runs with recorded model responses and tool outputs so tests are repeatable without fresh model calls.
1 in the library.