AI product Open source
ASSERT is a local-first, framework-agnostic evaluation harness from Microsoft's Responsible AI organization for AI agents and LLM applications. It turns natural-language product requirements, policies, system prompts, or launch criteria into structured behavior categories and single- or multi-turn test cases, runs them against hosted models, callable wrappers, or OpenTelemetry-traced agents, and uses an LLM judge to score each conversation against the specified policies. The harness can capture tool calls, routing, model calls, and latency as trace evidence; it writes JSON and JSONL artifacts locally and includes a viewer for comparing runs and inspecting per-behavior results and judge citations. ASSERT provides a CLI, integrations for model endpoints through LiteLLM and agent systems through OpenInference, a library of behavior presets, and optional sandboxed action mediation for configured agents. It requires Python 3.11 or later and is distributed as an open-source GitHub project.
1 use taken from transcripts — each links to the moment in the video.
A new tool from Microsoft Research that generates evaluation judges and test sets from a written spec, bootstrapping custom evals for agents and scoring behavior across models.
1 in the library.