Azure AI Evaluation SDK

Microsoft Foundry is Microsoft's enterprise AI platform for building, grounding, deploying, evaluating, and governing AI applications and agents. It provides access to a broad selection of foundation, open, reasoning, multimodal, and industry-specific models, with tools for model customization, model routing, agent development, organizational grounding, business-system connections, and hosted deployment. Its lifecycle features include tracing, built-in evaluation for qualities such as relevance and groundedness, runtime security controls, and unified governance across agents, tools, and knowledge sources. In the cited workshop, models hosted on Foundry were accessed through the OpenAI SDK for scenario testing and evaluations.

ai-evaluation-framework is an open-source framework by Dreamers Inc. for benchmarking model-based solutions by accuracy, p95 latency, and estimated cost against user-defined ground-truth cases. Each task specifies fields and a scoring rule per field, including exact identifiers, normalized dates, numeric tolerances, containment, token overlap, and expected-missing values; cases can also provide alternate accepted renderings. It runs models from OpenAI, Anthropic, or an OpenAI-compatible gateway, compares results in a table, emits detailed JSON reports, and caches responses by model, task, and case so scoring rules can be changed without another model call. The repository uses document-field extraction as its worked example but accepts arbitrary text inputs such as OCR output, transcripts, and PDF text. A separate feedback server collects thumbs-up or thumbs-down responses with required comments, stores append-only JSONL data, and exposes statistics, exports, and candidate cases for human review; it does not train models. The repository includes tests using scripted mock models and is licensed under Apache-2.0.

DeepEval is an open-source LLM evaluation framework for testing and benchmarking LLM applications. It provides more than 50 plug-and-play metrics for evaluating AI agents, multi-turn chatbots, voice agents, retrieval-augmented generation systems, and other AI applications, and supports building evaluation pipelines for application-specific testing.

OpenAI Evals is an open-source framework for evaluating large language models and systems built with them, including tool-using agents and prompt chains. It includes a registry of benchmark evals, supports custom model-graded and private evals based on a user's data, and uses a completion-function protocol for advanced workflows. The package can be installed with pip and run locally with an OpenAI API key; the repository also documents optional result logging to Snowflake and configuration through the OpenAI Dashboard.

ASSERT is a local-first, framework-agnostic evaluation harness from Microsoft's Responsible AI organization for AI agents and LLM applications. It turns natural-language product requirements, policies, system prompts, or launch criteria into structured behavior categories and single- or multi-turn test cases, runs them against hosted models, callable wrappers, or OpenTelemetry-traced agents, and uses an LLM judge to score each conversation against the specified policies. The harness can capture tool calls, routing, model calls, and latency as trace evidence; it writes JSON and JSONL artifacts locally and includes a viewer for comparing runs and inspecting per-behavior results and judge citations. ASSERT provides a CLI, integrations for model endpoints through LiteLLM and agent systems through OpenInference, a library of behavior presets, and optional sandboxed action mediation for configured agents. It requires Python 3.11 or later and is distributed as an open-source GitHub project.