Inspect

iFixAi is an open-source tool for independently auditing AI agents against business, governance, safety, and operational requirements. It runs a diagnostic through a CLI, an agent plugin or skill, or an HTTP/provider adapter; the agent under test is evaluated with inspections covering tool authorization, privilege escalation, prompt injection, policy violations, private-data handling, uncertainty, failures, instruction following, and related risks. Runs produce JSON and Markdown reports with a scorecard based on five core pillars: fabrication, manipulation, deception, unpredictability, and opacity. The repository includes 60 inspections, configurable suites, self- or independently judged evaluation, fixture authoring, and support for testing a deployed agent or a bare model. The engine is free and distributed under the Apache 2.0 license; iFixAi Pro is a separate auditing service with additional inspections, reports, and an audit dashboard.

ai-evaluation-framework is an open-source framework by Dreamers Inc. for benchmarking model-based solutions by accuracy, p95 latency, and estimated cost against user-defined ground-truth cases. Each task specifies fields and a scoring rule per field, including exact identifiers, normalized dates, numeric tolerances, containment, token overlap, and expected-missing values; cases can also provide alternate accepted renderings. It runs models from OpenAI, Anthropic, or an OpenAI-compatible gateway, compares results in a table, emits detailed JSON reports, and caches responses by model, task, and case so scoring rules can be changed without another model call. The repository uses document-field extraction as its worked example but accepts arbitrary text inputs such as OCR output, transcripts, and PDF text. A separate feedback server collects thumbs-up or thumbs-down responses with required comments, stores append-only JSONL data, and exposes statistics, exports, and candidate cases for human review; it does not train models. The repository includes tests using scripted mock models and is licensed under Apache-2.0.

AIIC.engineer is an AI security engineering training website covering six workshop modules, quizzes, certification, and practical security challenges. Its curriculum covers prompt injection, indirect injection, sensitive information disclosure, supply-chain risks, excessive agency, shadow AI, AI bills of materials, threat modeling, MITRE ATLAS, and secure development practices. Workshop exercises include scanning a repository into an inventory of models, datasets, and agents; analyzing a vulnerable student assistant; restricting database access; validating outputs; scanning agent skills and MCP servers; and using scan-fix-rescan workflows with coding-agent rules and hooks. The training also addresses secure prompting, dependency verification, development and production boundaries, IDE and CLI configuration, developer review, governance, and offensive agent workflows.

Evo by Snyk is a security and governance platform for AI-native applications, agentic development, and AI systems. It provides continuous visibility, governance, testing, and real-time control across the AI software lifecycle. Its Agentic Development Security capability discovers models, agents, MCP servers, datasets, plugins, tools, and services used by agents across repositories; generates an AI bill of materials; applies policy and guardrails to agent actions; and validates AI-generated code before it reaches repositories, pipelines, or production. Its Continuous Offensive Security capability autonomously stress-tests applications and AI systems like an adversary, with agents coordinating reconnaissance, vulnerability hunting, exploit validation, remediation, and reporting, and an independent validation model intended to confirm exploitability before reporting findings. Testing is intended to run on code changes and pull requests.

Impeccable is an open-source design skill for AI coding agents, developed by pbakaus. It provides a shared design vocabulary with 23 commands, including init, shape, critique, audit, polish, harden, animate, typeset, layout, and live. The init command gathers product and design context into PRODUCT.md and DESIGN.md files, while other commands review, plan, modify, or refine a frontend project. Its CLI and browser extension run 61 deterministic detector rules for common AI-generated frontend design problems without an LLM or API key; it also supports LLM-based critique checks. The live command provides browser-based visual iteration, and the package can be installed from a project root with npx impeccable install.