pytest

Pydantic Evals is a code-first Python framework from Pydantic for evaluating LLM applications and agents. It organizes evaluations into datasets of cases, tasks, experiments, and evaluators: each case runs against a task, and evaluators score its outputs or tool-call trajectory using deterministic checks, custom logic, LLM judges, or span-based analysis of OpenTelemetry traces. Evaluations can run over defined datasets or sampled live production traffic, with results available in Python reports, the terminal, or Pydantic Logfire for visualization and comparison. The package is distributed as `pydantic-evals`; Logfire integration is optional.

Pyright is a Python typing-verification tool used as a local quality gate during software development.

Ruff is a Python linter and code formatter written in Rust and developed by Astral. It provides a common interface for linting and formatting, with built-in rules, automatic fixes such as removing unused imports, caching for unchanged files, pyproject.toml configuration, hierarchical configuration for monorepos, and editor integrations. Ruff is designed as a replacement for tools including Flake8, Black, isort, pydocstyle, pyupgrade, and autoflake, and supports Python 3.15.

spec-ptc is a Python library and demo implementing speculative programmatic tool calling (sPTC) for code-based tool-use harnesses such as Recursive Language Models (RLMs) and CodeAct. While a language model streams code for a REPL call, it identifies closed tool-call statements in the partial code, queues eligible calls as futures, and runs them in a shadow REPL so submodel or tool execution overlaps with continued code generation. When the completed code executes, speculative calls return stored results while non-speculatable or missed calls run normally; tools can be registered with decorators, including controls for speculation and purity. The repository also provides an out-of-process JSON-lines daemon with turn, token-feed, resolution, and cleanup operations, plus wrappers for Claude Code, OpenCode, and Pi-mono.

Reverify is an AI-agent verification toolkit for reverse engineering and related code analysis. It places deterministic tools between an agent and its claims: the model proposes hypotheses about a binary or candidate implementation, while parsers, disassemblers, pattern scanners, emulators, equivalence checks, and optional analysis engines compare them with ground truth and return evidence-backed VERIFIED, REFUTED, INCONCLUSIVE, OBSERVED, or INVALIDATED results. Its pure-Python core handles PE, ELF, and Mach-O parsing, x86/x64 and ARM/ARM64 disassembly, byte and pattern inspection, CPU micro-emulation, Protobuf/TLV dissection, and Frida hook generation; optional installations add Capstone, Unicorn, LIEF, Z3, or angr for deeper analysis. The project exposes the verification loop through a command-line interface and an MCP server that agents such as Claude Code and Cursor can call. It also records verified, observed, proved, and refuted results in a content-keyed local ledger, allowing grounded facts and negative findings to survive context resets, compaction, and new sessions. A rollover and orchestration system hands sessions off through files rather than model-written summaries, keeping verified facts separate from unverified notes. Reverify includes claim types for bytes, typed reads, instructions, patterns, emulation, behavioral or formal equivalence, imports, exports, sections, and—when angr is installed—functions, calls, references, and reachability. It is distributed from PyPI and licensed under the MIT License. The repository states that it is intended for authorized reverse engineering, malware analysis, CTF work, interoperability research, and software the user owns or is permitted to analyze.