Strands Evals Kit

Strands Decider is a small decision model for agentic AI workflows, developed for use with the Strands Agents SDK. It selects among options, answers yes/no questions, or scores inputs against an ordered rubric, returning a calibrated confidence value for each decision. These capabilities support tasks such as model routing, tool selection, argument checking, triage, guardrail checks, and evaluation. The model uses a pretrained decoder LLM torso with its language-modeling head replaced by a pointer head. The head compares hidden states at the answer position and at each option's final token, scoring all options in one forward pass without text generation or a decoding loop. The same masked-softmax design supports its choice, noul, and score question types. The repository includes a command-line interface, a local HTTP server, an example that gates a Strands agent's tool call, and optional image inputs through a vision mode. It can run on CUDA, Apple silicon through MLX or MPS, and CPU. The project is distributed under the Apache License 2.0 and can be installed with pip.

Strands Agents Tools is a community-driven Python toolkit for the Strands Agents SDK, distributed as the `strands-agents-tools` package. It supplies agent-callable tools for file operations, shell and Python execution, HTTP requests, web search and crawling, browser and desktop automation, AWS services, image and video generation, memory backends, scheduling, MCP connections, and multi-agent patterns such as swarms, nested agents, graphs, handoffs, and workflows. Tools are loaded into Strands agents as callable functions, and the repository also supports dynamically loading custom tools and connecting to external MCP servers. The repository describes the tools as experimental and warns that several can execute code, access filesystems, call cloud APIs, connect to external servers, or control browsers and desktops; it recommends an independent security review, sandboxing, constrained permissions, and observability before production use. Installation uses pip, with optional dependency groups for selected tools. Several older tools are deprecated in favor of native Strands SDK capabilities or vendor-maintained MCP servers, and the repository states that it will eventually be archived. The project is licensed under Apache License 2.0.

OpenAI Evals is an open-source framework for evaluating large language models and systems built with them, including tool-using agents and prompt chains. It includes a registry of benchmark evals, supports custom model-graded and private evals based on a user's data, and uses a completion-function protocol for advanced workflows. The package can be installed with pip and run locally with an OpenAI API key; the repository also documents optional result logging to Snowflake and configuration through the OpenAI Dashboard.