PerceptionBench

CommerceAgentBench is a benchmark developed and maintained by the Accio team at Alibaba International for evaluating whether AI agents can complete long-horizon business workflows in high-fidelity, stateful, reproducible replicas of online services. Its tasks cover browser operations, native-style command-line tools, file and spreadsheet production, API and MCP workflows, public-web research, supplier analysis, product publishing, logistics, storefront configuration, and other commerce operations. Each task runs in a fresh container and is graded by a deterministic or LLM-assisted verifier. The benchmark uses local mock services representing commerce, SaaS, messaging, document, and operational systems; runs preserve the resolved configuration, agent trajectory, verifier result, artifacts, logs, and container metadata for auditing and reproducibility. The repository describes 107 tasks spanning CLI, browser, file, and API/MCP interfaces, with text-only, browser-text-capable, and vision-required capability slices. The project was previously called RealReplicaBench.

GQA is a dataset for visual question answering and visual reasoning that provides grounded, compositional questions and answers over real-world images, built from scene graph annotations (derived from Visual Genome). It is used to train and evaluate models on compositional, spatial and relational reasoning in images.

ai-evaluation-framework is an open-source framework by Dreamers Inc. for benchmarking model-based solutions by accuracy, p95 latency, and estimated cost against user-defined ground-truth cases. Each task specifies fields and a scoring rule per field, including exact identifiers, normalized dates, numeric tolerances, containment, token overlap, and expected-missing values; cases can also provide alternate accepted renderings. It runs models from OpenAI, Anthropic, or an OpenAI-compatible gateway, compares results in a table, emits detailed JSON reports, and caches responses by model, task, and case so scoring rules can be changed without another model call. The repository uses document-field extraction as its worked example but accepts arbitrary text inputs such as OCR output, transcripts, and PDF text. A separate feedback server collects thumbs-up or thumbs-down responses with required comments, stores append-only JSONL data, and exposes statistics, exports, and candidate cases for human review; it does not train models. The repository includes tests using scripted mock models and is licensed under Apache-2.0.

Terminal Bench 4 is a benchmark for comparing coding-model performance across tasks with different difficulty levels.

Bespoke Nimble is an open-source model, training recipe, and local scoring library from Bespoke Labs for making typed decisions about text. Given a context and a flat schema of boolean or fixed-choice fields, it selects an allowed answer and returns candidate logits and probabilities without generating a reasoning explanation or free-form answer. Nimble fine-tunes Qwen3.5-9B with LoRA using contrastive training pairs: two nearly identical examples differ in one relevant fact, which changes the correct label. At inference, the scorer reads the logits for one-token candidate codes and converts them to probabilities with softmax; the Python library constructs the typed result rather than parsing generated JSON. The MLX ParallelScorer processes shared context once and scores fields in parallel, while the CUDA scorer scores each field separately. The repository includes the model recipe, curation and evaluation code, datasets, examples, tests, and deployment documentation. Bespoke-Nimble-9B can run locally on Apple Silicon or an NVIDIA BF16-capable GPU. It accepts text only, does not support nested schemas or answers outside the supplied choices, and its probabilities are not guarantees of correctness.