Bindwidth

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.

CodeBurn is a free, open-source, local-first tool by AgentSeal that reads session files written by AI coding tools and reports token usage and estimated cost by provider, model, project, task, and activity. It provides a terminal dashboard and reports, a localhost web dashboard, desktop and tray or menubar views, exports, model comparisons, subscription-plan tracking, and optional cross-device aggregation. Its deterministic analyzers classify work into task categories from tool usage and message keywords, calculate token costs using LiteLLM pricing cached locally, and scan coding-agent sessions for patterns such as repeated file reads, low read-to-edit ratios, uncapped shell output, unused MCP servers, bloated configuration, and retry-heavy work. The optimize command produces estimated savings and fixes; applicable configuration changes are backed up and journaled so they can be undone and later compared with observed usage. The yield command heuristically correlates sessions with Git commits to classify spend as productive, reverted, abandoned, or ambiguous. The guard feature installs opt-in Claude Code hooks for soft and hard session-spending caps, checkpoints, and status-line reporting. CodeBurn also exposes local usage and savings through an MCP server over stdio. The CLI reads data from the local machine without wrappers, proxies, API keys, or uploads; the optional desktop applications can send anonymous bucketed telemetry after consent. It requires Node.js 22.13 or newer, and the repository is licensed under MIT.

ctrlb-decompose is a log-processing tool from ctrlb that converts large, repetitive log files into compact, LLM-ready patterns with typed variables, quantiles, anomaly information, severity scores, and example values. Its single-pass pipeline extracts and normalizes timestamps, applies CLP encoding to replace integers, floats, IP addresses, and strings with compact placeholders, clusters log types with the tree-based Drain3 algorithm, and merges wildcard positions with decoded values for semantic typing such as IPv4, UUID, duration, enum, and integer. It accumulates DDSketch quantiles, HyperLogLog cardinality estimates, top-k values, temporal buckets, and reservoir-sampled lines, and provides human-readable and LLM-optimization output modes. The project runs as a CLI, a Rust library, or client-side WebAssembly in the browser; its README states that logs remain on the local machine.

Needle 2 is an open-source, 45-million-parameter foundation model for tool calling, device use, and structured extraction on small devices. Its weights are compressed into a single 14 MB binary and integrated with its inference engine; the repository reports that a full session uses about 28 MB of RAM and performs inference without network access after setup. The model converts text and tool descriptions into schema-constrained JSON calls using a byte-level grammar, attaches a learned confidence score for threshold-based escalation, and can retrieve the top five tools from a larger catalogue. It uses a 256-token sliding window with tools pinned as key-value sinks to bound memory, and its Simple Attention Network combines a Hadamard MLP, grouped-query attention, engram key-value memory, and multi-lane hyper-connections. The Python package supports inference, LoRA fine-tuning, and checkpoint export; the inference engine is fetched from Hugging Face and cached, while the training stack is an optional installation.

CommerceAgentBench is a benchmark developed and maintained by the Accio team at Alibaba International for evaluating whether AI agents can complete long-horizon business workflows in high-fidelity, stateful, reproducible replicas of online services. Its tasks cover browser operations, native-style command-line tools, file and spreadsheet production, API and MCP workflows, public-web research, supplier analysis, product publishing, logistics, storefront configuration, and other commerce operations. Each task runs in a fresh container and is graded by a deterministic or LLM-assisted verifier. The benchmark uses local mock services representing commerce, SaaS, messaging, document, and operational systems; runs preserve the resolved configuration, agent trajectory, verifier result, artifacts, logs, and container metadata for auditing and reproducibility. The repository describes 107 tasks spanning CLI, browser, file, and API/MCP interfaces, with text-only, browser-text-capable, and vision-required capability slices. The project was previously called RealReplicaBench.