livenerf

FrontierHarness Eval is an open evaluation benchmark and repository for measuring how coding-agent harness configurations affect software-engineering task results when the underlying model and runtime are held constant. It publishes benchmark definitions, task instructions and metadata, harness-version metadata, normalized aggregate and task-level results, and an agent-neutral workflow for evaluating additional harnesses. The benchmark runs the same model through multiple harness configurations on 30 tasks, using a frozen golden checkpoint and a fresh restore for each task. Its scripts provision the environment, run trials, record verifier-based pass/fail outcomes, measure cost, cache behavior, and speed, then normalize results and generate charts and reports. The repository also provides a CLI and skill-based workflow for evaluating a third-party harness, with infrastructure failures marked separately from task failures. The project focuses on terminal-based software-engineering tasks and states that its results may not generalize to other knowledge-work domains. The repository is published under the frontier-harness-eval organization, with isolated runtimes and checkpoint restores provided by Runta.

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.

Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications. It ingests traces for LLM calls and related operations such as retrieval, embeddings, and agent actions, with integrations including OpenTelemetry, LangChain, the OpenAI SDK, LiteLLM, and other LLM frameworks. Its features include prompt management with version control and collaborative iteration, LLM-as-a-judge and code-based evaluations, user feedback and manual labeling, datasets for test sets and benchmarks, trace replay and debugging, metrics, and a playground for testing prompts and model configurations. Langfuse exposes APIs and typed Python and JavaScript/TypeScript SDKs for custom LLMOps workflows. It is available as a managed cloud service or for self-hosting with Docker Compose, virtual machines, Kubernetes via Helm, or Terraform templates.

ai-evaluation-framework is an open-source framework by Dreamers Inc. for benchmarking model-based solutions by accuracy, p95 latency, and estimated cost against user-defined ground-truth cases. Each task specifies fields and a scoring rule per field, including exact identifiers, normalized dates, numeric tolerances, containment, token overlap, and expected-missing values; cases can also provide alternate accepted renderings. It runs models from OpenAI, Anthropic, or an OpenAI-compatible gateway, compares results in a table, emits detailed JSON reports, and caches responses by model, task, and case so scoring rules can be changed without another model call. The repository uses document-field extraction as its worked example but accepts arbitrary text inputs such as OCR output, transcripts, and PDF text. A separate feedback server collects thumbs-up or thumbs-down responses with required comments, stores append-only JSONL data, and exposes statistics, exports, and candidate cases for human review; it does not train models. The repository includes tests using scripted mock models and is licensed under Apache-2.0.

LLM Space v4 is a local-first desktop workbench for prototyping and developing AI agents. It lets users version prompts, system messages, tools, and model settings; trace each model call and tool run in an agent loop; replay historical runs to debug failures; and evaluate agent performance across runs. It can generate prompts and tools with AI and turn a thread into a runnable LangGraph agent. Threads, project files, and API keys remain on the user's computer, while the project describes the app as cloud-ready for managed agents. The desktop application uses Electrobun with a React, Tailwind CSS, and shadcn/ui interface, and is built as a Bun monorepo around Pi Agent Core.