Langfuse

LangFlow is an open-source, low-code visual builder for creating and running language-model pipelines, agent workflows, and retrieval-augmented generation (RAG) applications. The project is developed and maintained by the open-source community led by the GitHub user logaretm and integrates with language-model providers and tools such as LangChain.

MLflow is an open-source AI engineering platform for agents, large language models, and machine-learning models, developed in the MLflow repository. It provides experiment tracking for models, parameters, metrics, and evaluation results, along with production observability, evaluation, prompt versioning and optimization, and model lifecycle tools. Its agent and LLM observability captures application traces using OpenTelemetry and supports different LLM providers and agent frameworks; its AI Gateway provides an OpenAI-compatible interface for routing requests, managing rate limits and credentials, handling fallbacks, controlling costs, and applying guardrails. MLflow can be run as a server and accessed through Python, TypeScript/JavaScript, Java, and other programming languages, with integrations including OpenTelemetry and MCP.

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

Lemmalog is a Datalog engine for LLM-agent memory, distributed as a Rust crate with an MCP server, REPL, and agent skill. It treats extracted statements as provenance-tracked base facts, then uses runtime-parsed stratified Datalog to derive temporal projections, closures, contradiction candidates, relevance relationships, and aggregates. The engine supports seminaive incremental fixpoint evaluation, stratified negation with negative-cycle rejection, bi-temporal facts, confidence and provenance annotations, proof trees through why(), scoped retraction recomputation, demand-driven ask_deep queries using magic sets, and persistence of episodes, rules, and base facts while rebuilding derived relations on load. Its AgentMemory facade connects an extractor to deterministic ADD, UPDATE, NOOP, and escalation policies, and assembles budgeted contexts from distilled facts and their source episodes. The MCP server exposes the engine to agent harnesses such as Claude Code and Kimi CLI through tools for observing facts, querying, explaining proofs, retrieving context, saving state, installing rules, and running hypotheticals.