Langfuse CLI

Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications. It ingests traces for LLM calls and related operations such as retrieval, embeddings, and agent actions, with integrations including OpenTelemetry, LangChain, the OpenAI SDK, LiteLLM, and other LLM frameworks. Its features include prompt management with version control and collaborative iteration, LLM-as-a-judge and code-based evaluations, user feedback and manual labeling, datasets for test sets and benchmarks, trace replay and debugging, metrics, and a playground for testing prompts and model configurations. Langfuse exposes APIs and typed Python and JavaScript/TypeScript SDKs for custom LLMOps workflows. It is available as a managed cloud service or for self-hosting with Docker Compose, virtual machines, Kubernetes via Helm, or Terraform templates.

Langfuse skill is a ready-made coding-agent skill for managing Langfuse prompts, traces, and evaluations through natural-language instructions. Installed into agents such as Claude or Codex, it connects the agent to Langfuse so it can investigate production traces, identify issues such as unnecessary tool calls, retries, latency, and incomplete evaluation coverage, and propose approved changes. Langfuse describes the skill as part of its developer tooling alongside the Langfuse CLI and MCP server.

Langfuse Workshop is a step-by-step learning resource about applying Langfuse across the AI engineering loop. It uses a small TypeScript Dad IT Support Agent, named Specs, to teach tracing agent steps, managing prompts, monitoring production behavior, building datasets, running experiments, and evaluating changes. The workshop includes self-guided learner lessons, instructor notes, preparation materials, checkpoint branches, a runnable sample application, and helper scripts; its material is maintained in the public langfuse/langfuse-workshop repository. The sample agent uses an OpenAI tool-calling loop with two local tools, and the monitoring exercises cover user disagreement and all-caps frustration. The workshop is licensed under MIT and recommends Langfuse Cloud because self-hosted instances may not match the features used in the lessons.

LangFlow is an open-source, low-code visual builder for creating and running language-model pipelines, agent workflows, and retrieval-augmented generation (RAG) applications. The project is developed and maintained by the open-source community led by the GitHub user logaretm and integrates with language-model providers and tools such as LangChain.

LangSmith is an AI-agent observability and evaluation platform from LangChain. It captures agent traces, including tool calls, sub-agent calls, and retrieved context, so teams can inspect agent runs, debug behavior, and use traces as evidence in feedback and memory workflows. The LangChain site describes step-by-step tracing, trace-based dataset creation, evaluation of agent steps, complex-trace querying, online evaluations, dashboards, and alerts for production agent runs.