LangSmith

Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications. It ingests traces for LLM calls and related operations such as retrieval, embeddings, and agent actions, with integrations including OpenTelemetry, LangChain, the OpenAI SDK, LiteLLM, and other LLM frameworks. Its features include prompt management with version control and collaborative iteration, LLM-as-a-judge and code-based evaluations, user feedback and manual labeling, datasets for test sets and benchmarks, trace replay and debugging, metrics, and a playground for testing prompts and model configurations. Langfuse exposes APIs and typed Python and JavaScript/TypeScript SDKs for custom LLMOps workflows. It is available as a managed cloud service or for self-hosting with Docker Compose, virtual machines, Kubernetes via Helm, or Terraform templates.

LangChain is an open-source framework and agent engineering platform for building agents and applications powered by large language models. It chains interoperable components and third-party integrations, providing standard interfaces for models, embeddings, vector stores, retrievers, tools, and other data sources. It supports sequential agentic workflows, context and tool-call orchestration, model substitution, and application development primarily through Python; the project also provides a separate JavaScript/TypeScript library. LangChain can be used standalone or with related tools for agent orchestration, evaluation, observability, debugging, and deployment.

Context Hub is LangSmith’s remote context store for turning agent-run evidence into durable memory. It analyzes traces, filters them for useful signal, and promotes selected lessons into context such as instructions, skills, or Markdown policy files that the agent reads on later runs. It supports a read-write cycle in which an agent reads context, runs, captures evidence, and updates the stored context; procedural changes may be subject to human review.

AgentOps is an open-source observability and developer-tools platform for building, evaluating, and monitoring AI agents. Its Python SDK records agent sessions and LLM calls for dashboard-based session replays, step-by-step execution graphs, debugging, analytics, cost tracking, and benchmarking. Developers instrument code with decorators for sessions, agents, operations, tasks, and workflows; these support nested span hierarchies, input and output recording, exception handling, asynchronous functions, generators, and custom attributes. The SDK provides integrations with agent frameworks and LLM systems including CrewAI, AG2/AutoGen, Agno, LangGraph, and the OpenAI Agents SDK, and the platform can be self-hosted with its dashboard and API backend. The app is released under the MIT license.

LangFlow is an open-source, low-code visual builder for creating and running language-model pipelines, agent workflows, and retrieval-augmented generation (RAG) applications. The project is developed and maintained by the open-source community led by the GitHub user logaretm and integrates with language-model providers and tools such as LangChain.