Best LLM Observability Tools

After the October 1, 2026 documentation review, Langfuse remains our pick when traces, datasets, evaluations, and open-source self-hosting need to live in one product. LangSmith is our first trial for an existing LangChain or LangGraph application.

Phoenix fits self-hosted OpenTelemetry. Braintrust centers experiments. Helicone centers gateway monitoring. Datadog now documents a dataset-to-experiment loop and standalone access, alongside its fit for an estate you already monitor.

We reached that verdict from first-party documentation, not from running the six tools. We do not rank trace latency, ingestion throughput, evaluator accuracy, or support quality.

This is the observability and evaluation layer of the AI App Stack. No partner or sponsored links appear in this roundup. The dated review covers the workflow, deployment, and billing boundaries discussed below, rather than every feature or plan entitlement.

Updated October 1, 2026: corrected Langfuse v4 availability, Datadog's experiments and standalone access, Helicone's OSS deployment route, and Phoenix's telemetry boundary. Added a trace-to-test example using authored synthetic controls. The original publication date is unchanged.

A trace without a test is a record of bad answers

A trace answers what happened. An evaluation answers whether the result was acceptable. A dataset makes that judgment repeatable, and an experiment tells you whether a prompt, model, retrieval step, or tool change improved the same cases. Buy a viewer and skip that loop, and you own a precise record of bad answers.

We checked five decision boundaries in each product's first-party documentation:

Table 1 Download CSV

Decision	What we looked for	Why it changes the pick
Trace model	Nested model, retrieval, tool, and custom spans	A flat request log cannot explain an agent path
Evaluation loop	Offline datasets, production scores, human review, and experiment comparison	Visibility alone does not catch a quality regression
Instrumentation	SDKs, framework hooks, proxy paths, and OpenTelemetry support	Portable instrumentation lowers switching cost
Data boundary	Hosted, open-source, hybrid, or self-hosted deployment	Prompts and tool results can contain sensitive data
Cost shape	Seats, traces, observations, requests, data, scores, and infrastructure	The cheap starting plan may not match production volume

The sources establish available features and published limits. They do not establish which interface is easier under incident pressure, which collector drops fewer spans at load, or which evaluator agrees most often with a domain expert. We would need the same instrumented application and labeled cases inside every product to make those claims.

Documentation is all we compared

Table 2 Download CSV

Tool	Best for	Ingestion path	Evaluation loop	Deployment boundary	Cost shape	Free starting route	Where it loses
Langfuse	Integrated tracing, prompts, datasets, and evals with cloud-to-OSS portability	OpenTelemetry-based Python and JS/TS SDKs; OTLP	Offline experiments plus code, model, API, and human scores	Cloud or MIT-licensed self-hosted core	Traces + observations + scores as billable units in Cloud	Hobby Cloud plan or self-hosted OSS	Version compatibility and self-hosted operations need active ownership
LangSmith	LangChain and LangGraph applications	Framework integrations, manual tracing, SDKs, and OpenTelemetry	Versioned datasets, offline experiments, and online evaluators	SaaS; Enterprise self-hosted and hybrid options	Seats, trace storage/retention, and platform compute	Single-seat Developer plan	Self-hosting is an Enterprise add-on; test the non-framework code path
Phoenix	OpenTelemetry-first teams that want the full data path in their infrastructure	OTLP with OpenInference instrumentation	Code, model, and human evals over traces, datasets, and experiments	Open-source Docker, Kubernetes, or cloud deployment	Your infrastructure for OSS; separate hosted Arize plans	Self-hosted open source	One Phoenix instance is one tenant; production operations remain yours
Braintrust	A production-to-evaluation feedback loop	Braintrust SDKs and an OTLP endpoint	Logs and experiments share a structure; offline and online scoring	Hosted or Enterprise customer-run data plane with a managed control plane	Processed GB, scores, model usage, and retention	Starter plan	The control plane remains managed in the documented self-hosted option
Helicone	Request, session, cost, user, and gateway monitoring	AI gateway, proxy, SDK, or logging integrations	Datasets and externally produced scores	Hosted, OSS self-hosting, and commercial on-premises offerings	Plan plus requests and storage	Hobby plan or self-hosted OSS	Its score docs assign evaluation execution to your framework or workers
Datadog	Model tracing and experiments beside an existing Datadog estate, or standalone	SDKs, HTTP API, and OpenTelemetry	Managed and external evaluations; versioned datasets and offline Experiments	Hosted in a supported Datadog site	LLM spans ingested, with retention options	Free tier	Check site availability, retention, and the intended ingestion path

We kept exact dollar figures and quotas out of the table because they change faster than the product boundary. Follow the linked Langfuse, LangSmith, Arize, Braintrust, Helicone, and Datadog pricing pages for the current numbers.

Langfuse is the integrated open path

Langfuse puts tracing, prompt management, datasets, experiments, and several evaluation methods in one product. Its evaluation concepts connect offline experiments to online scoring: a production failure can become a dataset item, and a later experiment can run the revised application against that case. Code evaluators, LLM-as-a-judge, API scores, UI scores, and annotation queues all write into the score model.

The deployment boundary is the reason it leads this list. Langfuse publishes its core product under the MIT license and documents self-hosted pricing separately from Cloud. Cloud billing counts traces, observations, and scores as billable units, which is clearer than pretending one request always equals one stored object.

Open source removes a core license bill, but the team still owns databases, storage, upgrades, and backups. The current compatibility docs list self-hosted v4 as generally available. Cloud runs v3 and v4 side by side until its published November 16, 2026 cutover. Check server, SDK, and API compatibility before upgrading: the Cloud removal schedule and a self-hosted migration are different deployment paths.

Langfuse is our default shortlist when the team wants one evaluation loop and a route between managed Cloud and self-hosting. If operating that infrastructure is off the table, compare its managed plan with the other hosted options rather than counting OSS as a cost saving.

LangSmith is the LangChain-native pick

LangSmith documents automatic tracing integrations for LangChain and LangGraph. Its observability model groups runs into traces and supports manual instrumentation, tags, and metadata. The same platform adds versioned datasets and both offline and online evaluation.

That integration is why we would trial it first for an existing LangChain application. A failing production run can move into a dataset, receive code or model-based evaluators, and appear beside later experiments. OpenTelemetry ingestion is also available. Include the application's non-framework code in the trial so an automatic integration doesn't hide a missing span.

Where it loses: the hosted starting path combines plan, seat, and usage decisions, while self-hosted LangSmith is an enterprise deployment with PostgreSQL, Redis, ClickHouse, and supporting services. It is not the same proposition as installing an MIT-licensed tool and owning every layer. For a small framework-neutral service, that can be more platform than the job needs.

Choose LangSmith when existing LangChain or LangGraph integration removes work you would otherwise build. Do not choose it merely because an agent tutorial already has the tracing environment variable in its final step.

Phoenix keeps the stack in your infrastructure

Phoenix is built on OpenTelemetry and OpenInference, and its current product overview covers traces, evaluations, prompt work, datasets, and experiments. It accepts model, retrieval, tool, and custom spans, then lets code, model-based, or human evaluators attach scores to traces and experiment results.

Its deployment architecture describes a web application, trace collector, and SQL backend. SQLite fits a local single-user start. PostgreSQL is the recommended production option. Docker and Kubernetes guides are public. The privacy docs say self-hosted application data stays in your infrastructure, while default UI analytics are separate. They document telemetry and external-resource opt-outs; review those controls before treating a deployment as isolated.

Where it needs operating work: a Phoenix instance represents one tenant. The current architecture describes separate instances or PostgreSQL schemas for stronger team and environment isolation, so a large organization must design that boundary. Arize AX is a separate managed platform; check its plan boundaries if you need managed operations.

Phoenix is our pick when keeping trace data in a private OpenTelemetry stack is the first requirement and the team is prepared to run it. It is not a free managed service disguised as a container.

Braintrust makes evaluation the product

Braintrust treats production observation and pre-release evaluation as one data problem. Its observability documentation says logs use the same structure as experiments, so teams can filter production traces, apply feedback, and turn selected failures into targeted datasets. The evaluation workflow covers offline experiments, online scoring, playground work, and regression checks.

This is the strongest fit when the organization's main question is not "where did the tokens go?" but "did this change improve the task?" Braintrust supports SDK instrumentation and OpenTelemetry export, which makes it possible to keep a broader trace graph while using the same spans in evaluation work.

Where it loses: its self-hosted deployment is a customer-run data plane, not a fully customer-run product. Sensitive traces, datasets, prompts, and experiment data stay in that plane, while Braintrust continues to provide the UI, authentication, metadata, and platform updates through its control plane. That split can satisfy a data-residency requirement, but it does not satisfy every full-isolation or open-source requirement.

Braintrust belongs on the shortlist when experiments and production feedback already drive releases. If the team mainly needs a request gateway and cost report, it starts too far down the evaluation stack.

Helicone is a request path, not a judge

Helicone's center of gravity is the request. Its gateway and logging paths expose models, tokens, costs, errors, users, custom properties, and sessions. The sessions documentation groups LLM calls, vector queries, tools, and other logged requests into a trace-like hierarchy, which is useful for multi-step agents that already pass through its gateway.

It also supports datasets and score storage, but the boundary is explicit. The current score documentation assigns evaluation execution to your framework or workers. Helicone stores and compares submitted scores. Budget the execution and judges separately rather than inferring them from a score column.

Helicone also publishes an OSS Docker self-hosting guide. For proxied model calls, it describes running the AI Gateway alongside Jawn. The commercial on-premises offer on its pricing page is a separate option. This is a documentation check, not a deployment we ran.

Its documented gateway path suits gateway-first monitoring. A team that wants per-user cost, request filtering, rate limits, caching, provider fallbacks, and session inspection can get those concerns close to the API path. Where it loses is the extra seam between production traces and repeatable evals.

Use Helicone when the gateway is the product boundary you want to observe. Do not infer an evaluator from a score column.

Datadog joins tracing and experiments in the same platform

Datadog Agent Observability brings model spans into the platform used for service and infrastructure monitoring. For an existing Datadog team, that makes a shared incident timeline a reason to trial it.

A latency alarm can come from a slow model call, a retrieval database, or a deployment elsewhere in the request path. The trial should establish whether those spans correlate correctly in your application. Documentation cannot establish how quickly your team will isolate the failure.

Datadog documents managed, custom, and external evaluations, plus Experiments with versioned datasets and result comparison. Its dataset docs support production traces, CSV files, and manual entries. The dataset-to-experiment loop belongs in this comparison; we have not tested its runtime quality against the specialists.

Confirm that those features are available in your Datadog site. The current Experiments and Evaluations docs exclude the listed government sites. Check retention and export needs before committing your regression cases to any hosted platform.

The current pricing page offers a standalone product and a free tier. Another Datadog subscription is not required. Billing counts ingested LLM spans, so one agent workflow can produce several billable calls. The benefit of an existing estate is shared context, rather than a procurement prerequisite.

Datadog belongs on the shortlist for shared model and service traces. Its documented experiment loop also warrants a trial for standalone use. Compare it with the same application and acceptance rules you use for the other finalists.

Turn one failure into a test you can rerun

Keep the failed input, write down the expected answer, and check the next output against that rule. A trace supplies the context for choosing the case. It doesn't supply the answer key.

The support-extraction kit makes that distinction inspectable. It contains authored support messages, a JSON schema, and separate task assertions. We used case-01 and the deliberately wrong neg-07 control here. Neither output is a model response or a captured production trace.

Synthetic example. The input is:

ticket_id: EX-001
message: Billing issue. This is urgent. Please cancel the duplicate charge.

Table 3 Download CSV

Field	Expected answer	Authored negative output
ticket_id	EX-001	EX-999
category	billing	billing
urgency	urgent	urgent
requested_action	Please cancel the duplicate charge.	Please cancel the duplicate charge.

The local assertion script returns Schema passed · Task check failed. EX-999 is a valid string in the schema's ticket-ID format, but it doesn't copy the input's EX-001. The other fields match. That result verifies the authored control and the check, not a model's reliability, a vendor integration, or a production failure rate.

The diagram's first path starts with an actual trace, when one exists, then redaction, expected-answer review, and offline replay. Its second path is this synthetic assertion demonstration. Its third path checks a provider notice against a serving route. We captured no production trace for this example, and no notice explains its deliberately wrong ID.

For an actual failed request, follow the same handoff:

1. Preserve the trace ID, input, output, exact API model ID, serving route, prompt and application versions, timings, and errors. Include retrieved document IDs or tool arguments when they affect the answer.
2. Redact secrets and personal data before moving the case. Keep the relationship that caused the failure: replacing both ticket IDs with the same placeholder would erase this mismatch.
3. Have a reviewer write the expected result from the task contract. Record the reason. Copying the failed output into the expected-answer field would turn the defect into a passing test.
4. Freeze the case, schema, assertions, and prompt configuration. Run the baseline and proposed change against the same versions, retaining failures and the declared retry policy.
5. Inspect the fields that failed, alongside successful cases. A fix for ticket copying still needs the ambiguity and instruction-noise cases covered by the workload testing guide.

Langfuse's documented dataset workflow supports inputs and expected outputs, CSV or SDK creation, and adding recorded observations to a dataset. That is a representative route from trace to case, rather than an integration we ran. A dataset item can retain its source reference while the expected answer is reviewed separately.

Versioning needs one extra check. Langfuse documents timestamped dataset-item versions, but schema changes do not create a dataset version. Record schema and assertion hashes separately, along with the dataset version and application configuration. Otherwise an improved score could come from changing the acceptance rule instead of changing the application.

Online scoring checks incoming production outputs. Offline replay runs a frozen case set through a chosen configuration. Keep their counts separate: a sample of live traffic and a curated regression set answer different questions. The synthetic control above proves only that the checker rejects a known wrong answer.

Data boundaries include the evaluators

A self-hosted trace store doesn't settle where every copy goes. Review the exporter, dataset storage, backups, UI analytics, and the evaluator. Sending a prompt to a hosted judge sends data outside your infrastructure even when the trace database stays local. Decide what to redact before that call and which reviewers can see the unredacted source.

Costs also follow stored objects and scoring work. As an illustrative calculation, one request with one trace, five observations, and two scores is eight Langfuse billable units. Those are authored counts, not measured usage. Datadog meters LLM spans; Helicone meters requests and storage. The same application request therefore has no universal observability unit. Add provider tokens, judge calls, retention, and review time separately when you price the trial.

Provider notices supply context for an investigation

A published retirement or endpoint change can explain a route's availability. Start with the exact model ID and serving route from the trace, then compare the provider's announcement and effective dates with the request timestamp. The model-ID guide and published deprecation records help locate that evidence. A notice for a direct API does not automatically describe the same model name served through a gateway or cloud platform.

Even a matching notice doesn't establish why an answer changed. The prompt, retrieval data, tool output, application code, or routing configuration may also have changed. Preserve those versions and replay the failed case before blaming the provider. Our synthetic ticket mismatch has no serving route or provider event attached to it.

Radar supplies context about published provider changes within its coverage. It doesn't capture application traces or inspect every answer, and it cannot establish behavioral equivalence after a replacement. Keep the notice beside the investigation as source evidence; use your traces and acceptance checks to decide whether the application still does its job.

A bake-off beats a six-product demo

Table 4 Download CSV

Scenario	First tool to trial	The disqualifying test
One integrated cloud and open-source path	Langfuse	Confirm the server and SDK versions match, then replay one dataset experiment
LangChain or LangGraph already owns the agent loop	LangSmith	Trace non-framework code and verify the dataset handoff is still clean
Trace data must stay inside your infrastructure	Phoenix	Run the intended PostgreSQL and authentication setup, not the local SQLite demo
Releases are governed by datasets and experiments	Braintrust	Turn a real production failure into a scored regression case
The gateway must own cost, routing, and request analytics	Helicone	Export one failure into the external evaluation system you will actually use
Shared Datadog traces matter, or its standalone experiment loop fits	Datadog	Trace one failure, turn it into a dataset case, and compare one experiment in your supported site
Existing OpenTelemetry APM already answers the incident question	Keep it, then add one specialist only if needed	Prove that the new product closes an evaluation or LLM-semantics gap

We would compare two finalists with the same small workload rather than six products with six demo applications. Instrument one representative RAG or agent path with model, retrieval, tool, and application spans. Include a successful case, a provider error, a bad retrieval, a malformed tool call, a slow response, and a prompt that contains sensitive text that must be redacted.

Then run this sequence:

1. Send the same trace shape to both products and check whether parent-child relationships survive.
2. Find the bad retrieval without knowing its trace ID.
3. Convert that failure into a labeled dataset case.
4. Run one code-based evaluator and one human review pass.
5. Compare a changed prompt against the original cases.
6. Export the data, estimate the production billing unit, and rehearse deletion.

For a local regression layer, the framework comparison runs one deterministic fixture through Promptfoo and DeepEval before any result is attached to an observability product.

Record setup time, missing spans, time to isolate the failure, evaluator reproducibility, redaction behavior, data export, and monthly cost at the expected volume. Do not use the vendor's sample chatbot as the test workload. It has already been arranged to make the vendor's happy path happy.

This article has not run that bake-off. Until it does, the verdict is about documented workflow and deployment fit, not runtime quality.

The loop is the product, not the trace store

Observability begins before deployment. The RAG model guide helps choose the generation layer, but a production trace must also show retrieved document identifiers, ranking or filtering steps, tool calls, and the final answer. Otherwise a groundedness failure looks like a model failure even when retrieval supplied the wrong evidence.

The deployment guide covers runtime boundaries such as secrets, timeouts, streaming, and background work. Add the trace exporter to that design before traffic arrives, and make sure it flushes under the actual serverless or worker lifecycle. A local span that never leaves the process is comforting but not observable.

Model tokens remain a separate cost layer. Use the LLM pricing comparison for provider rates, then add observability ingestion, evaluator model calls, retained data, and the people reviewing failures. The cheapest trace store can become the expensive choice if nobody can turn a production miss into a regression test.

The trace store is the commodity. The loop from production failure to repeatable test is the product, and it belongs to whichever tool your team will actually close it in.