LLM Evals and Observability: Eval Sets from Real Cases, LLM Judges and Tracing with OpenTelemetry
Building an eval set from real cases; code-based checks, LLM-as-a-judge and human review; judge biases (position, verbosity, self-preference) and calibrating a judge against human labels; offline evals in CI and sampled online evals; nondeterminism and regression testing; agent and trajectory evals; tracing with the OpenTelemetry GenAI semantic conventions; what to log per call; a tool overview and a starter checklist.
Updated:
Summary
In an LLM application, two things close the gap between "looks like it works" and "works": evals and observability. Evals are the repeatable way to measure a change (a new prompt, a new model, a new tool) on your own tasks before and after it ships. Observability is being able to read, after the fact, what every production call did: which model, how many tokens, how long it took, what it cost, where it broke.
They are two halves of one loop: production traces surface real failures, those failures go into the eval set, and the eval set catches the next change in CI.
Note: Facts were checked on 8 November 2026 against the papers (arXiv), OpenTelemetry's GitHub repositories and each tool's own documentation. The OpenTelemetry GenAI conventions are still in development; attribute names may change. Every sentence about a tool reflects what that tool's own docs say; we don't compare or rank them.
The eval set: start from real cases
An eval set has three parts: an input, the expected behavior (sometimes an exact answer, more often a criterion) and a scoring method. The most valuable inputs come from real usage, not invented examples.
Where to collect them
- Production traces: conversations where users disliked the answer, asked again, or opened a support ticket.
- What the team already tries by hand: every prompt pasted into Slack with "let's try this one" is a potential test.
- Known failure types: hallucination, wrong tool calls, broken formats, prompt injection attempts, refusing when it shouldn't.
- Edge cases: empty input, very long input, another language, typos.
How big? Anthropic's post "Demystifying evals for AI agents" (January 2026) says teams delay evals because they think they need hundreds of tasks, while 20-50 simple tasks drawn from real failures is a great start: early on, each change has a large effect, so a small set shows it. Anthropic's guide to building evaluations points the same way, "prioritize volume over quality": more automatically graded questions beat fewer hand-graded ones.
What makes a good eval example
- Unambiguous. A system that follows instructions correctly should be able to pass. Anthropic describes finding, in an audit of Terminal-Bench, tasks that didn't specify a file path while the tests assumed one; such tests measure the test, not the system.
- Labelled. Record the category (billing, refund, technical), difficulty and source (production, manual, synthetic) so you can read results by slice.
- Versioned. The set is versioned like code. Deleting an example because "it doesn't pass any more" is the easiest way to hide a regression.
- Scrubbed. Mask personal data in examples taken from production.
Synthetic data (having an LLM generate variations) is useful to widen coverage, but it doesn't replace real examples: the model tests the user it imagines.
Three ways to score: code, LLM judge, human
Anthropic's guide splits grading into three and is explicit about the order: code-based is fastest and most reliable but lacks nuance; human is the most flexible and highest quality but slow and expensive ("avoid if possible"); LLM-based is fast, flexible and suited to complex judgment, but "test to ensure reliability first, then scale."
| Code-based check | LLM-as-a-judge | Human review | |
|---|---|---|---|
| Examples | Exact match, regex, JSON schema, unit tests, SQL result, tool-call arguments | Pass/fail against a rubric, which of two answers is better, tone, faithfulness | Expert labelling, user feedback, A/B |
| Speed and cost | Instant, nearly free | Seconds, token cost | Hours to days, expensive |
| Repeatability | Full | Partial (the judge is nondeterministic too) | Low to medium (inter-annotator disagreement) |
| Weakness | Can't measure open-ended quality | Biases; needs calibration | Doesn't scale |
| When | If correctness can be checked mechanically: do this first | If the criterion can be written but not coded | Calibrating the judge, high-stakes decisions |
Rule of thumb: check in code everything you can check in code, use an LLM judge for the rest, and keep humans for measuring the judge and for the riskiest examples. One task can carry several checks: "is the JSON valid" (code) plus "does it follow the refund policy" (judge).
A code-based check
import json, re
def check_refund_reply(output: str) -> dict:
"""Everything code can check: format, required fields, forbidden content."""
results = {}
try:
data = json.loads(output)
results["valid_json"] = True
except json.JSONDecodeError:
return {"valid_json": False}
results["has_order_id"] = bool(re.fullmatch(r"ORD-\d{6}", data.get("order_id", "")))
results["amount_is_number"] = isinstance(data.get("refund_amount"), (int, float))
results["no_internal_link"] = "intranet." not in data.get("message", "")
return resultsLLM-as-a-judge and its biases
LLM-as-a-judge means giving an LLM a rubric and having it score another model's output (or its own). The work that popularised the term is Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023 Datasets and Benchmarks). It defines three variants:
- Pairwise comparison: a question and two answers; the judge picks the better one or declares a tie.
- Single answer grading: the judge scores one answer directly.
- Reference-guided grading: the judge also gets a reference solution (e.g. for math).
The paper's headline finding: strong judges like GPT-4 matched both controlled and crowdsourced human preferences with over 80% agreement, the same level as agreement between humans. The same paper documents the judge's limits:
- Position bias: the verdict can depend on the order of the answers. Wang et al. (2023), "Large Language Models are not Fair Evaluators", showed they could flip results just by reordering: with ChatGPT as the evaluator, Vicuna-13B "beat" ChatGPT on 66 of 80 tested queries.
- Verbosity bias: preferring the longer answer even when it isn't better. Zheng et al. tested this with a "repetitive list" attack that repeats items in an answer without adding information.
- Self-enhancement / self-preference: the judge favouring answers it generated. Zheng et al. saw some judges favour some models but state plainly that, with limited data and small differences, they could not determine whether self-enhancement bias exists. Panickssery, Bowman and Feng (2024), "LLM Evaluators Recognize and Favor Their Own Generations", reported that models can distinguish their own outputs and found a linear correlation between that self-recognition ability and the strength of self-preference bias.
- Limited reasoning: a judge can be misled by a wrong answer even on a simple math question it can solve on its own.
Mitigations (from the papers themselves)
- Swap positions: Zheng et al.'s "conservative" approach calls the judge twice with the answers swapped and only declares a win if the same answer wins in both orders; otherwise it's a tie. Wang et al.'s "Balanced Position Calibration" also aggregates results across orders.
- Evidence before score: Wang et al.'s "Multiple Evidence Calibration" asks the evaluator to produce evidence before rating; Anthropic's guide recommends a grader with thinking enabled.
- Give a reference: in Zheng et al.'s reference-guided method the judge first answers the question independently, and that answer goes into the judge prompt as a reference; on math questions the failure rate in the paper's table dropped from 70% to 15%.
- A judge from a different model family: the obvious way to reduce self-preference is not to use the model being evaluated as its own judge.
- A clear, narrow rubric: Anthropic recommends "detailed, clear rubrics" and specific outputs such as "only 'correct' or 'incorrect'." A separate binary (pass/fail) judge per criterion is usually easier to calibrate than one judge scoring five criteria at once (a practical observation, not a measured claim).
Calibrating the judge against human labels
A judge is a measuring instrument; you calibrate an instrument before you use it. The method is the same as evaluating a classifier in machine learning:
- Build a human-labelled sample. Take outputs from production or the eval set; a domain expert marks each pass/fail with a short reason. If possible have two people label; their agreement shows the ceiling you can expect from a judge.
- Split the sample. Iterate on the judge prompt on one part and measure on the other, unseen part. Tuning and measuring on the same examples teaches the judge those examples (overfitting).
- Read agreement through a confusion matrix, not just a percentage. "90% agreement" means nothing if 90% of labels are "pass." Look at the confusion matrix, precision/recall, true positive and true negative rates, and a chance-corrected agreement measure (Cohen's kappa) together.
- Read the disagreements. Is the error in the judge, the rubric or the human label? Often the rubric is ambiguous.
- Version the judge and its prompt. Recalibrate when the judge model or prompt changes; old and new scores aren't directly comparable.
from sklearn.metrics import cohen_kappa_score, confusion_matrix
# human: expert labels, judge: judge verdicts (same order, held-out sample)
human = ["pass", "fail", "pass", "pass", "fail", "fail", "pass", "fail"]
judge = ["pass", "fail", "pass", "fail", "fail", "pass", "pass", "fail"]
tn, fp, fn, tp = confusion_matrix(human, judge, labels=["fail", "pass"]).ravel()
print("TPR (judge passed when the human passed):", tp / (tp + fn))
print("TNR (judge failed when the human failed):", tn / (tn + fp))
print("Cohen's kappa:", round(cohen_kappa_score(human, judge), 3))If judge-human agreement is close to human-human agreement, the judge is doing its job. If it is clearly lower, use the judge's scores to watch trends, not to make decisions.
Offline evals (CI) vs sampled online evals
| Offline (before shipping) | Online (in production) | |
|---|---|---|
| Data | Fixed, versioned eval set | A sample of real traffic |
| Expected answer | Often available | Usually not; reference-free checks |
| When | Every PR, every model/prompt change | Continuously, asynchronously |
| Purpose | Catch regressions, compare two versions | Drift, new failure types, real-world quality |
| Scoring | Code + judge | Cheap code checks on every call, judge on a sample |
LangSmith's and Braintrust's docs draw the same line: offline evaluation on curated datasets during development, online evaluation on live traffic.
Offline, in CI
- Run the eval set on every PR; compare the score with the latest score on the main branch.
- Look at slices as well as the total: the overall score can hold steady while the "refund" category collapses.
- Run expensive judge evals when prompt or model files change, or nightly, rather than on every commit.
- Set a threshold and actually fail CI: once "the score dropped but we merged anyway" happens, the eval set becomes decoration.
Online, in production
- On every call, cheap checks: is the JSON valid, is there forbidden content, is the answer empty, did a tool error come back. These are also guardrail candidates.
- On a sample, the judge: send a percentage of traffic to the judge asynchronously; your budget sets the percentage. Don't make the user wait.
- Attach user signals (thumbs up, regenerate, abandoning the conversation) to the trace.
- Put traces the judge failed, or users complained about, into a review queue; add the confirmed ones to the eval set. This is where the loop closes.
Regression testing and nondeterminism
The same input can give different outputs twice, which breaks classic "expected == actual" tests. The causes sit in two layers:
- Sampling: with temperature or top-p above zero, the model samples randomly on purpose.
- Numerical nondeterminism: even temperature 0 may not be enough. OpenAI's API docs describe the
seedparameter as best effort, state that determinism is not guaranteed, and point tosystem_fingerprintto monitor backend changes. In "Defeating Nondeterminism in LLM Inference" (September 2025), Horace He and colleagues at Thinking Machines argue that the common "concurrency + floating point" explanation is not the real cause in LLM inference; the real cause is kernels that are not batch-invariant: as server load changes, the batch size changes, and the numerical result for the same request can change. In their experiment, 1000 temperature-0 completions from Qwen3-235B produced 80 unique outputs; with batch-invariant kernels enabled, all 1000 were identical.
What to do in practice
- Where possible, test a property of the output, not the output itself: "does the answer contain an id starting with
ORD-." - Run each example several times and report rates. Agent evals use two measures: pass@k (at least one of k trials succeeds) and pass^k (all k trials succeed). pass^k was proposed in Yao et al. (2024), "τ-bench", to measure agent reliability. Anthropic's example: with a 75% per-trial success rate, the probability of passing all three of three trials is (0.75)³ ≈ 42%.
- Don't trust small differences between versions; run the same version a few times to see how much the score moves on its own. If the difference is smaller than that, there is no improvement.
- Pin the model: use the provider's dated/pinned model id rather than an alias, and record the returned model name in the trace.
Agent and trajectory evals
With agents you evaluate a sequence of steps, not one answer: model calls, tool calls, intermediate results (the agent loop). Anthropic's agent evals post separates two things:
- Transcript (trajectory): the full record of the agent's steps.
- Outcome: the final state of the environment at the end of the trial. The post's example: a flight-booking agent may say "your flight has been booked"; the outcome is whether a reservation actually exists in the environment's database. τ-bench likewise scores by comparing the database state at the end of the conversation with the annotated goal state.
What to check
- Outcome: was the task done? For a coding agent, do the tests pass; for a support agent, was the record updated correctly?
- Efficiency: how many turns, tokens and tool calls, how long it took.
- Rules: was an unauthorised tool called, did the agent wait for human approval at a step that requires it, did private data leak?
- Tool-call correctness: the right tool with valid arguments (function calling).
Careful: Anthropic writes that checking whether tool calls happened in a specific order is too rigid and produces brittle tests, because agents find valid approaches the eval designers didn't anticipate. Its advice is to grade what the agent produced rather than the path it took. Keep path checks for things that are genuinely rules ("never call the payment tool without approval").
And the cheapest method of all: read the transcripts. Anthropic says you won't know whether your graders work unless you read the transcripts and grades from many trials; when a task fails, the transcript shows whether the agent or the grader made the mistake. For the machinery around an agent, see Anatomy of an AI Harness.
Tracing with the OpenTelemetry GenAI semantic conventions
A trace is the end-to-end record of one request, made of nested spans: under an "invoke agent" span sit "model call" and "execute tool" spans. OpenTelemetry (OTel) is the open source project that standardises the format and transport of this telemetry; its semantic conventions define which attribute is written under which name. When everyone uses the same names, you instrument once and send the data to whichever backend you like (Langfuse, Phoenix, LangSmith, Datadog, Grafana…).
Status (as of November 2026)
- The GenAI conventions have Development status; they are not stable.
- They have moved out of the main
semantic-conventionsrepository into a separate repository,open-telemetry/semantic-conventions-genai. The main repo's v1.41.0 (April 2026) still contained the GenAI pages; from v1.42.0 (June 2026) the old pages on opentelemetry.io show a "Moved" notice pointing to the new repo. The new repo has no published release yet. - Names keep changing. For example, the
gen_ai.client.token.usagemetric from earlier versions has, in the repo's current state, been split into separate metrics such asgen_ai.client.inference.usage.input_tokens. Check which version your instrumentation library follows.
Core attributes on a model call span (per the current repo)
| Attribute | What |
|---|---|
gen_ai.operation.name |
The operation: chat, generate_content, embeddings, execute_tool, invoke_agent… (required) |
gen_ai.provider.name |
Provider: openai, anthropic, aws.bedrock… |
gen_ai.request.model / gen_ai.response.model |
The model requested and the model that actually answered |
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens |
Input and output tokens |
gen_ai.usage.cache_read.input_tokens |
Input tokens served from a provider-managed cache |
gen_ai.usage.reasoning.output_tokens |
Output tokens spent on reasoning (thinking) |
gen_ai.response.finish_reasons |
Why it stopped: e.g. ["stop"], ["stop", "length"] (provider values) |
gen_ai.conversation.id |
Links calls that belong to the same conversation |
gen_ai.input.messages / gen_ai.output.messages |
The messages themselves: Opt-In |
The span name should be {gen_ai.operation.name} {gen_ai.request.model} (e.g. chat model-name). For agents there are create_agent, invoke_agent and execute_tool spans; on the metrics side, gen_ai.client.operation.duration plus agent metrics such as gen_ai.invoke_agent.duration and gen_ai.invoke_agent.tool_calls. There are also separate convention pages for A2A and MCP (see Agent Protocols).
Content capture is opt-in: per the conventions, recording instructions, prompts, completions, tool arguments/results and memory records is gated by an explicit user opt-in, and instrumentations that capture content must provide a setting for it. Attributes such as gen_ai.input.messages_ref let you store messages externally and record only their URI on the span.
A common trap, cached tokens: per the conventions, gen_ai.usage.input_tokens should include cached tokens. The Anthropic API's input_tokens field excludes cached tokens and returns cache reads and writes in separate fields; the conventions' Anthropic page says to compute the total by adding them. Skip that and your input tokens appear to "drop" the day you turn on prompt caching.
import os
from anthropic import Anthropic
from opentelemetry import trace
tracer = trace.get_tracer("support-bot")
client = Anthropic()
MODEL = os.environ["LLM_MODEL"] # a pinned, dated model id
def answer(question: str, conversation_id: str) -> str:
with tracer.start_as_current_span(f"chat {MODEL}") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "anthropic")
span.set_attribute("gen_ai.request.model", MODEL)
span.set_attribute("gen_ai.conversation.id", conversation_id)
resp = client.messages.create(
model=MODEL, max_tokens=1024,
messages=[{"role": "user", "content": question}],
)
u = resp.usage
cache_read = u.cache_read_input_tokens or 0
cache_write = u.cache_creation_input_tokens or 0
span.set_attribute("gen_ai.response.model", resp.model)
# Convention: input_tokens should include cached tokens
span.set_attribute("gen_ai.usage.input_tokens", u.input_tokens + cache_read + cache_write)
span.set_attribute("gen_ai.usage.output_tokens", u.output_tokens)
span.set_attribute("gen_ai.usage.cache_read.input_tokens", cache_read)
span.set_attribute("gen_ai.response.finish_reasons", [resp.stop_reason])
return resp.content[0].textThis example writes the attributes by hand; in a real project a ready-made OTel instrumentation for your provider SDK or framework is usually less work. Whichever you use, check the names it emits against a real trace once.
What to log on every call
| Field | Why |
|---|---|
| Trace / span / conversation id | Link a complaint to the relevant calls in one click |
| Provider, requested model, returned model | Notice when the model behind an alias changes |
| Prompt version / template name | Know which prompt produced which score (gen_ai.prompt.name, gen_ai.prompt.version) |
| Input, output, cache-read (and cache-write) tokens, reasoning tokens | Cost and token optimisation |
| Total latency, time to first token | Perceived speed; with streaming the two differ a lot |
| Finish reason | Answers cut off with length are a silent failure |
| Errors, retry count, rate limits | Reliability |
| Computed cost | Multiply token counts by your own price table; version the price table too |
| Tool calls (name, duration, error) | Where agents stall and fail |
| User feedback and eval scores | Tie quality to the trace |
| Prompt and response text | Only with a deliberate opt-in, with masking and a retention period |
Reconcile computed cost with the provider's invoice from time to time: cache writes and reads, batch discounts and reasoning tokens can be priced differently. To cut tokens, see Token Reduction Techniques.
A tour of the tools
Every description below comes from the tool's own docs or repository (checked on 8 November 2026); it's not a ranking or a recommendation. Licences and ownership can change.
- Langfuse: describes itself as an open source AI engineering platform for "agent evals and observability": tracing, prompt management, evaluations (LLM-as-a-judge, code evaluators, user feedback, manual labelling), datasets and experiments, dashboards. It says it is built on OpenTelemetry and accepts OTLP traces. Runs as a cloud service or self-hosted with Docker Compose. Licence: MIT outside the
ee/folders. Per its README, part of ClickHouse since January 2026. - Braintrust: its docs present it as an observability platform for agents. They split an evaluation into data, task and evaluators (scorers and classifiers), and describe a cycle of playground → experiment → CI/CD → online scoring in production → feeding production traces back into datasets. The open source (MIT)
autoevalslibrary provides ready-made scorers. - LangSmith: LangChain's platform. Its docs treat observability (traces, production monitoring) and evaluation (offline on datasets before shipping; online on live traffic) as separate product areas, and suggest adding real traces to datasets so "a failure you saw once becomes a test you run every time." It says it works with frameworks beyond LangChain and accepts traces via OpenTelemetry.
- Helicone: per its README, an AI gateway and LLM observability platform: access to many models through one OpenAI-compatible endpoint, routing and automatic fallbacks, trace and session inspection, cost/latency tracking, prompt management. Integration means changing the client's
baseURL. Licence: Apache-2.0. - Arize Phoenix: per its README, Arize's open source AI observability platform for experimentation, evaluation and troubleshooting: OpenTelemetry-based tracing (OpenInference instrumentation), LLM-based evals, versioned datasets, experiments, a playground, prompt management. You self-host it locally, in a container or in the cloud; the managed offering is Arize AX. Licence: Elastic License 2.0 (ELv2).
- promptfoo: a CLI and library for evaluating and red-teaming LLM apps: automated evals, vulnerability scanning, side-by-side model comparison, automated checks in CI/CD. Per its README, promptfoo is now part of OpenAI and remains open source (MIT).
- OpenAI Evals: the
openai/evalsrepository is a framework for evaluating LLMs and LLM-based systems plus a registry of existing evals. Its README notes that evals can now also be configured and run directly in the OpenAI Dashboard (the platform's Evals guide). - Inspect: an LLM evaluation framework from the UK AI Security Institute (Python, MIT). It defines an eval as a task (dataset + solvers + scorers) and ships components for prompt engineering, tool use, multi-turn dialogue, agents, model-graded scoring and running inside sandboxes. Per its README, it includes a collection of over 200 pre-built evaluations.
Questions to ask when choosing: Where will the data live (cloud, your servers, the EU)? Is open source/self-hosting required? Will you instrument with OTel or the vendor's SDK (OTel makes switching easier)? Will evals live as code in CI or in a UI? How does it display agent traces (nested spans)?
Starter checklist
Eval set
- Have you collected at least a few dozen examples from real usage (traces, support tickets, team experiments)?
- Does each example record a category, source and expected behavior?
- Is the set under version control, with personal data masked?
- Is each example unambiguous enough that a system following instructions correctly can pass?
Scoring
- Is everything that can be checked in code checked in code?
- Is the judge's rubric narrow and binary; is the judge from a different model family?
- In pairwise comparisons, is the judge asked twice with the order swapped?
- Has the judge been measured on a held-out, human-labelled sample with a confusion matrix and kappa?
- Are the judge model and prompt versioned?
Process
- Does the eval set run on every PR and fail CI below a threshold?
- Are scores reported by category slice?
- Is each example run several times, and do you know how much the score moves on its own?
- Is the model id pinned?
- For agents, is the outcome (final environment state) graded, and are transcripts read regularly?
Production
- Is every call tied to a trace, with model, tokens (including cache), latency, finish reason, cost and prompt version recorded?
- Is recording prompt/response text a deliberate opt-in, with masking and retention defined?
- Do cheap checks run on every call and the judge asynchronously on a sample?
- Do failed traces flow from a review queue back into the eval set?
Further reading
- Evals and LLM-as-a-Judge: short definitions.
- Benchmark: shared exams and why they don't replace your own eval set.
- Guardrails and Human-in-the-Loop: where production checks and human approval fit.
- Anatomy of an AI Harness: the loop, tools and traces around an agent.
- Token Reduction Techniques and Prompt Caching: lowering the cost you log.
- System Prompt Guide: the thing your evals end up measuring.
Sources
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S. et al. — "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685
- Wang, P., Li, L., Chen, L., Cai, Z. et al. — "Large Language Models are not Fair Evaluators", 2023. https://arxiv.org/abs/2305.17926
- Panickssery, A., Bowman, S. R., Feng, S. — "LLM Evaluators Recognize and Favor Their Own Generations", 2024. https://arxiv.org/abs/2404.13076
- Yao, S., Shinn, N., Razavi, P., Narasimhan, K. — "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", 2024. https://arxiv.org/abs/2406.12045
- He, H. and Thinking Machines — "Defeating Nondeterminism in LLM Inference" (September 2025): https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
- Anthropic — "Demystifying evals for AI agents" (January 2026): https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Anthropic — Define success criteria and build evaluations: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
- OpenAI — Chat Completions API,
seedparameter (openai-python): https://github.com/openai/openai-python/blob/main/src/openai/types/chat/completion_create_params.py - OpenTelemetry — GenAI semantic conventions repository: https://github.com/open-telemetry/semantic-conventions-genai
- OpenTelemetry — Former GenAI page ("Moved" notice): https://opentelemetry.io/docs/specs/semconv/gen-ai/
- OpenTelemetry — Document status definitions: https://opentelemetry.io/docs/specs/otel/document-status
- Langfuse: https://github.com/langfuse/langfuse · https://langfuse.com/docs
- Braintrust — Evaluate systematically: https://docs.braintrust.dev/evaluate/index.md · autoevals: https://github.com/braintrustdata/autoevals
- LangSmith — Evaluation: https://docs.langchain.com/langsmith/evaluation · Trace with OpenTelemetry: https://docs.langchain.com/langsmith/trace-with-opentelemetry
- Helicone: https://github.com/Helicone/helicone · https://docs.helicone.ai
- Arize Phoenix: https://github.com/Arize-ai/phoenix · https://arize.com/docs/phoenix
- promptfoo: https://github.com/promptfoo/promptfoo · https://www.promptfoo.dev/blog/promptfoo-joining-openai/
- OpenAI Evals: https://github.com/openai/evals · https://platform.openai.com/docs/guides/evals
- Inspect (UK AI Security Institute): https://github.com/UKGovernmentBEIS/inspect_ai · https://inspect.aisi.org.uk/
Frequently asked questions
What is an LLM eval, and how is it different from a benchmark?
An eval is a repeatable test of how well your own application performs on your own tasks: inputs, expected behavior and a scoring method. A benchmark is a shared exam where everyone answers the same questions so models can be compared. Benchmarks narrow model choice; your eval set decides whether a change ships.
Can I trust LLM-as-a-judge?
Conditionally. Zheng et al. (2023) reported that a strong judge like GPT-4 agreed with human preferences over 80% of the time, the same level as agreement between humans; the same paper documented position, verbosity and self-enhancement biases. Don't trust a judge's scores until you have measured it on your task against a human-labelled sample.
How many examples should an eval set start with?
There is no fixed number. Anthropic's post on agent evals says 20-50 simple tasks drawn from real failures is a great start, because early changes have large, visible effects. The set grows as production surfaces new failures.
Does temperature 0 make outputs deterministic?
Not guaranteed. OpenAI documents its seed parameter as best effort and states determinism is not guaranteed. Thinking Machines' 2025 post showed that the batch size, which changes with server load, can change the numerical result for the same request. Base regression tests on repeated runs and thresholds, not a single run.
Are OpenTelemetry's GenAI conventions stable?
Not as of November 2026. The GenAI semantic conventions have Development status and have moved out of the main semantic-conventions repository into a separate one (semantic-conventions-genai). Metric and attribute names are still changing; check which version your instrumentation library follows.
What should I log on every LLM call?
At minimum: requested and returned model, provider, input, output and cache-read tokens, latency (total and time to first token), finish reason, errors, computed cost, prompt version and a trace or conversation id. Recording prompt and response text can capture personal data, which is why OpenTelemetry makes it explicitly opt-in.
Which eval/observability tool should I pick?
This guide doesn't rank them. First decide on open source or self-hosting needs, OpenTelemetry support, CI integration and where the data may live; then try two tools against your own small eval set.