How AI Ops Observability and Telemetry Work

If observability is "what you watch," how it works is one trace_id following a request to stitch metrics, logs, and traces into a single thread. The flow is instrument (the OTel SDK wraps code in spans) → propagate trace context (the W3C traceparent header carries trace_id and span_id across service boundaries) → collect (a collector) → sample → correlate → export. The crux is injecting the shared trace_id into logs and attaching it to metrics as an exemplar - then moving between signals becomes a primary-key lookup, not a timestamp guess. Our publishing pipeline records funnel events (draft → gate → publish) the same way to watch pass rate.

In plain terms: a trace_id is a parcel tracking number. Each warehouse (service) stamps the same number, so metrics (how many shipped), logs (what happened), and traces (where it went) are tied together under one number.

How do the three signals connect into one?#

One shared trace_id links them. Three conditions give end-to-end correlation: (1) trace context propagates across every service hop and async boundary, (2) every log emitted inside a span carries trace_id and span_id, and (3) every metric measured inside a span attaches the trace_id as an exemplar. So "which request (trace) and which log caused this latency spike (metric)" resolves in one lookup. The standard matters - the propagation method (W3C, etc.) must be unified across the whole system or traces break.

First, the terms. A metric is a numeric aggregate like "requests per second," a log is a text record of "this happened at this time," and a trace is the path one request took across services. These three normally live apart, so when latency spiked you had to guess a time window and dig through logs. Plant the trace_id in all three and that guesswork becomes a look-it-up-by-one-number query.

Where and how is sampling decided?#

At two points: head and tail. Head sampling decides early at span creation to save resources, but it cannot see errors yet, so it cannot select error traces. So tail sampling runs in the collector - after a trace completes, it looks at the outcome, keeps all errors and slow ones (e.g. a measured over 500ms), and samples the rest at a low rate (e.g. 10%). There is a constraint: all spans of a trace must reach the same collector for the decision to be correct, and it must sit after context-dependent processors in the pipeline.

Why does this matter? Storing every trace makes cost explode, but blindly keeping only 10% can throw away the very error traces you need to see. Tail sampling, by "looking at the outcome after completion," strikes a middle path: keep 100% of problem traces and thin out only the healthy ones.

What does LLM ops instrument on top?#

Prompts, tokens, cost, and finish reason, into spans. The OTel GenAI semantic conventions wrap an LLM call in a span with standard attributes like gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens (the basis for cost tracking), and gen_ai.response.finish_reasons. For agents, each tool call, retrieval, and LLM step becomes a child span, so the whole reasoning chain becomes one trace. Important caution: putting full prompt text into indexed span attributes is a PII-exposure anti-pattern. And GenAI traces are nothing special - they are just OpenTelemetry - so the propagation, sampling, and correlation above apply directly (though the GenAI conventions are still in Development as of 2026).

How do you explore it (safely)?#

Small, by the standard.

• Instrument with the OTel SDK and unify one propagation method (mixing breaks traces).
• Link the trace_id into logs and metrics (exemplars) so moving between signals is one lookup.
• Use tail sampling to keep error and slow traces, and for LLM spans record tokens but do not put full prompt text in indexed attributes.

Reference links

• OpenTelemetry (observability standard)
• W3C Trace Context (propagation standard)
• OTel Collector contrib (tail sampling)
• OTel Semantic Conventions (GenAI)
• OpenLLMetry (LLM auto-instrumentation)

Note: standards, mechanisms, and figures are public 2026 docs and vary by implementation and version (the GenAI conventions are pre-stable, so attribute names may change). Do not store full prompt text in indexed attributes (PII). Measure exact cost and latency on your own system. Reviewed quarterly.