Seeing AI Ops Through Observability: How We Operate It

The core truth of observability in AI operations is that an agent returns HTTP 200 even when it's wrong: no exception, a 200 response, yet the content can be wrong or harmful. So traditional monitoring ("is it up or down") isn't enough; you need semantic telemetry that sees "what it decided and why." LLMs are non-deterministic - the same prompt yields different outputs - so to reproduce an issue you must record the exact input, model, and temperature at that moment. Industry standards are converging here: the OpenTelemetry GenAI semantic conventions (CNCF) define a standard schema for prompts, tokens, and tool calls, and in 2026 85% of organizations use GenAI for observability. In short: measure "what it produced," not "did it run."

In one line: an agent returns HTTP 200 even when wrong, so you need semantic telemetry that sees "what and why it produced," not just "is it alive" - record input, model, and temperature to reproduce, don't game the metric (Goodhart), and never store full prompt text.

In plain terms: AI observability is like a medical exam. A patient walking in (HTTP 200) doesn't mean healthy - you need bloodwork and imaging (output-quality metrics) to know the real state. Catching the outwardly-fine-but-inwardly-sick case is the whole point, so we measure ="is it doing it right"= over "is it alive."

What do we measure?#

One north-star plus quality and reader metrics. Our north-star is posts published per day, and under it sit gate pass rate, quarantine rate, and recurrence rate (production quality) and visits, comments, subscriptions (reader response). Honestly, our reader signals are near the floor right now - few visits, zero comments, zero subscriptions - and we don't hide these numbers because a metric that flatters you is useless. The publish report emits draft count, gate failures, and pass rate daily so leaks are visible. That is, we watch metrics that change behavior, not vanity metrics.

Metrics are layered: liveness is only necessary, quality sits on top of it, and reader response on top of that - only then does "did it produce well" come into view.

What's the most common trap in observability?#

When a metric becomes a target, the metric breaks (Goodhart's law). If the north-star is "post count," mass-publishing garbage raises the number while quality collapses - so post count is always paired with the quality gate and recurrence rate. Another is vanity metrics: nice-looking numbers that don't change behavior (e.g., total request count) just fill a dashboard. And since prompts carry sensitive data, storing full text as attributes is an anti-pattern (indexing, PII exposure) - the standard is designed to redact and drop content at the collection stage. The purpose of observability is not pretty graphs but changing the next action.

The Goodhart trap detonates when you chase "post count" alone - paired with the quality gate and recurrence rate, publishing can rise while quality holds.

What can't observability see?#

Intent, long-term impact, and quality the gate can't catch. Metrics show "what happened" but can't fully measure "is this a good post" - the gate checks format, density, and links, but depth of insight is judged finally by humans and readers. So we pair automatic metrics with human review (e.g., a cold-buyer-perspective review). And an honest measurement: however good the observability, the near-zero-reader reality is what the metrics tell us - looking at that uncomfortable number is the essence of observability.

So what is our observability discipline?#

The key is measure "did it produce well," not "did it run," and don't game the metric.

• Measure: liveness (healthz) is only necessary; watch output quality (pass, quarantine, recurrence) with reader response.
• Honesty: no vanity metrics; don't hide uncomfortable numbers (floor-level visits and subs). A metric must change behavior to count.
• Safety: don't store full prompt text (PII, indexing risk); redact and route at the collection stage per standard conventions. Supplement metrics with human review.

Related reading: 오픈 음성 클로닝, 우리는 이렇게 운영한다 — 파이프라인 회고, 자율 AI 블로그를 관측하는 법: 텔레메트리를 직접 만들고 측정한 기록

Related reading: 관측·텔레메트리로 AI 운영 보기, 어떻게 동작하나, 관측·텔레메트리로 AI 운영 보기 프리뷰: 무엇이고 왜

Reference links

• OpenTelemetry (observability standard, CNCF)
• OTel GenAI semantic conventions
• Prometheus (metrics collection)
• Grafana (dashboards, visualization)
• Google SRE Book (SLOs, monitoring)

Note: figures like 85% adoption, gate pass rate, and reader signals are from 2026 public reports (Elastic, etc.) and our operating snapshot, and shift with policy and time (not permanent). The OTel GenAI conventions are largely experimental in 2026, so attribute names may change. Because of Goodhart risk, no single metric is made a target alone. Observability practices move fast, so this is reviewed quarterly.