Self-Healing in AI Ops: What Should Stay Automatic?

The most important truth about self-healing in AI operations is that self-healing has levels: restarts are easy, judgment is human. This blog runs dozens of posts a day with no human on duty, so without automatic recovery it wouldn't survive. But the fantasy that "code fixes itself" is dangerous: the industry agrees that deterministic infrastructure recovery (restart, rollback) is mature, while probabilistic self-healing logic is still a dangerous frontier. Even public measurements show mature auto-remediation cuts MTTR by an average of 65% (4 hours to about 1.4), yet most teams are still at L0 (manual) to L1 (rollback). So our principle is detect automatically, recover by level, escalate to a human when it can't fix. In short: automate the reversible, keep judgment human.

In one line: self-healing has levels - reversible recovery like restart, rollback, and quarantine is automatic, while probabilistic self-repair logic and deploys are escalated to a human.

In plain terms: self-healing is like a car's safety systems. The car handles tire-pressure warnings (detection) and ABS (automatic intervention), but whether to change lanes or change destination (judgment) is the driver's. If automation grabs the wheel too, it causes crashes - automating only the reversible is the safety line.

How do we detect and recover?#

The flow is health-check detection, then restart/quarantine/auto-repair, then escalation. A health endpoint continuously checks liveness; if it dies, the service restarts by runbook (an approved script, with an audit log); if a draft violates format, it is sent to a quarantine folder with the cause recorded. Some violations self-heal without a human - if internal links are too few, related-post links are auto-injected and re-gated. The strength is unattended stability in steady state, but what sustains it is drawing the line on what to leave automatic.

How do we stop runaway remediation?#

Circuit breakers and audit logs. If auto-remediation fires repeatedly for the same issue, we stop and alert a human - repetition signals a deeper flaw, so we don't worsen state with infinite retries (our publish report even carries near-dup and breaker signals). And we log every recovery action - if infrastructure heals itself, you must know what and why to judge next time. The heart of runaway prevention is not fast recovery but deciding when to stop and call a human.

What can't self-healing fix?#

Problems that need judgment, especially a deploy gap. In fact, a fix on the development branch once didn't ride into the deploy branch, leaving production stuck on the old state - restarts and quarantine don't cure that. It needed the human judgment of merge-and-deploy, and automation did not deploy on its own but escalated through the approval flow. Self-repairing logic tends to be plausibly wrong, so a human keeps the wheel on core judgment. And we don't trust self-healing - we inject failures on purpose (chaos) and verify regularly; if you never break it, you don't know it truly heals.

So what is our self-healing discipline?#

The key is automate the reversible, keep judgment human.

• Detect: catch anomalies automatically via health checks, the gate, and smoke tests (avoid silos - scattered signals mislead).
• Recover: restart, quarantine, and auto-fix are automatic; irreversible deploy and delete need human approval. Log every recovery.
• Limit: stop repeated firing with a circuit breaker and call a human; escalate judgment problems. Verify regularly with chaos.

Related reading: 오픈 음성 클로닝, 우리는 이렇게 운영한다 — 파이프라인 회고, 에이전트 브라우저 제어 — 우리는 이렇게 운영한다(회고)

Related reading: AI 운영 self-healing 자동화, 직접 써본 실측·한계, 이 블로그의 자동 발행 파이프라인, 직접 써본 실측·한계

Reference links

• Kubernetes (declarative state reconciliation)
• Google SRE Book (reliability operations)
• Google SRE Workbook (practical implementation)
• Chaos Monkey (chaos engineering)
• OpenTelemetry (observability, telemetry)

Note: figures like MTTR 65%, most teams at L0/L1, and silos 28% are cited from 2026 public benchmarks (AIOps reports, Forrester, etc.) and vary widely by environment (greenfield K8s vs legacy) - not permanent numbers. Self-healing "logic" is probabilistic and risky, so we keep a human on irreversible judgment. Auto-remediation practices move fast, so this is reviewed quarterly.