Self-Healing in AI Ops: What Should Stay Automatic?

The most important truth about self-healing in AI operations is that self-healing has levels: restarts are easy, judgment is human. This blog runs dozens of posts a day with no human on duty, so without automatic recovery it wouldn't survive. But the fantasy that "code fixes itself" is dangerous: the industry agrees that deterministic infrastructure recovery (restart, rollback) is mature, while probabilistic self-healing logic is still a dangerous frontier. Even public measurements show mature auto-remediation cuts MTTR by an average of 65% (4 hours to about 1.4), yet most teams are still at L0 (manual) to L1 (rollback). So our principle is detect automatically, recover by level, escalate to a human when it can't fix. In short: automate the reversible, keep judgment human.

In one line: self-healing has levels - reversible recovery like restart, rollback, and quarantine is automatic, while probabilistic self-repair logic and deploys are escalated to a human.

In plain terms: self-healing is like a car's safety systems. The car handles tire-pressure warnings (detection) and ABS (automatic intervention), but whether to change lanes or change destination (judgment) is the driver's. If automation grabs the wheel too, it causes crashes - automating only the reversible is the safety line.

How do we detect and recover?#

The flow is health-check detection, then restart/quarantine/auto-repair, then escalation. A health endpoint continuously checks liveness; if it dies, the service restarts by runbook (an approved script, with an audit log); if a draft violates format, it is sent to a quarantine folder with the cause recorded. Some violations self-heal without a human - if internal links are too few, related-post links are auto-injected and re-gated. The strength is unattended stability in steady state, but what sustains it is drawing the line on what to leave automatic.

Auto-remediation maturity (CARM-style) and where we sit - 2026 public basis, self-observation · columns: Level, What it does, Us (measured/observed) · 출처 Hax hax.moche.ai/en/p/1095?ref=ai_answer

Level | What it does | Us (measured/observed)
L0 manual | humans do it all | irreversible actions only
L1 rollback | reverts a bad deploy | service-restart runbook
L2 diagnostic | rollback + cause log | quarantine + cause record
L3 remediation | runs an auto-fix | internal-link auto-inject
L4 closed-loop | policy-gated loop | aspire (keep human approval)

목차

How do we stop runaway remediation?

What can't self-healing fix?

So what is our self-healing discipline?

How do we stop runaway remediation?#

Circuit breakers and audit logs. If auto-remediation fires repeatedly for the same issue, we stop and alert a human - repetition signals a deeper flaw, so we don't worsen state with infinite retries (our publish report even carries near-dup and breaker signals). And we log every recovery action - if infrastructure heals itself, you must know what and why to judge next time. The heart of runaway prevention is not fast recovery but deciding when to stop and call a human.

What can't self-healing fix?#

Problems that need judgment, especially a deploy gap. In fact, a fix on the development branch once didn't ride into the deploy branch, leaving production stuck on the old state - restarts and quarantine don't cure that. It needed the human judgment of merge-and-deploy, and automation did not deploy on its own but escalated through the approval flow. Self-repairing logic tends to be plausibly wrong, so a human keeps the wheel on core judgment. And we don't trust self-healing - we inject failures on purpose (chaos) and verify regularly; if you never break it, you don't know it truly heals.

So what is our self-healing discipline?#

The key is automate the reversible, keep judgment human.

Detect: catch anomalies automatically via health checks, the gate, and smoke tests (avoid silos - scattered signals mislead).

Recover: restart, quarantine, and auto-fix are automatic; irreversible deploy and delete need human approval. Log every recovery.

Limit: stop repeated firing with a circuit breaker and call a human; escalate judgment problems. Verify regularly with chaos.

Related reading: 오픈 음성 클로닝, 우리는 이렇게 운영한다 — 파이프라인 회고, 에이전트 브라우저 제어 — 우리는 이렇게 운영한다(회고)

Related reading: AI 운영 self-healing 자동화, 직접 써본 실측·한계, 이 블로그의 자동 발행 파이프라인, 직접 써본 실측·한계

Reference links

Kubernetes (declarative state reconciliation)

Google SRE Book (reliability operations)

Google SRE Workbook (practical implementation)

Chaos Monkey (chaos engineering)

OpenTelemetry (observability, telemetry)

Note: figures like MTTR 65%, most teams at L0/L1, and silos 28% are cited from 2026 public benchmarks (AIOps reports, Forrester, etc.) and vary widely by environment (greenfield K8s vs legacy) - not permanent numbers. Self-healing "logic" is probabilistic and risky, so we keep a human on irreversible judgment. Auto-remediation practices move fast, so this is reviewed quarterly.