Building a Reconciliation Loop That's Safe to Interrupt

The problem with naive deploy scripts

A deploy script that runs top-to-bottom has one fatal assumption: that it will finish. The moment a pod gets evicted, a network blips, or someone hits Ctrl-C, you're left in an unknown state — half-applied manifests and a rollout nobody can reason about.

ForgeOps takes a different stance: the engine never "does a deploy." It continuously drives the cluster toward a desired state.

Desired state vs. live state

Every service has a declarative spec:

service: payments-api
image: registry/payments:1.8.2
replicas: 4
strategy:
  type: canary
  steps: [10, 50, 100]

The control loop does three things, forever:

1. Observe — read live state from the Kubernetes API.
2. Diff — compute the delta between desired and live.
3. Act — apply the smallest step that reduces the delta.

Because each tick is idempotent, killing the process mid-rollout is a non-event. On restart, it observes, diffs, and continues exactly where it left off.

Modelling rollout as a state machine

The key insight: a canary rollout is a deterministic state machine, not a sequence of commands.

Pending → Step(10%) → Step(50%) → Step(100%) → Healthy
                ↘ (health gate fails) → RollingBack → RolledBack

State lives in PostgreSQL, not in the process. The process is disposable; the state is durable. That single decision is what makes the whole system recoverable.

Takeaway

If a process holds the only copy of "what we're doing," you've built something fragile. Push the state out, make every step idempotent, and interruption stops being a failure mode.