Skip to main content

Durable Storage, Migrations, and Restart Recovery

Week 13 answered a narrow settlement question: which exact TLC does an on-chain proof resolve? Week 14 asks the operational question that follows immediately: Will that answer still be correct after a crash, upgrade, restore, or restart? An in-memory reconciliation can be logically perfect and still be operationally unsafe. A process can die after changing a payment but before recording its event. A migration can rewrite half a database. A backup can be internally consistent but belong to another node or network. A restart can spawn duplicate trackers and repeat work that was already completed. The week’s central invariant is therefore: Durability is not “the database contains rows”; it is the ability to recover one coherent state transition and prove what may safely resume.

Begin with an atomic durability boundary

A settlement transition often changes several records together: the TLC or payment state, an audit event, an idempotency marker, and perhaps an outbox item. Publishing only some of them creates a state that no restart algorithm can interpret safely. The first exercise stages those records as one candidate generation. A crash before commit exposes none of them. A crash after commit exposes all of them. A monotonic revision rejects stale writers, and a durable command ID makes replay a no-op. The exact batching primitive is deliberately local; production Fiber backends have their own transactions and write batches. This is the application equivalent of the CKB consume-and-create mental model established earlier in the course: readers should observe a complete old state or a complete new state, not a hand-built mixture. CKB consensus does not make the operator database atomic, but confirmed on-chain evidence remains an external checkpoint against which restored local claims can be reconciled. See the official How CKB Works guide for the chain-side state transition model.

Migrations are a graph, not a version assignment

Fiber’s v0.8.0 migration guide documents a concrete legacy upgrade path, including channel replay fields, index re-keying, a pre-migration backup, and idempotent reruns. Since v0.9.0, supported migrations are integrated into fnn startup rather than shipped as a standalone current-version migrator. That does not mean an operator should blindly change schemaVersion = latest. Each edge must exist, run in order, and publish atomically. The Week 14 model rejects:
  • a missing migration edge;
  • an older binary opening a future schema;
  • a migration without a verified pre-migration snapshot;
  • a partially completed sequence;
  • and a rerun that would duplicate already-applied transformations.
Ephemeral indexes may be rebuilt, while durable channel/payment intent must survive. The distinction belongs to the schema design, not to guesswork during an incident.

A backup is useful only if restore can distrust it

Fiber v0.9.0 added backup and restore support for storage backends. Subsequent hardening also applied restrictive permissions to backup files. Those are verified implementation facts; the manifest in Exercise 55 is a teaching policy layered on top. The local backup records a payload digest, entry count, schema and store revisions, network, node identity, key fingerprint, and expected owner-only file mode. Restore validates all of them before exposing the image. It also requires an empty staging destination so recovery cannot silently merge an old snapshot into live state. The identity checks matter because a byte-perfect backup can still be the wrong backup. Mainnet state must not be loaded into a testnet process. One node must not inherit another node’s channel state without the corresponding key material and an explicit recovery procedure. A database from a newer schema must not be opened by a binary that cannot interpret it. The exercise does not claim Fiber uses this JSON manifest or these exact fields. It demonstrates why checksum-only restore is incomplete.

Restart from durable intent, then consult external evidence

Process memory is gone after a restart. Recovery must rebuild work from committed intent and terminal state:
  • completed payments stay completed;
  • only incomplete or failed attempts become retry candidates;
  • duplicate LND invoice trackers coalesce within the same tenant;
  • receive-side creation resumes from its durable phase;
  • tenant boundaries remain part of every recovery key;
  • and exact committed CKB settlement evidence chooses reconciliation over blind retransmission.
Weak or unconfirmed chain evidence stays unknown. It is neither proof of settlement nor permission to resend. Applying the same recovery plan twice produces one logical transition because every action has a durable idempotency key. Fiber v0.9.0’s release history specifically reports improvements to CCH payment recovery, LND payment recovery, receive-side creation, channel recovery, external-funding persistence, and restart safety. Exercises 53–56 turn those themes into inspectable invariants without pretending to reproduce Fiber’s actor runtime.

Verified behavior versus teaching model

Primary implementation references include the Fiber v0.9.0 release, backup and restore PR #1197, unified migration design PR #1268, durable CCH recovery PR #1602, LND restart recovery PR #1546, recoverable receive creation PR #1573, and tracker coalescing PR #1556.

Run the exercises

Read them in order:
  • 53-durable-write-boundaries.ts tests crashes on both sides of one atomic publication point, duplicate commands, stale revisions, and torn recovery images.
  • 54-ordered-idempotent-migrations.ts applies ordered schema edges, proves rerun safety, and rejects gaps, future schemas, missing backups, and partial failures.
  • 55-backup-restore-integrity.ts validates payload, manifest, identity, network, key, schema, staging, and permission boundaries.
  • 56-restart-recovery-invariants.ts rebuilds only safe work, coalesces trackers, honors exact committed chain evidence, and applies restart plans idempotently.
The exercises follow CKB Quest in making each checkpoint visible and executable. Nervos Nation remains a useful ecosystem discovery layer, while protocol and operational claims stay anchored to official documentation and source history. Week 14 leaves the node ready for Week 15’s harder network problem: after local state survives, how does the node reestablish channels and synchronize with peers without replaying the wrong work or accepting unbounded traffic?