Backup, restore and DR drill

Back up the audit ledger, MAQPNA resources, sealed signing keys and PostgreSQL state, restore them, prove it with a disaster-recovery drill, and rotate keys without downtime.

maqpna backup create copies everything you need to rebuild an installation: the verified audit ledger, every maqpna.com object, the control-plane Secrets sealed to your recovery key, and a dump of the PostgreSQL state. maqpna restore puts it back, and maqpna dr drill proves a backup restores without touching the live installation.

flowchart LR
  K[maqpna-sovereign x25519-keygen<br/>recovery key, kept offline] --> A
  A[maqpna backup create<br/>--seal-to recipient.pub] --> B[backup dir<br/>manifest.json, SHA256SUMS]
  B --> V[maqpna backup verify]
  V --> D[maqpna dr drill<br/>monthly]
  V --> R[maqpna restore<br/>--dry-run, then for real]
  R --> C[maqpna doctor]

Goal#

A scheduled, verified backup you have restored at least once in a drill, with a known recovery time and data-loss window.

Prerequisites#

• Admin access to the gateway admin API (the ledger is read through it): an OIDC token with the admin or auditor role, or the break-glass token.
• Kubernetes access to the control-plane namespace (objects and Secrets).
• For state.backend=postgres: pg_dump and pg_restore on PATH, and a file holding the DSN.
• An X25519 recovery key pair. The private key never goes near the cluster.

What a backup holds#

Keys held in a KMS or HSM (kms://, pkcs11: URIs) are referenced by the Secrets and never exported.

Steps#

1. Create the recovery key (once)#

maqpna-sovereign x25519-keygen -out-dir ~/maqpna-recipient

It writes recipient.key and recipient.pub. Keep recipient.key offline (a safe, an HSM, a sealed envelope); you need it only to restore Secrets.

2. Take a backup#

maqpna backup create --out /backups/2026-10-02 --gateway "$GW" \
  --seal-to ~/maqpna-recipient/recipient.pub --postgres-dsn-file dsn.txt

--include limits the parts (ledger,crs,secrets,postgres, all by default). The output directory must not exist or must be empty. A ledger-only backup of a local MAQPNA, real output:

$ maqpna backup create --out ../backup-2026-10-02 --include ledger
ledger: 2 records, head seq 2 8ed287bc7026ca7825f171ab57d487f3bbbacff9ed95ef9da5b00b31ad87c2a7
backup written to ../backup-2026-10-02; check it with: maqpna backup verify ../backup-2026-10-02

Schedule it (a CronJob or CI job) and copy the directory to storage in the same jurisdiction.

3. Verify the backup#

maqpna backup verify /backups/2026-10-02 --open-key ~/maqpna-recipient/recipient.key

It checks the checksums, the ledger hash chain and head against the manifest, the signed checkpoints, the object YAML, the sealed Secrets (with --open-key) and pg_restore --list of the dump. It exits 3 on any failure.

$ maqpna backup verify ../backup-2026-10-02
backup of maqpna-system taken 2026-10-03T03:52:19Z (ledger)
ok    checksums                    2 files ok, 0 failed
ok    ledger hash chain            2 records, head seq 2 8ed287bc7026ca7825f171ab57d487f3bbbacff9ed95ef9da5b00b31ad87c2a7

4. Drill the restore#

# Take a fresh backup and drill it
maqpna dr drill --gateway "$GW" --seal-to recipient.pub --open-key recipient.key
# Or drill an existing backup, with a scratch database
maqpna dr drill --from /backups/2026-10-02 --open-key recipient.key --postgres-scratch-dsn-file scratch-dsn.txt

The drill:

1. restores the ledger to a scratch file and requires the identical head hash;
2. validates every object with a server-side dry run in scratch namespaces (maqpna-drill-<namespace>, deleted afterwards unless --keep);
3. opens the sealed Secrets;
4. with a scratch database, restores PostgreSQL and verifies its audit_chain against the ledger head;
5. reports the recovery time (how long the drill took) and the data-loss window (the age of the backup).

Nothing in the live installation changes. It exits 3 when a check fails. Run it monthly, and after every upgrade and key rotation.

5. Restore#

Install the chart first: the CRDs come from the chart, not from the backup. Then:

maqpna restore /backups/2026-10-02 --open-key recipient.key --dry-run
maqpna restore /backups/2026-10-02 --open-key recipient.key \
  --postgres-dsn-file dsn.txt --ledger-out /restore/audit.jsonl
maqpna doctor

A ledger-only restore, then an offline verify of the result:

maqpna restore /backups/2026-10-02 --components ledger --ledger-out restored.jsonl --yes
maqpna audit verify restored.jsonl

OK: 2 records, hash chain intact

Rotate keys without downtime#

maqpna keys rotate rotates each customer-held key. Every key lives in a Secret mounted as a directory; the rotation edits the Secret and restarts the Deployment that reads it. Progress is recorded on the Secret (annotation maqpna.com/key-rotation), so an interrupted rotation can be finished later. Common flags: -n, --dry-run, --timeout (per rollout, default 5m), --secret and --deployment to override the chart names, --gateway to confirm the result through the admin API.

No call fails during an identity rotation: a token is only ever signed with a key the gateways already trust. Keys in a KMS or HSM rotate in the KMS instead. Other keys:

Troubleshooting#

Next steps#

• Audit: verify the ledger and ship it to write-once storage (WORM).
• Monitoring and alerts: the PostgreSQL and ledger runbooks.
• Command reference: maqpna backup create, maqpna backup verify, maqpna restore, maqpna dr drill, maqpna keys rotate.