HA and failure handling#

MAQPNA's rule for failures is simple to state: if a control cannot be checked, the call is denied — with a short, explicit list of exceptions where availability was chosen on purpose and the choice is configurable. This page lists every component's replica model and every dependency's failure behaviour, as implemented.

Replicas#

PodDisruptionBudgets. The gateway PDB (minAvailable: 1) is rendered whenever gateway.pdb.enabled is true, which is the default, whatever the replica count. With the default single replica this blocks voluntary evictions such as node drains; run at least two replicas, or disable the PDB, before draining nodes. The attestation PDB is rendered only with more than one replica. The identity broker and operator have no PDB.

With more than one gateway replica, the chart adds a preferred pod anti-affinity so replicas spread across nodes.

Background jobs and leaders#

Gateway jobs that must run once per installation are guarded by PostgreSQL advisory-lock leases (see state backend schema): siem/<sink>, worm/<kind>, tenant-ledgers, memory-sweep, capture-sweep. Shipping positions are shared blobs, so a new leader resumes where the old one stopped. Delivery is at least once at the edges; receivers can de-duplicate on the record hash.

sequenceDiagram
    participant Agent
    participant LB as Gateway Service
    participant A as Replica A
    participant B as Replica B
    participant PG as PostgreSQL
    Agent->>LB: tools/call (held for approval, sync)
    LB->>A: route to A
    A->>PG: approvals journal: create apr_1
    Note over A: A is killed
    PG-->>PG: A's lease session ends, its locks are released
    B->>PG: takes leases siem/ocsf and worm/audit
    Agent->>LB: retry with X-Maqpna-Async: 1 (the TCP connection was lost)
    LB->>B: route to B
    B->>PG: create apr_2 and return -32002
    Note over B: an approver approves apr_2 on any replica
    Agent->>LB: retry with X-Maqpna-Approval-Id: apr_2
    LB->>B: consume apr_2 under the journal lock, then forward
    B->>PG: append to audit_chain (one chain, advisory lock)

Walkthrough:

1. A replica dies. Kubernetes removes it from the Service; in-flight requests on it fail and agents retry them (tool calls are retried by the agent, and the SDKs surface the error).
2. Its PostgreSQL sessions end, releasing its leases at once (or within about 8 seconds if the host vanished without a TCP reset).
3. A standby takes each lease at its next retry (leader.retryMillis, 1 second) and resumes shipping from the saved cursor.
4. Pending approvals, revocations and taint are in the shared journals, so any surviving replica serves them. A caller blocked on a synchronous approval on the dead replica lost its connection and must retry; the approval itself is not lost.
5. The ledger is one chain: every replica appends under the same advisory lock and keeps a verified local mirror.

Failure modes#

What you see#

maqpna doctor and GET /v1/posture evaluate these settings as posture checks. A gateway running three replicas on the file backend fails the state-backend check (text from pkg/posture/gateway.go):

{
  "id": "state-backend",
  "component": "state",
  "title": "Shared state backend for HA replicas",
  "severity": "fail",
  "status": "fail",
  "detail": "stateBackend.type=file with 3 gateway replicas: approvals, revocations, taint, budgets and audit ledgers diverge per replica",
  "fix": "set state.backend: postgres (docs/state-backend.md) or run one gateway replica"
}

Each check also carries a source, and the report has a score from 0 to 100; maqpna doctor exits 3 when a check fails. See maqpna doctor and maqpna status.

Related#

• State backend
• State backend schema and leader election
• Deployment topologies
• Observability