Observability#

MAQPNA is observable at three levels that fit together: metrics (Prometheus text on /metrics), traces (OpenTelemetry, off by default and never carrying contents) and evidence (the audit ledger and the session timeline). Every ledger record of a traced request carries ext.traceId and ext.spanId, so you can go from a dashboard spike to a trace to the exact audit records, and back.

flowchart LR
    OP["operator :8080/metrics"] --> PR[(Prometheus)]
    GW["gateway /metrics<br/>(admin listener when set)"] --> PR
    AT["attestation :8082/metrics"] --> PR
    PR --> AL["PrometheusRule<br/>maqpna alerts"]
    PR --> GR["Grafana dashboards<br/>Sessions, Gateway, Evidence"]
    GW -- "OTLP/HTTP JSON<br/>(otel.enabled)" --> OC["OpenTelemetry collector"]
    OC --> TB["Tempo or Jaeger"]
    GW -- "ext.traceId on every record" --> L[("audit ledger")]
    L --> TL["timeline API<br/>traceUrl from traceURLTemplate"]
    TL -. "deep link" .-> TB

Metrics#

Operator#

Metric	Labels	Meaning
`maqpna_session_ready_seconds`	`tier`, `mode` (`direct`, `claim`)	Time from session creation to first `Running`
`maqpna_sessions`	`namespace`, `tier`, `phase`	Sessions by phase (leader only)
`maqpna_session_phase_transitions_total`	`from`, `to`, `reason`	Phase changes
`maqpna_session_outcomes_total`	`tier`, `result`	Results when sessions end
`maqpna_token_mint_seconds`	—	Identity broker mint latency
`maqpna_warm_pool_ready`	`tier`	Ready warm sandboxes
`maqpna_reconcile_errors_total`	`controller`, `reason`	Reconcile errors
`maqpna_reconcile_duration_seconds`	`controller`	Reconcile latency
`maqpna_rendered_configmap_bytes`	`name`	Size of each rendered gateway ConfigMap (limit 1 MiB)
`maqpna_billable_nodes`	`period` (`current`, `day`, `month`)	Nodes running at least one session pod
`maqpna_license_state`, `maqpna_license_days_left`, `maqpna_license_node_limit`	`edition`, `state`	Licence status (reported only)
`maqpna_build_info`	version labels	Build information

Gateway#

Area	Metrics
Requests and decisions	`maqpna_gateway_requests_total{server,method,code}`, `maqpna_gateway_decisions_total{action,server}`, `maqpna_gateway_auth_failures_total{reason}`, `maqpna_gateway_upstream_latency_seconds{server}`, `maqpna_gateway_upstream_errors_total{server}`, `mcp_server_operation_duration_seconds{mcp_method_name,error_type}`
Policies	`maqpna_gateway_policy_reloads_total{result}`, `maqpna_gateway_policy_revision`, `maqpna_gateway_policies_loaded`, `maqpna_gateway_dryrun_decisions_total{policy,would}`, `maqpna_gateway_cedar_available`, `maqpna_gateway_rego_evaluations_total`
Approvals	`maqpna_gateway_approvals_pending`
Kill switch	`maqpna_gateway_revoked_total{scope}`, `maqpna_gateway_revocations_active{source}`, `maqpna_gateway_revocation_list_ok`
DLP, taint, pins, guards	`maqpna_gateway_dlp_hits_total{detector,direction,action}`, `maqpna_gateway_taint_events_total{op,label}`, `maqpna_gateway_tainted_sessions`, `maqpna_gateway_tool_pin_events_total{server,event}`, `maqpna_gateway_guard_checks_total`, `maqpna_gateway_guard_blocked_total`, `maqpna_gateway_guard_latency_seconds`
Budgets and models	`maqpna_gateway_budget_denied_total{scope}`, `maqpna_gateway_budget_alerts_total{scope,percent}`, `maqpna_gateway_rate_limited_total`, `maqpna_gateway_metered_cost_usd`, `maqpna_gateway_model_requests_total{route,status}`, `maqpna_gateway_model_tokens_total{route,kind}`, `maqpna_gateway_model_failovers_total`, `maqpna_gateway_model_circuits_open`, `gen_ai_client_token_usage`
Audit	`maqpna_gateway_audit_errors_total`, `maqpna_gateway_audit_commit_seconds`, `maqpna_gateway_audit_committed_records_total`, `maqpna_gateway_audit_head_seq`, `maqpna_gateway_audit_signed_checkpoint_seq`, `maqpna_gateway_audit_sink_lag{sink}`, `maqpna_gateway_audit_sink_delivered_total`, `maqpna_gateway_audit_sink_errors_total`, `maqpna_gateway_audit_retention_below_ai_act`
HA and config	`maqpna_gateway_state_backend_info{type}`, `maqpna_gateway_leader{job}`, `maqpna_gateway_configsync_last_success_timestamp_seconds{configmap}`, `maqpna_gateway_configsync_errors_total{configmap}`, `maqpna_gateway_trusted_keys`
Other features	`maqpna_gateway_a2a_calls_total{hop,kind,decision}`, `maqpna_gateway_svid_checks_total{result}`, `maqpna_gateway_egress_requests_total`, `maqpna_gateway_exec_total`, `maqpna_gateway_web_calls_total`, `maqpna_gateway_otel_spans_{exported,dropped,failed}_total`

Attestation service#

maqpna_attest_success_total, maqpna_attest_failure_total{reason}, maqpna_attest_releases_created_total, maqpna_attest_releases_pending, maqpna_attest_renewal_success_total, maqpna_attest_renewal_failure_total{reason}, maqpna_attest_renewals_active.

Alerts#

With monitoring.prometheusRule.enabled the chart renders a PrometheusRule; thresholds live under monitoring.prometheusRule.thresholds and any alert can be switched off with disabled: [<name>]. Each alert links to a runbook anchor.

Alert	Severity	Fires when (default threshold)
`MaqpnaSessionReadySLO` (two rules)	warning	p90 session start above 1 s for warm-pool sessions, or 5 s for direct sessions, over 15 minutes
`MaqpnaSessionFailureRate`	warning	More than 25% of ended sessions failed in 30 minutes (at least 5 sessions)
`MaqpnaWarmPoolEmpty`	warning	A tier's warm pool has had no ready sandbox for 10 minutes
`MaqpnaTokenMintSlow`	warning	p99 token mint above 1 s
`MaqpnaOperatorReconcileErrors`	warning	Reconcile errors (conflicts excluded) above 0.05 per second
`MaqpnaConfigMapNearLimit`	warning	A rendered ConfigMap above 800 KiB
`MaqpnaGatewayAuditCommitSlow`	warning	p99 audit group commit above 50 ms
`MaqpnaAuditLedgerUnavailable`	critical	Any audit append error in 5 minutes
`MaqpnaGatewayNotReady`	critical	No ready gateway pod for 5 minutes (needs kube-state-metrics)
`MaqpnaApprovalsBacklog`	warning	More than 50 pending approvals for 10 minutes
`MaqpnaRevocationListUnavailable`	critical	A replica cannot read the revocation list (it is denying every call) for 10 s
`MaqpnaRevocationLag`	critical	The last successful read of the revocations ConfigMap is older than 10 s for a minute
`MaqpnaConfigSyncFailures`	warning	A policy reload failed in the last 10 minutes
`MaqpnaGatewayDenyRateHigh`	warning	More than half of a namespace's calls are denied
`MaqpnaGatewayUpstreamErrors`	warning	More than 5% of upstream round trips fail
`MaqpnaBudgetExceeded`	info	A budget denied calls in the last 15 minutes
`MaqpnaModelCircuitOpen`	warning	A model endpoint's circuit has been open for 10 minutes
`MaqpnaAuditSinkLag`	warning	More than 1000 records not yet delivered to a SIEM sink
`MaqpnaAttestationFailureRate`	warning	More than 10% of attestations fail
`MaqpnaLicenseExpiring` (two rules)	warning, critical	Licence expires in under 30 days, or under 7 (agent traffic is never affected)
`MaqpnaLicenseNodeLimitExceeded`	info	This month's billable nodes exceed the licence's node limit

hack/check-alerts.sh unit-tests the rules with promtool.

Dashboards and scraping#

monitoring.dashboards.enabled renders the Sessions, Gateway and Evidence Grafana dashboards as ConfigMaps labelled for the Grafana sidecar.
serviceMonitor.enabled renders a ServiceMonitor (it fails if the monitoring.coreos.com/v1 API is missing) that scrapes the operator, the gateway (on the admin port when adminListen is set) and the attestation service, every 30 seconds by default.

Traces#

pkg/otel is a standard-library tracer: W3C Trace Context and Baggage, a parent-based ratio sampler and a batching OTLP/HTTP JSON exporter to <otel.endpoint>/v1/traces.

Off by default. With otel.enabled: false the gateway only propagates the caller's trace context to upstreams; it creates no spans and exports nothing.
Context in. Read from MCP params._meta.traceparent (and tracestate, baggage), else from HTTP headers. Trace context is never used for authorisation.
Spans. A server span per request (POST /mcp/{server}, /llm/..., /a2a/...), an operation span (tools/call github), children policy.evaluate and approval.wait, and a client span to the upstream; model calls add chat <model> with gen_ai.* token usage.
Attributes. Governance facts only: maqpna.decision, maqpna.policy, maqpna.rule, maqpna.revocation.id, maqpna.dlp.hits (detector counts), maqpna.taint, maqpna.budget.exceeded, and error.type on denials. Spans never carry prompts or arguments; recordArgs adds only their hash and size, recordUser the user ID.
Context out. The gateway injects its span context into the upstream request and writes ext.traceId/ext.spanId to every ledger record of the request, even with export off.
Export. A bounded queue (2048 spans; drops are counted, never blocking), batches of 512 every 5 seconds, retries with backoff; the collector is reached through the residency-checking dialer.

sessions.traceURLTemplate (for example a Grafana Explore URL with {traceId}) turns ext.traceId into a traceUrl on each timeline item.

Logs and posture#

Components log structured lines to stdout; failing and warning posture checks are logged at gateway start-up. GET /v1/posture and maqpna doctor report checks such as admin-auth-mode, audit-failure-policy and state-backend, each with a fix and a score from 0 to 100.

What you see#

A scrape of the gateway (illustrative values):

maqpna_gateway_decisions_total{action="allow",server="github"} 18412
maqpna_gateway_decisions_total{action="deny",server="github"} 37
maqpna_gateway_approvals_pending 2
maqpna_gateway_revocation_list_ok 1
maqpna_gateway_leader{job="worm/audit"} 1

See maqpna status, maqpna doctor and maqpna audit sinks.