`maqpna eval`

Score the policy decisions of an evaluation suite

Govern-o json | yaml

Synopsis#

maqpna eval --suite SUITE.yaml [--policy FILE | --gateway URL [--agent-token-file F]] [--mock-listen ADDR]
            [-o table|json|yaml] [--min-score 1.0]       (exit 3 below --min-score)

Description#

Runs the suite's tasks (ordered tool calls of one simulated session each) and scores the policy outcomes against each step's expect / expectRule. Exit status: 0 score >= --min-score, 3 below, 1 error.

Modes:

  local     --policy FILE (or suite "policy:"): pkg/policy engine, offline
  evaluate  --gateway URL: the gateway's live policies via POST /v1/policies:evaluate
            (admin token: --token / --oidc-token-file / env)
  live      --gateway URL --agent-token-file F: real tools/call through
            /mcp/{server} with an agent token (X-Maqpna-Async: 1); all tasks
            share the token's session

In local and evaluate mode session taints are simulated like the gateway does: after an allowed step the tool's toolLabels taints are added and, when its (mock) response is not a JSON-RPC error, its clearTaints removed. --mock-listen ADDR serves the suite's mocks as an MCP server (POST /mcp/{server}, /mcp) for gateways whose upstreams point at it.

Flags#

Flag	Type	Description	Default
`--agent-token-file`	string	agent token file: live mode (tools/call through the gateway)	none
`--format`	string	table\|json (same as -o)	`table`
`--gateway`	string	gateway base URL (env MAQPNA_GATEWAY_URL; default: the context's gateway)	none
`--min-score`	float	exit 3 when the score is below this (0..1)	`1`
`--mock-listen`	string	serve the suite mocks as an MCP server on ADDR while running	none
`--oidc-token-file`	string	file holding an OIDC access token for the admin API (env MAQPNA_OIDC_TOKEN_FILE); wins over `--token`	none
`--policy`	string	policies for local mode (overrides the suite's policy)	none
`--suite`	string	suite file (YAML or JSON)	none
`--token`	string	static admin token (env MAQPNA_ADMIN_TOKEN; dev/break-glass)	none

The global flags (--context, -o, --no-color, ...) work with every command.

Examples#

$ maqpna eval --suite suite.yaml --policy policies/
$ maqpna eval --suite suite.yaml --gateway http://127.0.0.1:8080 --min-score 0.95

What happens when you run it#

Talks to the gateway: --gateway, else MAQPNA_GATEWAY_URL, else the current context's gateway (maqpna context).
Authenticates to the admin API with the token stored by maqpna login, --oidc-token-file, or a static --token (MAQPNA_ADMIN_TOKEN).
Prints a table by default; -o json or -o yaml print the data, and --jq EXPR filters the JSON.
Exits 3 when the check fails or a result does not match (see exit codes below), so scripts and CI can act on it.

Exit codes#

Code	Meaning
`0`	success
`1`	error (the message says what failed, with a hint when there is one)
`2`	usage error: unknown flag, missing argument or bad value; the synopsis is printed
`3`	a check failed, a change is blocked, or a result did not match (tamper, policy mismatch)

Terminal demo#

maqpna eval.cast