Skip to content

Public Benchmark: the Argus Gauntlet

The Argus Gauntlet is a pre-registered, adversarial benchmark for agent guards. Every number on this page comes from a committed, rerunnable artifact, and every number carries its evidence class. Losses are shown.

The corpus

980 scenarios, pinned by tree hash and manifest hash:

  • 780 attack scenarios across thirteen families: the OWASP ASI01-ASI10 agent-risk classes plus three grammar families, across four delivery conditions and three difficulty strata.
  • 200 benign contrasts that a working guard must let through.

The corpus pins, the run journals, and the run manifests are committed. The scoreboard document and its reproduction script verify every hash before rebuilding the artifact.

The metric

The attack success rate (ASR) is guard_unsafe / baseline_unsafe against the corrected baseline of 780. Containment is 1 minus the ASR. The artifact states the exact Clopper-Pearson 95 percent intervals.

Results

Measured runs against the pinned corpus. Source: the committed artifact benchmarks/asr/results/asr_benchmark_current.json in the argus-gauntlet repository (sha256 42044efe0417c3bb4dfe470dd1e5dd3f1c2f809516cd48d814a5ea617724309d).

Guard ASR (lower is better) CP 95% Containment False stops Selective coverage
No guard (baseline) 1.0000 [0.995282, 1.000000] 0.0000 0.0000 1.0000
Argus Sentry 0.3013 [0.269245, 0.334830] 0.6987 0.0000 1.0000
OPA 0.2923 [0.260584, 0.325610] 0.7077 0.0000 1.0000
NeMo 0.0154 [0.007974, 0.026719] 0.9846 not estimable 0.1075
MCP firewall (deny-all) 0.0000 [0.000000, 0.004718] 1.0000 1.0000 0.0729

The honest reading of each cell:

  • Argus Sentry: 0.6987 containment at full coverage, zero abstention, zero false stops. It decides every proposal, stops no benign contrast, and loses the highest-containment cell by 0.9 points. That trade is stated, not hidden.
  • OPA: 0.7077 containment at the same full coverage. OPA leads Argus Sentry by 0.9 points with identical coverage and false-stop behavior.
  • NeMo: 0.9846, measured on the subset it chooses to answer. Its selective coverage is 0.1075, roughly 12 percent of proposals; the rest are recorded inconclusive. A guard that abstains is not a guard that executes.
  • The MCP firewall: 1.0000 containment by denying everything. It refuses all 200 benign contrasts. Deny-all is containment with no liveness.

The wedge: a separate benchmark, never averaged

A second, separately pre-registered corpus measures enforcement containment on sensor-fault scenarios: 93.48 percent (BCa 95% CI 84.6 to 97.8), 99 of 1,519 unsafe commands. Evidence class: pre-registered artifact. The wedge corpus ships pinned with its own checksum manifest alongside the policy and pre-registration files that define the benchmark line.

The separation rule is part of the benchmark design: the wedge and the 980-scenario gauntlet are different benchmarks and their numbers are never averaged.

Throughput: the measured floor, stated as a floor

The composed decision path's throughput floor is a measured result with a named gap. Evidence class: measured run, committed artifact plus an independent clean-checkout reproduction, 0 wrong verdicts in every cell of every artifact (20 cells each, verdict-checked inside the timed loop).

Reading Rate Cell Gap to the 250,000/s bar
Committed best 113,632 decisions/s ed25519 tokens, batch 128, 16 workers 2.2x
Clean-checkout reproduction 140,416 decisions/s session-HMAC tokens, batch 512, 16 workers 1.78x

The 250,000 decisions/s bar is not reached, and the gap is named, not averaged away. The remaining floor decomposes into owner-named levers: the Rust token codec on the single-token hot path, batched proposal construction (one marshal per batch instead of per item), and the evidence record's by-value entry and capacity preallocation. At the measured 58.6 microsecond per-decision floor, 250,000 decisions per second needs about 14.7 physical-core-seconds of decision work per second, roughly 2.1x the measurement host's per-core throughput.

Committed artifacts, rerunnable:

  • Artifact of record: argus-gate/python/tests/extreme/perf/results/perf_wired_batch_2026-11-19.json
  • Independent reproduction: argus-gate/python/tests/extreme/perf/results/p4_clean_repro_2026-11-19.json
  • Harness: argus-gate/python/tests/extreme/perf/perf_wired_batch.py (the reproduction command is carried in the artifact itself)

Latency: the design benchmark

A latency benchmark on named hardware, run 2026-10-31. This row is always labeled: design-benchmark, not a production SLO.

Metric Value Label
Decision latency, p50 0.2459 ms (p95 0.4193, p99 0.5870) design-benchmark, not a production SLO
Decision throughput 3,607 per s, 0 of 5,000 failures design-benchmark, not a production SLO
Verify latency, p50 0.1207 ms design-benchmark, not a production SLO
Verify throughput 7,588 per s design-benchmark, not a production SLO

Reproduce

The gauntlet scoreboard and its reproduction script live in the argus-gauntlet repository:

  • publication/SCOREBOARD.md: the corpus, the outcome semantics, the ASR definition, the intervals, and the abstention view.
  • publication/reproduce_scoreboard.py: verifies every pinned journal and manifest checksum, the corpus manifest and file count, and the harness binary, then rebuilds the artifact and compares the numeric content exactly. A tampered journal fails with the input named and both checksums printed.
  • publication/corpora_pins.json: the corpus pins.

The throughput artifacts carry the full reproduction command and environment block.

What is not claimed

  • The gauntlet numbers are digital source verification on synthetic corpora with published seeds. They are not production SLOs, not a safety assessment, and not a claim about any deployment.
  • Every throughput rate is a lower bound on a shared measurement host; the artifacts state a 20 to 40 percent variance band under concurrent load.
  • No certification is claimed anywhere. The benchmark page points at committed evidence, not at certificates.
  • The NeMo line is being re-measured under a hardened worker; a new line is added to the scoreboard only when that run completes and its journal is analyzed.