AI agent control evaluation kit

Download the kit

The checklist and procurement questions are free to reuse under CC BY 4.0. The Python runner uses the standard library and is Apache-2.0 licensed. No account is needed for these materials or the public verifier.

• 14-test control checklistScope, expected observations, evidence, findings, and retest fields · Markdown
• Procurement questionsSeventeen areas to cover in an RFP or technical evaluation · Markdown
• Reproducible export exerciseFour checks against a supplied aaesverify binary · Python 3.9+
• Recorded sample resultsActual CLI outputs, exit codes, input hashes, and limitations · JSON

The blank control checklist reports no completed control tests. The recorded results concern only the separate synthetic export exercise described below.

Define the evaluation before running it

Choose one workflow and an authorized test environment with synthetic data and controlled effects. Name the agent, accountable person, provider operations, credentials, routes, and exclusions. Agree each expected observation and the time window in advance.

Capture what the downstream service actually does. A denial response alone does not prove that no effect occurred; a recorded dispatch alone does not prove success. Mark checks as not run, observed, failed expectation, or inconclusive. Label local simulations and live provider-tenant tests separately.

The method is vendor-neutral in its test questions. AAES is the publisher and a vendor in this category; this is not an independent ranking or a certification.

Fourteen control checks

Use the downloadable checklist for individual test IDs, expected observations, and evidence fields. These are proposed tests for the implementation you evaluate, not claims that every product supports or passes them.

Do not collapse missing or inconclusive observations into a passing aggregate score. Retain the workflow, revision, configuration, evidence hashes, and remaining uncertainties with every finding.

Reproduce the export-integrity exercise

This exercise checks a vendor-supplied synthetic export with the released aaesverify v0.2.0. It runs no agent and calls no downstream provider. It does not test approval binding, bypass prevention, budget enforcement, or complete capture.

1. Download the Python runner, inspect it, and retain the synthetic export and sample public key.
2. Choose the binary for your operating system and architecture from the verification page. Check its checksum manifest and build provenance. These come from AAES-controlled distribution; matching them does not independently authenticate the publisher. Build from the public source if your review requires it.
3. Run the command below with local paths. The runner copies the binary and creates edited inputs in a temporary directory; it checks that the original inputs remain unchanged.
4. Retain your JSON output, compare its input hashes and verifier identity with the recorded exercise, and record any differences. Establish network isolation separately if that is part of your review.

python3 verify-sample.py \
  --verifier /path/to/aaesverify \
  --export /path/to/sample-export.jsonl \
  --pubkey /path/to/sample-pubkey.txt \
  --output my-sample-results.json

The runner returns 0 when all four narrowly specified expectations match, and 1 when an expectation fails. Missing files, an unusable binary, or an execution error are inconclusive; inspect the error before drawing a conclusion.

Recorded sample observations

AAES performed this vendor-run synthetic exercise on October 13, 2026, using Darwin arm64. The full JSON record includes the precise UTC execution time, hashes, outputs, and verifier identity.

All four specified expectations matched, and the runner confirmed that its original input files were unchanged. This is not an independent assessment, adversarial security audit, multi-platform result, or proof of a live deployment.

The released v0.2.0 CLI may call an ordinary successful integrity check PASS. That wording does not establish independence: the sample has no independent witness or timestamp, and its independence-required check fails. A signing-key holder can create a different internally consistent history. Verification does not prove complete capture, truthful inputs, correct decisions, or provider execution.

Turn requirements into procurement questions

The procurement worksheet covers enforcement boundaries, credentials, identity, scope, approval, spending, retries, outage, evidence, independence, outcomes, recovery, integrations, security assessments, operating responsibilities, data flows, and commercial terms.

For each answer, request an exact supported version, configuration prerequisites, known exclusions, and evidence. Distinguish roadmap statements from delivered behavior and vendor-run tests from independent or customer-tenant results. Agree acceptance criteria for your workflow before buying.

Reuse and attribution

Copy and adapt the two worksheets under CC BY 4.0. Plain-text attribution to AAES is sufficient; a backlink is not required. The runner is Apache-2.0 licensed. See the license and scope notice for the individual files.

If you cite the method, identify the kit version and the tests you used. If you publish results, retain the tested scope and label synthetic, vendor-run, independent, and live-tenant observations accurately. Reviewers choose their own conclusions, including unfavorable or inconclusive findings.

Sources and product boundaries

The MCP source below describes the specific 2025-11-25 authorization profile. OWASP Excessive Agency discusses limiting functionality, permissions, and autonomy. This kit is a practical evaluation aid, not a normative mapping or a complete implementation of either source.

• MCP authorization specification, 2025-11-25
• OWASP LLM06:2025 Excessive Agency
• Released verifier source and license
• AAES trust model and evidence limits

AAES Govern applies its controls to supported requests routed through its controlled paths. Direct credentials and alternate routes can bypass that boundary. AAES Operate has separate operational controls. The public kit does not include the proprietary platform evaluation package. Check current availability and the scoped product evaluation for those terms.