A small policy change ships on a Thursday. The release pipeline runs every test, and everything is green: the assigned underwriter u-uw-w1 can record a decision on SUB-1004, the broker u-broker-n1 can read Q-5531. Every test asked "can the right person do the right thing?", and every answer was yes.
No test asked the opposite question. The new version had lost the assignment check on decision.record, so the peer underwriter u-uw-w2, same unit, not assigned, could now decide SUB-1004. This site's own case UW-A09 records exactly this drift between authz-2026.11.1 and authz-2026.10.3. Positive tests cannot see it: a policy that allows too much still allows everything they ask for.
This page is about the other half of the suite: deciding what must never happen, at which boundary, and writing the test that fails the day it does.
Threat models and negative tests, in plain words#
A threat model is a written list of where a system can be attacked, what could go wrong there, what stops it, and how you know it works. A tiny example: "at the quote API, a token issued for another service could be presented; the API checks the audience; test UW-A05 proves the check."
The places worth listing are the trust boundaries: points where a request leaves one principal's control and enters another's (the vocabulary is on Primitives). The broker's browser to the portal backend is one; the agent application to the tool server is another.
A negative test sends a request that must be refused and passes only when it is refused and nothing changed. The useful kind keeps every input genuine except one. Broker u-broker-s1 gets a real, valid token and asks for Q-5531, which belongs to brk-north; the test expects 404 and no body. With a broken token it would only test authentication.
Before AI. Spring teams have written these for years: a MockMvc test with @WithMockUser(roles = "BROKER") that calls an admin endpoint and expects 403, or a test that reads another customer's order and expects 404. The agent adds new boundaries (the token service, the tool server, the model's output); the discipline is the same.
Guideline (OWASP Authorization Cheat Sheet, read 2026-11-18): automated access-control tests "can help reduce the number of security flaws that make it into production" and should ask "is access being denied by default?"; they catch the "low-hanging fruit" and do not replace a skilled manual tester.
STRIDE: six questions per boundary#
Listing threats from memory finds the ones you already thought of. STRIDE is a checklist of six categories to ask about at each boundary.
Pattern (STRIDE, as described by Microsoft on 2026-11-18): Microsoft uses it in its Threat Modeling Tool to categorize threats; the table quotes or follows that page.
| Letter | Category | Microsoft's description (short) | The property it attacks |
|---|---|---|---|
| S | Spoofing | using another user's authentication information | authentication |
| T | Tampering | "the malicious modification of data", stored or in transit | integrity |
| R | Repudiation | users deny an action and nobody can prove otherwise | accountability (audit) |
| I | Information disclosure | "the exposure of information to individuals who are not supposed to have access to it" | confidentiality |
| D | Denial of service | attacks that "deny service to valid users" | availability |
| E | Elevation of privilege | "An unprivileged user gains privileged access" | authorization |
The fourth column is this site's mapping, not Microsoft's wording.
The trust boundaries of the insurance flow#
The diagram keeps one threat per boundary; the table after it is the working threat model. The last column names the test or case that proves each control.
Diagram: Trust boundaries of the insurance flow · the worst STRIDE threat at each, its control, its test
Each row is one trust boundary: a place where a request crosses from one principal's control into another's. Read across: the threat that matters most there, the control that stops it, and the negative test or case that proves the control works. The full table on the page has more rows; this view keeps one per boundary.
- Broker browser to portal
- S: stolen session or forged post: Spoofing the broker: a script reads a token in the browser, or another site posts a form with the broker's cookie.
- HttpOnly cookie, tokens stay server-side: The BFF keeps tokens on the server; the cookie is HttpOnly, Secure, SameSite; state-changing requests need a CSRF defense.
- session lab outcome table: The recorded outcomes of XSS and cross-site post against both designs.
- Portal BFF to quote API
- S, T: substituted or edited token: A genuine token for another audience, an ID token, an edited scope, alg none.
- ordered validation: alg, kid, signature, iss, aud, exp: The resource server validates the token in a fixed order and trusts no claim before the signature verifies.
- UW-A05 to UW-A08: Wrong audience, expired, tampered and alg none all rejected at the first failing check.
- Quote service to database
- I: another tenant's row: A valid caller asks for a record id that belongs to another brokerage.
- PDP on the record + repository predicate: The service decides on the loaded record; the repository adds the tenant from the verified subject.
- cross-tenant matrix: Every principal with a genuine token against every other tenant's or unit's record, by both ids.
- Agent application to tool server
- E: agent asks for uw.decide: A prompt or a bug makes the agent application ask for a power no agent may hold.
- token exchange cap + R-AZ-09: The token service never issues uw.decide to an agent; the PDP refuses decisions by any non-person even if the scope slipped through.
- UW-A10, UW-A22: Exchange refused; misgranted scope still refused.
- Cockpit to policy decision point
- E, D: peer decides; PDP down: A policy version without the assignment check, or a PEP that allows when the PDP does not answer.
- assignment check, fail closed (503): Rule R-AZ-09 checks assignment on the record; the PEP refuses when it cannot get a decision.
- peer row + outage test: A negative row for the peer underwriter under every policy version shipped; a test with the PDP made unavailable.
- Services to audit log
- R: edited record; I: token in a log: Repudiation by editing a deny into an allow; disclosure by writing a bearer token into a log line.
- hash chain + redaction: Each record carries the hash of the previous one; tokens and keys are never written.
- UW-A24: Scan every audit record for token or key material; verify both chains.
| Boundary | STRIDE | Threat in this flow | Control | Proved by |
|---|---|---|---|---|
| broker browser to portal BFF | S | script steals a token; cross-site form post | tokens kept at the BFF, HttpOnly cookie, CSRF defense | session lab outcome table |
| portal BFF to quote API | S | token for another audience, or an ID token, presented | audience check before any claim is used | UW-A05; substitution run below |
| portal BFF to quote API | T | scope edited, alg set to none, signed with a guessed key | algorithm allowlist, signature before claims | UW-A07, UW-A08; token lab |
| portal BFF to quote API | S | expired or revoked token replayed | exp, deny list or introspection | UW-A06, UW-A16 |
| gateway to quote service | I | valid token, another tenant's quote (BOLA) | PDP decides on the record, not the route | UW-A03; test_service_hop_decides_on_the_record_not_the_token |
| caller to quote service | S | forged x-app-id: svc-cockpit-bff header | peer identity from mTLS, not a header | UW-A04; test_x_app_id_header_is_not_identity |
| quote service to database | I | forgotten check reads another tenant's row | repository predicate from the verified subject | test_repository_hop_applies_the_tenant_predicate_even_without_a_pep_check |
any caller to decision.record | E | broker, auditor or agent calls the decision function (BFLA) | R-AZ-02 scope, R-AZ-09 person and role | test_auditor_cannot_write; matrix below |
| agent application to token service | E | agent asks for uw.decide | token exchange caps agent scopes | test_agent_cannot_obtain_uw_decide_by_token_exchange |
| agent application to tool server | E | agent acts beyond its delegated task (confused deputy) | task names one submission, actions and expiry | UW-A11; agent run step 7 |
| cockpit to PDP | E | peer underwriter decides | assignment check in R-AZ-09 | UW-A09 |
| cockpit to PDP | T | decision written on an answer that is no longer true (race) | re-check at write, or one transaction | TOCTOU run below |
| cockpit to PDP | D | PDP unreachable and the PEP allows | fail closed: 503, audit, alert | UW-A23; test_pdp_outage_fails_closed_for_writes |
| services to audit log | R | a deny edited into an allow | hash chain over records | UW-A24; incident lab |
| services to audit log | I | a bearer token written to a log | redaction; scan for token-shaped strings | test_audit_log_never_contains_a_token_or_key |
| identity governance to directory | E | leaver keeps access | deactivation, session revoke, short token life | test_leaver_loses_access_and_session_at_once |
Most rows are E or I: authentication is usually a library, authorization is usually your code, and your code is where it breaks. A row whose last column is empty is the next test to write.
The smallest negative test#
The smallest useful negative test changes one fact and keeps everything else valid, even generous. This one, from examples/uw-auth/test_negative.py, gives the underwriting agent everything a decision needs: scope, entitlement, role, assignment, a fresh strong sign-in, the cockpit as client. Only the principal type is wrong.
def test_agent_decision_refused_even_with_misgranted_uw_decide(self) -> None:
bad = agent_subject(scopes=["uw.read", "uw.recommend", "uw.decide"], entitlements=["uw.decide"], roles=["underwriter"],
assignments=["SUB-1004"], acr=ACR_STRONG, auth_time=NOW, client_id="svc-cockpit-bff")
for v in (pdp.CURRENT_POLICY, pdp.PREVIOUS_POLICY):
r = pdp.decide(bad, "decision.record", resource_for("SUB-1004", "submission"), CTX, v)
self.assertEqual((r["allow"], r["rule"]), (False, "R-AZ-09"), v)
It checks both policy versions, because you may roll back. It asserts the rule that refused, so a refusal for the wrong reason does not pass. The model recommends; a person decides, and this test makes that sentence true in code.
The negative tests, attack by attack#
Each part explains an attack only as far as the control needs. The tokens are demo tokens signed with keys that say they are not secrets; use a maintained JWT library and a real identity provider in production. The agent here is a scripted stand-in, so these results show how the access controls work, not how good any model is.
Tampered, expired and revoked tokens#
An attacker edits the payload and keeps the old signature, sets alg to none, signs with a guessed key, or replays an expired or revoked token. The controls are the ordered checks of Tokens: an algorithm allowlist, the signature before any claim is read, then iss, aud, exp, nbf and revocation.
Standard (RFC 8725, read 2026-11-18): "Libraries MUST enable the caller to specify a supported set of algorithms", and RFC 7518 says implementations "MUST NOT accept Unsecured JWSs by default".
Guideline (OWASP API2, read 2026-11-18): an API is vulnerable if it "Accepts unsigned/weakly signed JWT tokens" or does not validate expiry.
The token lab opens on the attack presets: the validator stops at the first failing check, and authorization is never reached.
| Attack preset | Validation result | First failed check says | quote.read Q-5531 |
|---|---|---|---|
| wrong-aud | rejected at aud | aud is https://assessment-svc.internal.example; this verifier accepts https://api.uw.insurer.example | not reached |
| expired | rejected at exp | exp 1794993300, now 1794993301: expired or missing | not reached |
| tampered (scope edited, old signature kept) | rejected at signature | HMAC-SHA256 with demo key demo-2026a does not match | not reached |
| alg-none | rejected at alg | alg none is not on the allowlist (HS256) | not reached |
| forged-key | rejected at signature | HMAC-SHA256 with demo key demo-2026a does not match | not reached |
| revoked | rejected at revocation | jti jti-1d180a7211eb is revoked | not reached |
| valid (control) | valid | (all checks pass) | 200 (R-AZ-03) |
The control row matters: a validator that rejects every token passes all six attack rows.
Token substitution: the right token in the wrong place#
A token substitution presents a genuine, unmodified token at a recipient it was not issued for; the signature is perfect. Two forms matter here: an assessment-service token at the quote API, and the portal's ID token (which tells the portal backend who signed in) used as an access token.
Standard (RFC 8725, read 2026-11-18): section 2.7 describes attacks in which a recipient given a JWT "will attempt to use it at a different recipient for which that JWT was not intended", and section 3.9 says that when one issuer serves several relying parties, "the JWT MUST contain an "aud" (audience) claim", which the recipient validates. The snippet, written for this page, demonstrates the quote API's validator on each token; a successful run proves the audience check refuses both substitutions in this example.
import demo_keys
import oauth
import tokens
from clock import START_EPOCH_S as NOW
api = {"keyset": demo_keys.keyset(), "expected_issuer": tokens.ISSUER, "expected_audience": tokens.API_AUD, "now": NOW}
candidates = {
"access token for the quote API": tokens.issue_for("u-uw-w1", aud=tokens.API_AUD, now=NOW),
"access token for the assessment service": tokens.issue_for("u-uw-w1", aud=tokens.ASSESSMENT_AUD, now=NOW),
"ID token for the portal BFF": tokens.sign(oauth.auth_code_trace(NOW)["id_token_claims"]),
}
for name, tok in candidates.items():
r = tokens.verify(tok, **api)
detail = next((c["detail"] for c in r["checks"] if c["name"] == r["first_failed"]), "all checks pass")
print(f"{name:40} valid={r['valid']!s:5} first_failed={r['first_failed']} | {detail}")
Run in a scratch copy of examples/uw-auth on 2026-11-18:
access token for the quote API valid=True first_failed=None | all checks pass
access token for the assessment service valid=False first_failed=aud | aud is https://assessment-svc.internal.example; this verifier accepts https://api.uw.insurer.example
ID token for the portal BFF valid=False first_failed=aud | aud is svc-portal-bff; this verifier accepts https://api.uw.insurer.example
Diagram: Token substitution · two genuine tokens, the wrong place, refused at the audience check
Time runs down the page. Both tokens are correctly signed by the identity provider. The first was issued for the assessment service; the second is the portal's ID token, issued so the portal backend knows who signed in. Presented at the quote API, both fail the same check: the audience is not this API. Only the third request, with an access token for the quote API, reaches the policy decision point.
- Caller with a stolen or reused token to Quote API (resource server): Bearer: assessment-service token. A genuine access token whose aud is https://assessment-svc.internal.example.
- Quote API (resource server) to Caller with a stolen or reused token: 401: rejected at aud. Signature, issuer and expiry pass; the audience does not name this API.
- Caller with a stolen or reused token to Quote API (resource server): Bearer: portal ID token. An OpenID Connect ID token, aud svc-portal-bff, used as if it were an access token.
- Quote API (resource server) to Caller with a stolen or reused token: 401: rejected at aud. Same check, same answer. An RFC 9068 typ check (at+jwt) would refuse it a second way.
- Caller with a stolen or reused token to Quote API (resource server): Bearer: access token for the API. The right token for this recipient.
- Quote API (resource server) to Policy decision point: decide quote.read. Now, and only now, the record is decided on.
The demo validator relies on the audience alone. Standard (RFC 9068, read 2026-11-18): JWT access tokens carry typ at+jwt, and a resource server rejects a token whose typ is not that value. That second check refuses an ID token even in a deployment where a careless configuration gave it the API's audience. The demo tokens carry typ JWT and the example does not check it; a production validator should.
BOLA and BFLA: the wrong record, and the wrong function#
Two different holes share one cause: the check ran on the token instead of on the request. Guideline (OWASP API Security Top 10 2023, read 2026-11-18) lists them as API1, Broken Object Level Authorization, and API5, Broken Function Level Authorization.
BOLA (also called IDOR) is the right function on the wrong record: broker u-broker-n1, with a valid token and the scope quotes.read, asks for Q-6620 of brk-south. The gateway-only design returns it (UW-A03); the service check answers 404. The full explanation, with the hops lab, lives on Enforcement.
BFLA is the wrong function: "legitimate API calls to an API endpoint that they should not have access to" (OWASP API5). Here, a broker or the auditor calling decision.record, or the agent calling the tool record_underwriter_decision. OWASP's advice is to "deny all access by default, requiring explicit grants to specific roles". pdp.decide refuses each at a named rule: R-AZ-02 for a missing scope, R-AZ-09 for anyone who is not a person with the underwriter role.
Common confusion. Hiding the decision button for brokers is not a BFLA control. The cockpit's UI is a convenience; the check that counts runs on the server, for every caller, including ones that never load the UI.
Cross-tenant access: a matrix you can add to your suite#
Cross-tenant tests are repetitive and easy to leave half-done. Tenants lists the paths to cover; this is the shape that keeps them complete: a table of rows, one test that walks it, and unittest's subTest so one failing row does not hide the others. Standard library only; copy it next to test_negative.py.
"""A cross-tenant and cross-function matrix against pdp.decide (written for the threat-testing page)."""
import unittest
import pdp
from clock import START_EPOCH_S as NOW
from directory import resource_for, subject_for
CTX = {"now": NOW}
# principal, action, record, expected status. Every subject is genuine; only the record or the function is wrong.
DENY = [
("u-broker-s1", "quote.read", "Q-5531", 404), # other tenant, by quote id
("u-broker-s1", "quote.read", "SUB-1004", 404), # the same record, by submission id
("u-broker-s1", "quote.submit", "Q-5532", 404), # other tenant, a write
("u-broker-n1", "quote.read", "Q-6620", 404), # the other direction
("u-uw-e1", "submission.read", "SUB-1004", 404), # other business unit
("u-uw-w1", "submission.read", "SUB-2001", 404),
("u-broker-n1", "decision.record", "SUB-1004", 403), # BFLA: a function brokers never call
("u-auditor-1", "decision.record", "SUB-1004", 403), # BFLA: read-only role, write function
]
# Control rows: if these fail, the suite is testing a PDP that denies everything.
ALLOW = [
("u-broker-n1", "quote.read", "Q-5531"),
("u-broker-s1", "quote.read", "Q-6620"),
("u-uw-w1", "submission.read", "SUB-1004"),
]
class CrossTenantMatrix(unittest.TestCase):
def test_every_cross_boundary_row_is_refused(self) -> None:
for principal, action, record, expected in DENY:
with self.subTest(principal=principal, action=action, record=record):
d = pdp.decide(subject_for(principal, now=NOW), action, resource_for(record), CTX)
self.assertFalse(d["allow"], d["reason"])
self.assertEqual(d["status"], expected, d["reason"])
def test_control_rows_are_allowed(self) -> None:
for principal, action, record in ALLOW:
with self.subTest(principal=principal, action=action, record=record):
self.assertTrue(pdp.decide(subject_for(principal, now=NOW), action, resource_for(record), CTX)["allow"])
if __name__ == "__main__":
unittest.main()
Run in the scratch copy on 2026-11-18 (/opt/homebrew/bin/python3 -m unittest test_cross_tenant_matrix -v):
test_control_rows_are_allowed (test_cross_tenant_matrix.CrossTenantMatrix.test_control_rows_are_allowed) ... ok
test_every_cross_boundary_row_is_refused (test_cross_tenant_matrix.CrossTenantMatrix.test_every_cross_boundary_row_is_refused) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
Three choices carry the value. The expected status is in each row, so a 403 (which confirms the record exists) fails as surely as a 200. The same record appears as Q-5531 and as SUB-1004, because authorization is decided on the record both ids point to. And the control rows catch the cheapest false pass: a policy that denies everything. Guideline (OWASP Authorization Testing Automation Cheat Sheet, read 2026-11-18) describes the same structure at larger scale: "Formalize the authorization matrix" in a file programs can process and people can review, and run one test per point of view.
Confused deputy: the agent's own authority is not the task's#
A deputy is a component with more authority than its caller, acting on the caller's behalf. It is "confused" when a caller gets it to use its own authority for something the caller could not do.
Guideline (MITRE CWE-441, read 2026-11-18) gives "Confused Deputy" as the alternate term for an intermediary that forwards a request without keeping its original source.
The underwriting agent is a deputy. As a workload of unit commercial-west it could reach every west submission; acting for u-uw-w1 on run run-7f3a, its delegated token names one submission, SUB-1004, a list of actions and an expiry. The test asks for something the workload could reach but the task does not cover: a recommendation for SUB-1007. Case UW-A11 expects 403 ("outside the delegated task"), and refusal again after the task expires. The token service applies the same idea to audiences: an exchange targeting the LLM gateway gets invalid_target (UW-A22).
What this is not. The injected passage that asked the agent for such writes (case UW-005 shape) is prompt injection; defenses against it are on Guardrails Engineering. This page tests that the request, once sent, is refused. Delegation design: Agents and MCP.
Privilege escalation: asking for more, and policies that check less#
Privilege escalation is a principal ending up with a power it was never granted. The insurance flow has two shapes of it.
Vertical: the agent asks for uw.decide. Three controls stack: the token service refuses the exchange with invalid_scope (agent run step 2); the tool server refuses record_underwriter_decision at its scope check (step 6); and with uw.decide misgranted, R-AZ-09 still refuses. Each layer has its own negative test, so removing one fails a test.
Horizontal: the peer underwriter decides a colleague's submission. This is the opening's drift, and the case shows both versions:
UW-A09 PASS Peer underwriter decides (policy drift) current={"allow": false, "status": 403, "reas...
ok current: expected {"allow": false, "status": 403, "reason": "underwriter is not assig..., actual {"allow": false, "status": 403, "reason": "underwriter is not assig...
ok previous: expected {"allow": true, "status": 200, "policy_version": "authz-2026.10.3"}, actual {"allow": true, "status": 200, "policy_version": "authz-2026.10.3"}
The previous line is the hole, recorded on purpose. A negative row for u-uw-w2 on SUB-1004, run against every deployable policy version, is the test the Thursday release lacked (versions: Policy as code).
Race conditions: check, then the facts change, then act#
A TOCTOU race (time of check to time of use) happens when access is checked, the facts change, and the action runs on the old answer.
Guideline (MITRE CWE-367, read 2026-11-18): "The product checks the state of a resource before using that resource, but the resource's state can change" in between.
In the cockpit, u-uw-w1 opens SUB-1004 and the backend asks the PDP: allowed. Five seconds later a supervisor reassigns the submission to the peer. If the backend then writes on the answer it got when the page loaded, a decision is recorded by someone no longer assigned. The snippet was written for this page; a successful run demonstrates the stale answer and the re-check, not a real concurrent race.
# Written for the threat-testing page (not part of the example): check-then-act on a reassigned submission.
import pdp
from clock import START_EPOCH_S as NOW
from directory import ACR_STRONG, resource_for, subject_for
assignments = {"SUB-1004": "u-uw-w1"} # the live directory; a supervisor changes it below
res = resource_for("SUB-1004", "submission")
def cockpit_subject(now):
s = subject_for("u-uw-w1", now=now, acr=ACR_STRONG, auth_time=NOW, client_id="svc-cockpit-bff")
s["assignments"] = [sub for sub, uw in assignments.items() if uw == "u-uw-w1"] # read the facts now
return s
checked = pdp.decide(cockpit_subject(NOW), "decision.record", res, {"now": NOW})
print("09:00:00 check:", checked["status"], checked["reason"])
assignments["SUB-1004"] = "u-uw-w2" # 09:00:05 the submission is reassigned to the peer
print("09:00:06 check-then-act writes:", "yes (stale answer)" if checked["allow"] else "no")
at_write = pdp.decide(cockpit_subject(NOW + 6), "decision.record", res, {"now": NOW + 6})
print("09:00:06 re-check at write:", at_write["status"], at_write["reason"])
09:00:00 check: 200 assigned underwriter decides through the cockpit after step-up
09:00:06 check-then-act writes: yes (stale answer)
09:00:06 re-check at write: 403 underwriter is not assigned to SUB-1004
Diagram: Check then act · a reassignment between the check and the write (TOCTOU)
Time runs down the page. The cockpit backend asks the policy decision point at 09:00:00 and gets allow. Five seconds later a supervisor reassigns SUB-1004 to the peer. If the backend writes on the earlier answer, a decision is recorded by someone no longer assigned. The fix is to decide at the moment of the write: re-check, or make the write itself conditional on the assignment in one transaction.
- Cockpit backend (PEP) to Policy decision point: 09:00:00 decide. u-uw-w1, decision.record, SUB-1004.
- Policy decision point to Cockpit backend (PEP): allow 200. True at 09:00:00.
- Supervisor reassigns to Submission store: 09:00:05 reassign. The fact behind the allow changes.
- Cockpit backend (PEP) to Submission store: write on the old allow. The race: the write trusts an answer about a world that no longer exists.
- Cockpit backend (PEP) to Policy decision point: re-check at write. The safe shape: decide again inside the write path, or make the write conditional on the assignment.
Re-checking at write time narrows the gap; closing it means deciding and writing in one step. In a database that is a conditional write: the update names the fact the decision depended on, and zero rows updated means "refuse".
-- shape only, not run here: the write itself carries the assignment condition
UPDATE submissions
SET decision = :decision, decided_by = :caller, authz_policy_version = :version
WHERE submission_id = :submission_id
AND assigned_underwriter = :caller
AND decision IS NULL;
-- 0 rows updated: the assignment changed or someone already decided; answer 409 or 403, write nothing
A SELECT ... FOR UPDATE on the row, in the same transaction as the PDP call, is the other common shape. And never cache a decision for a protected write (Enforcement).
Access after revocation and failing open#
Access after revocation: a leaver's session and directory access must go at once, while a self-contained token stays valid until exp unless the API checks a deny list or introspects. The lifecycle tests prove both halves, and UW-A16 shows the window with and without a deny list. How fast access must go is on Operations.
Failing open: a PEP that allows when the policy decision point is down has turned an outage into a breach. Fail open is usually an accident of code shape (a try/except that logs and continues), which is why it needs a test. The example's PEP answers 503 and writes nothing (UW-A23):
UW-A23 PASS IdP or PDP outage: writes fail closed pdp_down_write=503
ok pdp_down_write: expected 503, actual 503
ok pdp_down_read: expected 503, actual 503
ok idp_down_write: expected 401, actual 401
ok idp_down_read_with_valid_token: expected 200, actual 200
ok decisions_written: expected 0, actual 0
The full design, including why 503 and not 403, is on Enforcement. The testing point is narrower: for every dependency a decision needs (the PDP, the key set, the directory, the audit log), write one test that makes it fail and asserts the refusal.
The definition-of-done evidence#
The site's specification (section 10) asks for three demonstrations: item 4, a resource-specific decision at every sensitive hop, not a valid token or an x-app-id header; item 5, negative tests proving that unauthorized users and agents cannot read another tenant's quotes, execute restricted tools or perform protected underwriting writes; item 6, accurate lifecycle consequences. Each maps to a test class:
| Definition-of-done item | Test class in test_negative.py | Tests |
|---|---|---|
| 4: a resource-specific decision at every sensitive hop | Dod4ResourceDecisionAtEverySensitiveHop | 4 |
| 5: no cross-tenant read (broker, underwriter, agent, retrieval, precedent search, MCP) | Dod5NoCrossTenantRead | 7 |
| 5: no restricted tool execution | Dod5NoRestrictedToolExecution | 5 |
| 5: no protected underwriting write by an agent | Dod5NoProtectedWriteByAgent | 5 |
| 6: provisioning, review, JIT elevation, revocation, audit | Dod6LifecycleConsequences | 7 |
Run in a scratch copy of examples/uw-auth on 2026-11-18 with /opt/homebrew/bin/python3 -m unittest test_negative -v, output as printed:
test_every_audit_record_of_a_decision_names_rule_resource_and_policy_version (test_negative.Dod4ResourceDecisionAtEverySensitiveHop.test_every_audit_record_of_a_decision_names_rule_resource_and_policy_version) ... ok
test_repository_hop_applies_the_tenant_predicate_even_without_a_pep_check (test_negative.Dod4ResourceDecisionAtEverySensitiveHop.test_repository_hop_applies_the_tenant_predicate_even_without_a_pep_check) ... ok
test_service_hop_decides_on_the_record_not_the_token (test_negative.Dod4ResourceDecisionAtEverySensitiveHop.test_service_hop_decides_on_the_record_not_the_token) ... ok
test_x_app_id_header_is_not_identity (test_negative.Dod4ResourceDecisionAtEverySensitiveHop.test_x_app_id_header_is_not_identity) ... ok
test_agent_cannot_read_another_units_submission (test_negative.Dod5NoCrossTenantRead.test_agent_cannot_read_another_units_submission) ... ok
test_broker_can_never_search_precedents (test_negative.Dod5NoCrossTenantRead.test_broker_can_never_search_precedents) ... ok
test_broker_cannot_read_another_tenants_quote_by_quote_id_or_submission_id (test_negative.Dod5NoCrossTenantRead.test_broker_cannot_read_another_tenants_quote_by_quote_id_or_submission_id) ... ok
test_mcp_tool_cannot_reach_another_units_submission (test_negative.Dod5NoCrossTenantRead.test_mcp_tool_cannot_reach_another_units_submission) ... ok
test_precedent_search_never_returns_another_units_precedent (test_negative.Dod5NoCrossTenantRead.test_precedent_search_never_returns_another_units_precedent) ... ok
test_retrieval_never_returns_another_tenants_chunk (test_negative.Dod5NoCrossTenantRead.test_retrieval_never_returns_another_tenants_chunk) ... ok
test_underwriter_cannot_read_another_units_submission (test_negative.Dod5NoCrossTenantRead.test_underwriter_cannot_read_another_units_submission) ... ok
test_agent_cannot_obtain_uw_decide_by_token_exchange (test_negative.Dod5NoProtectedWriteByAgent.test_agent_cannot_obtain_uw_decide_by_token_exchange) ... ok
test_agent_decision_refused_even_with_misgranted_uw_decide (test_negative.Dod5NoProtectedWriteByAgent.test_agent_decision_refused_even_with_misgranted_uw_decide) ... ok
test_agent_recommendation_outside_task_or_after_expiry_refused (test_negative.Dod5NoProtectedWriteByAgent.test_agent_recommendation_outside_task_or_after_expiry_refused) ... ok
test_injected_instruction_does_not_produce_a_write (test_negative.Dod5NoProtectedWriteByAgent.test_injected_instruction_does_not_produce_a_write) ... ok
test_pdp_outage_fails_closed_for_writes (test_negative.Dod5NoProtectedWriteByAgent.test_pdp_outage_fails_closed_for_writes) ... ok
test_agent_cannot_execute_the_decision_tool (test_negative.Dod5NoRestrictedToolExecution.test_agent_cannot_execute_the_decision_tool) ... ok
test_auditor_cannot_write (test_negative.Dod5NoRestrictedToolExecution.test_auditor_cannot_write) ... ok
test_discovery_hidden_tool_called_by_name_is_refused (test_negative.Dod5NoRestrictedToolExecution.test_discovery_hidden_tool_called_by_name_is_refused) ... ok
test_listed_tool_outside_delegation_is_refused (test_negative.Dod5NoRestrictedToolExecution.test_listed_tool_outside_delegation_is_refused) ... ok
test_revoked_grant_is_refused_although_still_listed (test_negative.Dod5NoRestrictedToolExecution.test_revoked_grant_is_refused_although_still_listed) ... ok
test_access_review_revoke_removes_the_group (test_negative.Dod6LifecycleConsequences.test_access_review_revoke_removes_the_group) ... ok
test_audit_log_never_contains_a_token_or_key (test_negative.Dod6LifecycleConsequences.test_audit_log_never_contains_a_token_or_key) ... ok
test_break_glass_leaves_alert_review_and_audit_trail (test_negative.Dod6LifecycleConsequences.test_break_glass_leaves_alert_review_and_audit_trail) ... ok
test_jit_elevation_expires (test_negative.Dod6LifecycleConsequences.test_jit_elevation_expires) ... ok
test_leaver_loses_access_and_session_at_once (test_negative.Dod6LifecycleConsequences.test_leaver_loses_access_and_session_at_once) ... ok
test_provisioning_needs_an_approver_other_than_the_requester (test_negative.Dod6LifecycleConsequences.test_provisioning_needs_an_approver_other_than_the_requester) ... ok
test_self_contained_token_outlives_deactivation_until_exp_only (test_negative.Dod6LifecycleConsequences.test_self_contained_token_outlives_deactivation_until_exp_only) ... ok
----------------------------------------------------------------------
Ran 28 tests in 0.008s
OK
Each test name is a claim that would be false without its control. The timing is what unittest printed, not a benchmark.
This site's own cases UW-A01 to UW-A24 run the same controls as scenarios, including holes on purpose. Run with /opt/homebrew/bin/python3 run_cases.py in the same scratch copy:
case result title first check
UW-A01 PASS Own-tenant quote read status=200
UW-A02 PASS Cross-tenant quote read is 404 token_valid=true
UW-A03 PASS Gateway-only check lets BOLA through gateway_only={"status": 200, "data_returned": true...
UW-A04 PASS Forged x-app-id header trusting_service={"status": 200, "data_returned": true...
UW-A05 PASS Token for another audience valid=false
UW-A06 PASS Expired token valid=false
UW-A07 PASS Tampered payload valid=false
UW-A08 PASS alg none valid=false
UW-A09 PASS Peer underwriter decides (policy drift) current={"allow": false, "status": 403, "reas...
UW-A10 PASS Agent calls record_underwriter_decision as_delegated={"allowed": false, "check": "scope"}
UW-A11 PASS Agent recommendation outside its task allowed=false
UW-A12 PASS Decision without step-up status=401
UW-A13 PASS RAG never returns another tenant's chunk filter_first={"leaked": [], "A3_returned": false, ...
UW-A14 PASS Precedent from another unit P2_returned=false
UW-A15 PASS Kafka: consumer write and DLQ replay refused consumer_write=false
UW-A16 PASS Revocation window self_contained_without_deny_list=true
UW-A17 PASS Leaver directory_allows=false
UW-A18 PASS JIT elevation expires before=false
UW-A19 PASS Break-glass is audited alerts=1
UW-A20 PASS mTLS: expired, not allowlisted, rotated expired={"handshake_ok": false, "failed": ["v...
UW-A21 PASS MCP: discovery is not authorization hidden_tool={"listed": false, "allowed": false, "...
UW-A22 PASS Token exchange narrows issued_scope=uw.read uw.recommend
UW-A23 PASS IdP or PDP outage: writes fail closed pdp_down_write=503
UW-A24 PASS The audit log contains no token or key leaks=[]
24/24 cases match their expected block. (Scripted stand-in agent: this shows the access controls, not model quality.)
"PASS" means the outcome matched the expected block, which for UW-A03, UW-A04 and the previous-policy line of UW-A09 is a recorded leak. Recording the hole next to the fix keeps the reason for each control visible.
The refusals, as an investigator sees them#
In production the same refusals show up as security events. The incident lab opens on the negative preset: only the authn.failure and authz.deny records, 13 of the 27 the example's audit log wrote.
| Time (UTC) | Kind | Principal | Request | Reason |
|---|---|---|---|---|
| 09:00:00 to 09:00:10 | authn.failure (6) | unknown | req-0001 to req-0006 | gateway: signature |
| 09:01:32 | authz.deny | u-broker-s1 | req-0008 | record belongs to another tenant: not found (existence not revealed) |
| 09:01:37 | authz.deny | u-broker-s1 | req-0009 | record belongs to another tenant: not found (existence not revealed) |
| 09:03:47 | authn.failure | u-broker-s1 | req-0011 | gateway: revocation |
| 09:05:32 | authz.deny | wl-underwriting-agent | req-0015 | insufficient_scope: uw.decide required |
| 09:05:47 | authz.deny | wl-underwriting-agent | req-0016 | record belongs to another business unit: not found (existence not revealed) |
| 09:11:17 | authz.deny | u-uw-w1 | req-0018 | step-up required: strong authentication within 300 s |
| 09:22:22 | authn.failure | unknown | req-0020 | gateway: kid |
The two agent rows are the injected passage's two requests. The records themselves are explained on Audit.
In real frameworks (these ran)#
The same negative tests ran in maintained libraries (examples/frameworks/). Product (FastAPI 0.165.0 with PyJWT 2.15.1, run on 2026-11-18): ./run.sh <scratch-dir>, 19 passed. The negative lines:
test_app.py::test_no_token_401 PASSED [ 5%]
test_app.py::test_expired_401 PASSED [ 10%]
test_app.py::test_wrong_audience_401 PASSED [ 15%]
test_app.py::test_wrong_issuer_401 PASSED [ 21%]
test_app.py::test_alg_mismatch_rs256_401 PASSED [ 26%]
test_app.py::test_alg_none_401 PASSED [ 31%]
test_app.py::test_signed_by_other_key_401 PASSED [ 36%]
test_app.py::test_broker_reads_other_tenant_quote_404 PASSED [ 47%]
test_app.py::test_other_tenant_and_missing_look_the_same PASSED [ 52%]
test_app.py::test_tenant_claim_in_query_is_ignored PASSED [ 63%]
test_app.py::test_missing_scope_403 PASSED [ 68%]
test_app.py::test_peer_underwriter_403 PASSED [ 84%]
test_app.py::test_other_unit_underwriter_404 PASSED [ 89%]
test_app.py::test_agent_with_uw_decide_scope_403 PASSED [ 94%]
test_app.py::test_broker_cannot_decide_403 PASSED [100%]
PyJWT proves the token is genuine and meant for this API; the tenant, assignment and agent checks are application code, and the tests prove that code.
Product (Spring Boot 4.1.2, Spring Security 7.1.2, run on 2026-11-18): mvn -q test, 17 tests, 0 failures. Among them:
"wrongAudience_401"
"algMismatchHs256_401"
"wrongIssuer_401"
"expired_401"
"signedByOtherKey_401"
"brokerReadsOtherTenantQuote_404"
"peerUnderwriter_403"
"agentWithUwDecideScope_403"
"jwtPostProcessor_skipsTheDecoder_soWrongAudienceStillPasses"
The last line is a warning written as a test. Product (Spring Security, as documented on 2026-11-18): the resource server does not validate aud by default, so the demo adds a validator; and the jwt() MockMvc helper builds a mock token "through any authentication APIs" without, as the run shows, the JwtDecoder: a jwt() request with the wrong audience got 200. Use jwt() for authorization tests; audience tests need real signed tokens, as the first five lines do.
What goes wrong without negative tests#
The opening is the common failure: positive tests only. Badly built negative suites fail more quietly.
| Failure | What it looks like | The fix |
|---|---|---|
| Negative rows only | a policy that denies everything passes the whole suite | positive control rows next to every negative matrix |
| Broken credentials in authorization tests | "cross-tenant" test sends an expired token; it proves authentication | genuine credentials, one wrong fact |
| Status not asserted | a cross-tenant read returning 403 passes, and confirms the record exists | assert the exact status and the absent body |
| One policy version tested | rollback reintroduces the peer-underwriter hole | run negative rows against every deployable version |
Choosing how much testing machinery#
| Situation | Simplest credible approach | Grow when | Alternatives |
|---|---|---|---|
| One service, a handful of roles | negative unit tests against the decision function, plus HTTP tests for 401/403/404 | more than one team writes policy | policy-engine tests (opa test ran in examples/frameworks) |
| Multi-tenant SaaS | generated cross-tenant matrix, both identifiers, positive controls | tenants with custom roles | matrix in a reviewed file |
| Agents with tools | negative tests per layer: token exchange, tool server, PDP | many tools and delegations | per-tool argument tests |
| High-impact writes | the above plus race and outage tests, and a known-deny probe in production | always | scheduled penetration tests |
This site's recommendation: start with the threat model table and give each row one negative test; generate matrices once there are more than a few principals; keep the cases that demonstrate holes (like UW-A03).
In a coding agent. A coding agent editing authorization code works through a repo-scoped GitHub App and proposes a pull request. The negative suite runs in CI on it; if the change letsu-uw-w2decideSUB-1004, a test fails and the merge is blocked. A person with merge rights reviews the policy diff, and the agent's sandbox holds no production credential. See Coding agents.
What this is not. Measuring how often the agent proposes an out-of-scope action is evaluation, on Eval Engineering. This page proves that whatever it proposes, the controls refuse what is not allowed.
Security notes#
- Keep demonstrations safe: explain an attack as far as the control needs; never point attack strings at systems you do not own.
- Negative tests use genuine credentials with one wrong fact; otherwise they test authentication, not authorization.
- Assert the status, the absence of a body and that nothing was written. A 403 where 404 is expected is a disclosure.
- Test helpers that bypass token validation are for authorization tests only.
- A refusal is a security event: negative paths must write audit records without tokens or keys.
Monday checklist#
- Write the threat model table for your service: boundary, STRIDE letter, threat, control, test. Count the empty last cells.
- Add a cross-tenant matrix with both identifiers per record and positive control rows.
- Add one BFLA row per protected function for every role that must not call it, including read-only and agent principals.
- Add a substitution test: a token for another audience, and an ID token, at each API.
- Find every check-then-act on a protected write; make the write conditional or re-check inside it.
- For each dependency of a decision, add a test that makes it fail and asserts refusal.
Exercises#
Exercise 1. A teammate's cross-tenant test uses a token whose exp is in the past, "to be safe". It passes. What has it proved?
AnswerOnly that expired tokens are refused
The request was refused at the expiry check, before any authorization ran. It proves nothing about tenant isolation. Use a valid token of the other tenant, so the only wrong fact is the record.
Exercise 2. Your matrix test has 40 deny rows and no allow rows. A refactor makes pdp.decide return deny for every request. What does the suite report?
AnswerAll green
Every deny row still passes. Positive control rows (u-broker-n1 reads Q-5531) are what fail, and asserting the expected rule or status on each deny row catches refusals for the wrong reason.
Exercise 3. Which STRIDE letter and which control fit "the cockpit records a decision on an allow it got two minutes earlier, after the submission was reassigned"?
AnswerE (or T), closed by deciding at write time
It is elevation of privilege through a race: a person acts with a power they no longer hold (some teams file it under tampering, an unauthorized change of data). The control is a conditional write on the assignment in one transaction, or a re-check immediately before the write, and no decision cache for protected writes.
Advanced perspective#
Generate the matrix from the directory and the policyHand-written rows miss the combinations nobody thought of
A hand-written matrix encodes the author's mental model, which is where the gaps come from. The stronger form generates rows: every principal in the directory × every action × every record, with the expected outcome computed from a small, separately written statement of intent (for example "brokers only read and submit in their own tenant; only an assigned person who is an underwriter decides"). The test then compares the PDP's answer with the intent for every cell, which for this example is 9 principals × 10 actions × 3 records, a few hundred cells. Two cautions. The intent must not be a copy of the policy code, or the test agrees with every bug. And generated suites need a readable failure report: print the cell (principal, action, record, expected, actual, rule) so a reviewer can tell a policy bug from an intent bug. The OWASP Authorization Testing Automation sheet's "one test per point of view" grouping is a good report shape.
Does the suite notice when a check disappears?Mutation testing for policy
A negative suite is only as good as the controls it would notice losing. The direct test: remove one condition from the policy (the assignment check, the tenant comparison, the client check for svc-cockpit-bff, the step-up), run the suite, and confirm at least one test fails for each removal. The example has one such mutation built in, the previous policy authz-2026.10.3, and UW-A09 shows it is detected. Doing this for every condition, by hand or with a tool, turns "we have negative tests" into "every condition in R-AZ-09 has a test that fails without it". Conditions with no failing test are either untested or unnecessary; both are worth knowing. This is the same idea measurement sites apply to evaluation suites, applied to access control.
Negative tests in productionA known-deny probe, and why it needs its own principal
CI proves the code; it does not prove the deployed configuration: a gateway route added by hand, a policy bundle that failed to load, a PDP that answers but with a stale version. A synthetic probe in production sends a few known-deny requests on a schedule (the peer underwriter on SUB-1004, a cross-tenant read) and alerts if any is ever allowed, alongside a known-allow to detect deny-everything. The probe needs its own principals, clearly named as synthetic, with no access to real tenant data even when the control fails: its cross-tenant target should be a synthetic record in a real tenant's boundary, created for the purpose. Its refusals are security events and should be tagged so they do not drown real alerts, and its credentials are as sensitive as any service account's.
Where STRIDE is weak for agent systemsIt finds threats per boundary; delegation crosses several
STRIDE asks its six questions at one boundary at a time. The agent threats that matter most cross several: a passage injected through retrieval (one boundary) makes the model propose a write (no boundary at all, it is text), which the agent application sends with a delegated token (a second boundary) to a tool server that calls a domain service (a third). Per boundary each step looks fine. Two habits help. Draw the delegation chain as its own path and ask, at the end, "whose authority is this write using, and did that person grant this action on this record?" And treat the model's output as an untrusted input crossing a boundary into the agent application, so every tool call it suggests meets a deterministic check. The model suggests an action; deterministic code decides whether it may run.
In one breath#
A threat model lists the trust boundaries, asks STRIDE's six questions at each, and gives every threat a control and a test that would fail without it. Negative tests keep every input genuine except one (the tenant, the function, the audience, the moment), assert the exact refusal, and sit next to positive control rows. In the example, 28 negative tests and 24 cases are the definition-of-done evidence. Races and outages need their own tests, because check-then-act and fail-open are bugs of timing and code shape, not of policy.
The tests prove the controls hold today. Keys still rotate, caches still need purging, the identity provider will go down one morning, and a leaver's access must vanish in a stated time. Operations runs those lifecycles without opening the holes this page tests for.