All chapters
  1. Start hereA valid token is not a permission
  2. Foundations · 1Identity and trust primitives: who is calling, and who checks
  3. Foundations · 2People and identity proofing: accounts, passwords and recovery
  4. Foundations · 3Sessions and browser security: cookies, CSRF, XSS and the BFF
  5. Foundations · 4Authenticators and MFA: passwords, passkeys and phishing resistance
  6. Foundations · 5API keys and application credentials: what a key proves
  7. Foundations · 6Tokens: bearer, JWT anatomy, validation and revocation
  8. Foundations · 7OAuth, OpenID Connect and federation: delegation and login
  9. Authorization · 8Authorization from first principles: subject, action, resource, context
  10. Authorization · 9Access control models: RBAC, ABAC, ReBAC and ownership
  11. Authorization · 10The enforcement architecture: PEP, PDP, PIP and PAP
  12. Authorization · 11Policy as code: versions, tests, engines and decision logs
  13. Authorization · 12Tenant and data isolation: one platform, many brokerages
  14. Authorization · 13Identity governance and lifecycle: joiner, mover, leaver
  15. Authorization · 14Privileged access: standing rights, just-in-time elevation and break-glass
  16. Workloads · 15TLS, PKI and certificates: proving a host, and a client
  17. Workloads · 16Workload and service identity: SPIFFE, mTLS and caller allowlists
  18. Workloads · 17API gateways and zero trust: the edge checks, the service decides
  19. Workloads · 18Event-driven identity: Kafka principals, ACLs and replay
  20. Workloads · 19Cloud IAM and secretless deployment: roles, federation and no stored keys
  21. Workloads · 20External SaaS, webhooks and trust: calls that cross company lines
  22. Workloads · 21Private controlled environments: what private networking does and does not secure
  23. AI-native · 22Authenticating to model providers: keys, gateways and cloud IAM
  24. AI-native · 23MCP identity and authorization: discovery is not permission
  25. AI-native · 24RAG and retrieval authorization: only what this caller may read
  26. AI-native · 25Agents as principals and delegates: whose authority is this?
  27. AI-native · 26Agent-to-agent trust: identity, capabilities and forwarded authority
  28. AI-native · 27Coding agents and CI: repository scopes, OIDC and no production keys
  29. Operations · 28Auditing and observability of access: decisions you can prove later
  30. Operations · 29Threat modeling and testing: the negative tests that prove access control
  31. Operations · 30Operational lifecycles: rotation, revocation and outages
  32. Reference · 31Standards, frameworks and products: what each one is, and what it is not
  33. Reference · 32Architecture decision playbooks: choosing an auth strategy
  34. ReferenceGlossary
  35. AboutAbout this site and its sources
Operations · 28

Auditing and observability of access: decisions you can prove later

Sign-ins, token issuance, allow and deny decisions, elevations and tool calls as security events. Correlation by request, run, trace, principal, tenant and policy version; integrity; SIEM and alerts; never a token in a log.

You will separate
Diagnostic traces, security events and durable audit records: three jobs, three readers, three lifetimes.
You will correlate
A request, a trace and an agent run with request_id, trace_id and run_id, and know which one joins what.
You will protect
The trail itself: no token or secret in any record, tampering made visible, and only the auditor reading it.

Three months after run-7f3a, the auditor u-auditor-1 receives a question from the business: "The roof on PROP-2207 turned out to be fine and the broker complains about the inspection delay. Who decided to send an inspector on SUB-1004? Was it the AI? And did anything ever try to approve that submission without a person?"

The agent's prompts were never kept, and the tracing system's detailed spans expired long ago. What is left is whatever the system chose, at the time, to write down as a record of who did what.

If that record was designed well, the auditor answers in minutes: the decision was recorded at 09:12:02 by u-uw-w1, a person, through the cockpit, after a phishing-resistant step-up, under authorization policy authz-2026.11.1; the agent recommended refer; and at 09:05:32 the agent did try to call record_underwriter_decision with approve, an instruction it found in a broker's attachment, and was refused. If the record was designed badly, the answer is a guess, or the logs hold the underwriter's bearer token in plain text and the audit becomes an incident of its own.

This page is about the design that makes the first outcome normal.

Three records that look alike and are not#

Every request in the example produces output that could be called "a log". Three different things are hiding in that word, and the first job is to keep them apart.

A diagnostic trace says how a request ran: which services it passed through, how long each took, where it failed. Engineers read it to fix performance and bugs. Tiny example: the trace of req-0015 shows the tool server's span and its duration.

A security event says that something security-relevant happened, so that a person or a rule can react now. Tiny example: "six requests with bad signatures in ten seconds" is a set of security events; security operations should see them while they happen.

A durable audit record says who did what to which record, under which policy, with what result, and it is kept so that the question can be answered later. Tiny example: "the decision on SUB-1004 was recorded by u-uw-w1 under authz-2026.11.1". The series' word for one such entry is an audit record (Identity and trust primitives defines it); the ordered, protected collection of them is the audit trail.

Diagnostic traceSecurity eventDurable audit record
Question it answershow did this request run?is something bad happening now?who did what, to which record, under which policy?
Main readerengineers on callsecurity operations, alert rulesauditors, investigators, the business
Typical contentspans, timings, errors, service namesfailed sign-ins, denials, revocations, key changesprincipal, on behalf of, action, resource, decision, rule, policy version
Lifetimeshort, set by operations need; sampling is normallong enough to correlate an attackset by policy or regulation; never sampled
Integritybest effortprotected in transit to the analysis systemtamper evident, append only, stored outside the writer's control
In the examplenot modelled; a trace_id and span_id on every record point to itauthn.failure, authz.deny, token.revoked, key.rotatedauthz.allow and authz.deny with rule and authz_policy_version; the human_decision

The row that matters most is lifetime. A trace system that samples is doing its job; an audit trail that samples is broken. That is why the three must not share one pipeline with one retention setting.

Before AI. Backend teams already run all three, usually without naming the difference. The web server's access log, an APM tool's traces, and an audit table that a Spring service writes on every state change are three systems for a reason. Agents add new kinds of events (tool calls, delegations, refusals of model proposals) but not a new category.

The example's audit.py writes the second and third kinds into one append-only log, and leaves traces to the tracing system. Logs, metrics and traces as an observability discipline are taught on observe.sammuthu.com.

What counts as a security event#

A security event is any occurrence that changes or tests who can do what. The example's incident timeline (incident.py) runs its own services on a simulated clock and writes 27 records of seven kinds:

KindCount in the timelineWritten byExample record
authn.failure8the gatewaysix forged signatures (req-0001 to req-0006), a revoked token (req-0011), a retired key (req-0020)
authn.success4the gatewaya broker's token validated before the quote service decides
authz.allow7quote service, tool server, cockpitthe agent's roof read; the underwriter's decision
authz.deny5quote service, tool server, cockpittwo cross-tenant reads (404), two refused tool calls, one decision without step-up (401)
token.revoked1security operationsthe token of u-broker-s1 after cross-tenant probing
key.rotated1security operationsdemo-2026a to demo-2026b, overlap begins
key.retired1security operationsdemo-2026a removed from the verifier key set

Two kinds the spec of this site asks for are not in the timeline, and they belong in a real one. Token issuance: the example's token service does not write an event when it issues a token, so a forged token that was accepted could not be told apart from a legitimate one by its jti (Operations explains why that matters after a key compromise). Privileged elevation: the example's PAM module keeps its own trail (requested, approved, action, expired; see Privileged access), separate from this log. This site's recommendation: write issuance, elevation, approval and every tool call into the same security event stream, so one query sees them all.

Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "Where possible, always log" includes "Authentication successes and failures" and "Authorization (access control) failures". Guideline (OWASP Authorization Cheat Sheet): one of its headings is "Implement Appropriate Logging". Standard (A2A 1.0.0, §13): agents "SHOULD log security-relevant events (authentication failures, authorization denials, suspicious requests)". Three documents, one floor: failures and denials are always written.

The smallest primitive: one record#

Here is one record exactly as the example wrote it, the second cross-tenant read: the broker u-broker-s1 of tenant brk-south asking for Q-5531, which belongs to brk-north.

{
  "seq": 10,
  "time": "2026-11-18T09:01:32Z",
  "kind": "authz.deny",
  "principal_id": "u-broker-s1",
  "decision": "deny",
  "reason": "record belongs to another tenant: not found (existence not revealed)",
  "request_id": "req-0008",
  "run_id": null,
  "trace_id": "390fda6d572fbd48f316d4b43cdb9e15",
  "span_id": "f17a325ccec201a3",
  "action": "quote.read",
  "resource": "Q-5531",
  "tenant": "brk-south",
  "business_unit": null,
  "rule": "R-AZ-03",
  "authz_policy_version": "authz-2026.11.1",
  "status": 404,
  "prev_hash": "803ee0bb7a2a73152234f050e641d4bcc4ea1f1f4e6af0b17efc79d9b64279e2",
  "hash": "3094a55624ce63e654889b62b9d4196b1484b477c5157107ff4f8f8d419f008d"
}

Read it as the answer to four questions. Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "The application logs must record "when, where, who and what" for each event." When: time. Who: principal_id and the caller's tenant. What: action on resource, with decision, status and reason. Where: the request_id and trace_id tie it to the edge request and to its trace. Under which rules: rule and authz_policy_version.

Note what is absent. There is no token, no Authorization header, no cookie, and no copy of the quote. The broker saw a 404 and the record says so; it does not record the quote's contents to prove the point. The last two fields, prev_hash and hash, make the record part of a chain; they come up below.

Correlation: which id joins what#

A single question ("did anything try to approve SUB-1004?") touches records written by different services at different times. A correlation id is an identifier carried across those records so they can be joined. The example carries several, and each joins something different:

FieldWhat one value identifiesWhat it lets you join
request_idone inbound HTTP request at the edgeevery record about that request, at every hop
trace_idone trace; on this site one trace per inbound request through every hopthe audit record and the diagnostic trace of the same request
span_idone hop of that requestthe record and the exact span in the trace
run_idone execution of the agent workflow for a submissionevery request the agent made in that run, across many traces
principal_idwho acted (with on_behalf_of when delegated)everything one principal did, across runs
tenant or business_unitthe caller's tenant or unitprobing across tenants; activity per unit
authz_policy_versionthe authorization policy that decidedwhich decisions a policy change affects
jtione tokenwhere a stolen or revoked token was used (logged as an id, never with the token)

A request id is created at the edge, once per inbound request, and passed to every service that handles it. A trace id is the tracing system's identifier for the same journey; on this site each inbound request starts one trace, so request_id and trace_id correspond one to one. A run id is wider: run run-7f3a made six tool requests in the timeline, each its own inbound request with its own trace, so the run spans six traces. Agents are the reason run_id exists: without it, an agent's twelve requests look like twelve unrelated calls.

Production pitfall. Correlation ids must be created by your code at a trust boundary, not accepted from the caller. A client that sends its own request_id can make its requests look like someone else's in the trail. Accept an inbound id only as a separate, labelled field.

The spec of this site names tenant_id and policy_version; the example writes tenant (or business_unit) and authz_policy_version, the second on purpose: the business policy UW-GUIDE 2026.2 also has a version, and a record that says "policy version" without saying which policy invites the wrong answer (Policy as code explains the drift case UW-A09).

Guideline (OWASP Microservices Security Cheat Sheet, read 2026-11-18): structured logs with correlation IDs, and "Exclude secrets and unnecessary sensitive data before the microservice emits a log entry".

How the example writes a record#

AuditLog.append in audit.py builds the record with the correlation ids, redacts it, then chains it to the previous record. This is the part that matters, quoted from the file:

rec = redact(copy.deepcopy(rec))
prev = self.records[-1]["hash"] if self.records else GENESIS
rec["prev_hash"] = prev
rec["hash"] = _hash(prev, {k: v for k, v in rec.items() if k not in ("hash", "prev_hash")})
self.records.append(rec)

The order is deliberate. Redaction runs before hashing, so the chain protects the redacted form and nothing secret is ever hashed into a value someone might later try to reverse. _hash is SHA-256 over the previous hash plus the record serialized as canonical JSON (sorted keys, no spaces), so the same record always gives the same hash.

The snippet replays the incident timeline, checks the chain, scans everything written for token-shaped strings and key material, then plays an insider who edits one record. It demonstrates tamper evidence and redaction in the example's own log; it does not demonstrate a production audit store, which must also keep the records somewhere the writer cannot change.

import copy
import json

import audit
import incident
from audit import AuditLog, leaks, redact

out = incident.run()
log = AuditLog()
log.records = out["records"]
print("records:", len(log.records), "| chain:", log.verify_chain())
print("token or key strings found:", leaks(log.dump()))

# An insider edits record 10 (a denied cross-tenant read) to say "allow".
edited = AuditLog()
edited.records = copy.deepcopy(log.records)
edited.records[9]["decision"] = "allow"
print("edit record 10 only:", edited.verify_chain())

# The insider also recomputes record 10's own hash, so record 10 looks consistent.
r10 = edited.records[9]
r10["hash"] = audit._hash(r10["prev_hash"], {k: v for k, v in r10.items() if k not in ("hash", "prev_hash")})
print("edit and rehash 10: ", edited.verify_chain())

# Redaction runs on every record before it is written.
print(json.dumps(redact({"authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4In0.c2ln",
                         "reason": "client sent eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4In0.c2ln", "jti": "jti-d5ecb5057011"})))

Run in a scratch copy of examples/uw-auth on 2026-11-18 (the agent in the example is a scripted stand-in, so these results show how the access controls work, not how good any model is):

records: 27 | chain: {'intact': True, 'first_bad_seq': None}
token or key strings found: []
edit record 10 only: {'intact': False, 'first_bad_seq': 10}
edit and rehash 10:  {'intact': False, 'first_bad_seq': 11}
{"authorization": "[REDACTED]", "reason": "client sent [REDACTED:token]", "jti": "jti-d5ecb5057011"}

The token string in the last line is a fabricated, unsigned fragment written for this page; the redactor removes it by shape.

Integrity: tamper evidence, not tamper proof#

Audit integrity is the property that a record, once written, cannot be changed or removed without the change being detectable. The example's chain gives a precise version of it. Each record's hash covers the previous record's hash, so changing record 10 breaks record 10's own hash check. An insider who also recomputes record 10's hash moves the break to record 11, whose prev_hash no longer matches. To hide the edit, the insider has to recompute every hash from record 10 to the end.

That last sentence is the limit. A chain stored next to the code that writes it detects careless tampering and accidental corruption. It does not stop someone who controls the whole log and recomputes it. The docstring of audit.py says it plainly: tamper evidence, not tamper proof. Three practices close the gap, and none is in the example:

Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "Build in tamper detection so you know if a record has been modified or deleted"; "Store or copy log data to read-only media as soon as possible"; "All access to the logs must be recorded and monitored (and may need prior approval)".

Redaction: what never enters a record#

Log redaction is removing or masking sensitive values before a record is written, so the trail never becomes a store of credentials or personal data. A token in a log is a token anyone with log access can use until it expires; a password in a log is a breach.

The example's redact works three ways. Values of sensitive keys (authorization, access_token, cookie, client_secret, code_verifier and others) become [REDACTED]. Anything shaped like a compact JWS or an opaque token handle, wherever it appears, becomes [REDACTED:token], which catches the token pasted into an error message. And demo key material becomes [REDACTED:key]. The token's jti stays: it is an identifier, not a credential, and it is what lets an investigator ask where one token was used. This site's own case UW-A24 scans every record of the run and the timeline and finds nothing (leaks=[]), with both hash chains intact.

Guideline (OWASP Logging Cheat Sheet): data that "should usually not be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed, or encrypted" includes "Session identification values", "Access tokens", "Authentication passwords" and "Encryption keys and other primary secrets".

Agents add a category of their own: unredacted user content. The broker's attachment A2 contained an injected instruction. The audit record of the refused call (record 21) names the action, the resource, the principal, on whose behalf, the reason and the policy version; it does not copy the attachment text, the prompt or the model's output. This site's recommendation: security events and audit records carry identifiers and decisions, never prompts, retrieved passages or document contents. If an investigation needs the content, keep it in a separate store with its own access control and retention, referenced by id from the record.

Security note. Redaction by key name alone fails the day a developer logs str(request). Redact by shape as well, scan written records for token-shaped strings in a test (as test_negative.py does), and treat a hit as a production incident.

Who may read the audit trail#

An audit trail every engineer can read is a second copy of the business. Reading is its own permission. In the example, audit.read is a scope only the auditor holds, and rule R-AZ-10 grants reading audit records to auditors only. The snippet asks the policy decision point for three principals, then tries the tool server's read_audit_log tool.

import mcp_matrix
import pdp
from clock import START_EPOCH_S as NOW
from directory import subject_for

audit_log = {"type": "audit_log"}
subs = mcp_matrix.subjects(NOW)
for label, s in (("auditor u-auditor-1", subject_for("u-auditor-1", now=NOW)),
                 ("underwriter u-uw-w1", subject_for("u-uw-w1", now=NOW)),
                 ("underwriting agent", subs["underwriting_agent"])):
    r = pdp.decide(s, "audit.read", audit_log, {"now": NOW})
    print(f"{label:<20} {r['status']} {r['rule']}: {r['reason']}")
for label, key in (("auditor", "auditor"), ("underwriting agent", "underwriting_agent")):
    r = mcp_matrix.execute(subs[key], "read_audit_log", {}, now=NOW)
    print(f"read_audit_log as {label:<18} allowed={r['allowed']} check={r['check']}")
auditor u-auditor-1  200 R-AZ-10: auditor reads audit records
underwriter u-uw-w1  403 R-AZ-02: token scope lacks audit.read (insufficient_scope)
underwriting agent   403 R-AZ-02: token scope lacks audit.read (insufficient_scope)
read_audit_log as auditor            allowed=True check=pdp
read_audit_log as underwriting agent allowed=False check=scope

The underwriter whose decision is in the trail cannot read it through the API, and the agent cannot read it at all. The auditor reads and writes nothing: u-auditor-1 holds no write scope, which is the separation of duties between the people who act and the person who checks (Governance). This site's recommendation: also log every read of the audit trail as a security event, and keep administrative access to the audit store (deleting, changing retention) under just-in-time elevation with a second approver (Privileged access).

The incident investigator#

The lab below holds the 27 records of the timeline. Filter by kind, decision, principal or id; reveal the answer to each of seven questions and see the browser recompute it from the records; edit record 10 and watch the chain break at 10, or at 11 if the insider also recomputes record 10's hash.

The recorded answers, with the fields that produce each one:

QuestionAnswerAnswered from
Which requests failed authentication with a bad signature?req-0001 to req-0006kind authn.failure, reason signature, request_id
Which requests were cross-tenant reads, and what did the caller see?req-0008, req-0009 (u-broker-s1); 404, existence not revealedkind authz.deny, status 404, tenant brk-south
How many traces does run run-7f3a span?6 (one trace per inbound request)distinct trace_id where run_id is run-7f3a
Was a revoked token used again, and was it refused?yes: req-0011, rejected by the gateway (revocation)authn.failure with reason revocation, after token.revoked
Which request used a token signed with the retired key?req-0020, rejected (kid demo-2026a retired at 09:22Z)authn.failure with reason kid, after key.retired
Which tool calls of run-7f3a were refused?record_underwriter_decision, record_recommendation (for SUB-2001)decision deny, run_id run-7f3a, action
Who recorded the underwriting decision?u-uw-w1, a person, through the cockpit, after step-upaction decision.record, decision allow, principal_id

Chain status: intact (27 of 27 hashes recompute). No token, key or secret appears in any record.

The last two rows answer the auditor's question from the opening. The agent's attempt is record 21: principal_id wl-underwriting-agent, on_behalf_of u-uw-w1, action record_underwriter_decision, rule null because the tool server's scope check refused it before the policy decision point, reason insufficient_scope: uw.decide required. The decision is record 25, by u-uw-w1, rule R-AZ-09, acr phishing-resistant, policy authz-2026.11.1. The response to incidents built on these answers is on Operations.

Diagram: One refused tool call, three destinations · same ids, different contents and readers

The request is req-0015 of run run-7f3a: the underwriting agent, acting for u-uw-w1, proposes record_underwriter_decision on SUB-1004 and is refused at the scope check. Rows are the three destinations; columns are what each keeps, who reads it and how long it lives. All three carry the same request_id and trace_id, so they can be joined, but only the audit record is kept and never sampled.

  1. Diagnostic trace
    1. spans, timings, errors: The tool server's span and how long it took, under trace_id acfd2b61...
    2. engineers on call: Used to fix latency and bugs.
    3. short, sampled: Sampling is normal here, which is why it cannot be the audit trail.
  2. Security event
    1. authz.deny on a decision action: An agent principal proposed a tool it may not hold.
    2. security operations, SIEM rules: A rule can flag decision tools proposed by agents.
    3. long enough to correlate: Protected in transit to the analysis system; never sampled.
  3. Durable audit record
    1. principal, on behalf of, action, rule, version: wl-underwriting-agent for u-uw-w1, record_underwriter_decision on SUB-1004, deny, rule null, authz-2026.11.1.
    2. the auditor only: audit.read is held by u-auditor-1; no token or content is in the record.
    3. policy retention, hash-chained: Append only, tamper evident, shipped to storage the writer cannot change.

The diagram follows one request, the agent's refused call, into the three destinations. The important observation is that each destination keeps a different subset for a different reader, joined by the same ids.

Security events to a SIEM, and the alerts worth having#

A SIEM (security information and event management system) collects security events from many systems, stores them for search, and runs detection rules that raise alerts. The audit log is one of its sources; it does not replace the durable trail, which may live elsewhere with longer retention.

The timeline contains five patterns a rule could catch, each from fields already in the records:

Pattern in the timelineRule (illustrative)Records
Six bad signatures in ten seconds for one claimed subjectburst of authn.failure with reason signature1 to 6
One principal getting 404 on another tenant's recordsauthz.deny, status 404, same principal, more than one resource10, 12
A revoked token presented againauthn.failure with reason revocation17
An agent run proposing a decision tool it may not holdauthz.deny on a decision action where the principal is an agent21
A token signed with a retired keyauthn.failure with reason kid, after key.retired27

The thresholds are design choices, not numbers from any standard. Signals can also arrive from outside. Standard (OpenID Shared Signals Framework 1.0 and CAEP 1.0, listed Final, read 2026-11-18): a transmitter streams events about subjects to receivers as Security Event Tokens (RFC 8417). Standard (RFC 9967, May 2026): a SCIM profile for Security Event Tokens, so identity changes such as a deactivation reach receivers without polling. Both bring events in; they do not replace your own records of what your services decided.

Diagram: The hash chain · what an edit breaks, and what only an outside copy catches

Each record's hash covers the previous hash and the record's canonical JSON. Editing record 10 breaks the check at 10. Recomputing record 10's hash moves the break to 11. Recomputing every hash to the end verifies again, which is why records are also shipped to storage the writer cannot change, or the latest hash is checkpointed somewhere independent.

  1. The log the service writes
    1. record 9: authn.success for u-broker-s1, req-0008.
    2. record 10: authz.deny, 404, cross-tenant read of Q-5531.
    3. record 11: Its prev_hash still holds the original h10.
  2. Outside the writer's control
    1. write-once copy: A copy the writing service cannot modify or delete.
    2. published checkpoint: The latest hash, recorded independently at intervals.

The second diagram shows the hash chain and the two edits from the snippet, and where an independent checkpoint would catch the third.

What goes wrong without these rules#

FailureWhat the auditor findsControl
The bearer token is logged with the requestthe audit store is now a token store; the audit is an incidentredaction by key and by shape; a leak scan in tests (UW-A24)
Denials go only to the trace system, which samplesthe agent's refused call is gonesecurity events are never sampled
No authz_policy_version on decisionscannot tell whether the peer's decision was allowed under the old policy (UW-A09)record the authorization policy version on every decision
No run_idthe agent's six requests look unrelateda run id on every request an agent makes
No on_behalf_ofthe agent's actions look like the underwriter's, or the reverserecord both principals on delegated requests
The writing service can rewrite the logan edited denial is indistinguishable from the originalhash chain plus storage outside the writer's control
Everyone can read the trailthe trail leaks the business it recordsaudit.read for auditors only; reads are logged
What this is not. This page does not teach tracing, metrics or log pipelines as an engineering discipline (observe.sammuthu.com), how to respond to an incident and repair access (Operations), or how to design negative tests (Threat modeling and testing). It is about which security-relevant facts to record, how to join and protect them, and who reads them.

Architecture choice: how much audit machinery#

SituationSimplest credible choiceGrow whenAlternatives
One application, few usersstructured records for sign-in, denials and sensitive writes, with request id, principal and policy version; redaction; a separate table the app can only insert intomore than one service, or an auditor who must trust the recordsthe identity provider's own sign-in log plus application records
Several servicesone security event stream with shared correlation ids; traces separate; a leak scan in CIregulated data, or agents acting for peopleper-service logs joined at query time
Agents acting for peopleadd run id, on_behalf_of, every tool call and every refused proposal; no prompts in recordsmany agents, external agentsa separate store for content, referenced by id
Regulated or multi-partywrite-once storage, independent checkpoints of the chain, audited access to the trail, SIEM rulesdisputes with partnerscontractual log exchange with partners (Agent-to-agent trust)
In a coding agent. A coding agent's actions show up in records it does not write: the repository's pull request and review history, the CI run log, and the cloud provider's record of which CI identity assumed which role. Because CI reaches the cloud through OIDC federation, the cloud's log names the repository and branch that asked, not a shared key; because a person merges, the merge record names that person. Join them with the pull request number and the CI run id. Depth on Coding agents.

Security notes#

Monday checklist#

Exercises#

Exercise 1. A developer adds the full HTTP request to every authz.deny record "to help debugging". What two problems does that create, and what do you keep instead?

AnswerTokens and user content in the trail; keep ids and decisions

The request carries the bearer token or cookie, and its body may carry user content. Keep the request id, trace id, action, resource, rule and reason; the trace system holds the debugging detail for its short lifetime.

Exercise 2. An insider recomputes every hash from record 10 to record 27. Does verify_chain notice? What would?

AnswerNo; an independent checkpoint or write-once copy would

A fully recomputed chain verifies. A copy shipped at write time to storage the insider cannot change, or a latest-hash checkpoint published elsewhere, no longer matches.

Exercise 3. The business asks for the prompts the agent saw on run-7f3a to be kept "for the audit". Where do they go, if anywhere?

AnswerA separate content store with its own access control, referenced by run id

Not in security events or audit records. If they must be kept, store them separately with restricted access and a retention decision, linked from the audit records by run_id, and redact credentials from them before storage.

Advanced perspective#

When the audit write failsThe record is part of the transaction for sensitive writes

If the decision on SUB-1004 is authorized but its audit record cannot be written, allowing the write leaves a decision nobody can attribute. The example's cockpit_record_decision appends the audit record before storing the decision, so an audit failure surfaces before the effect. For sensitive writes and privileged actions, refuse the action when no durable record can be written, locally or remotely; reads may continue with the gap recorded. The full fallback design, including a protected local buffer, is on Operations.

Logging every allowVolume, value and the decisions you will be asked about

Logging every denial is cheap because denials are rare. Logging every allow can be the largest data stream in the system. Two compromises are common. Log every allow for writes, privileged actions, delegated and agent actions and cross-unit reads, and aggregate plain reads (counts per principal, resource class and hour). Or log every allow with a short retention and keep the long-retention trail for the first set. What you must not do is sample decisions randomly: the one you dropped is the one someone will ask about. Policy engines often produce decision logs of their own; the trade-offs are on Policy as code.

Recording delegation honestlyTwo principals, one record, and the refused proposals too

An agent's request is made by the agent for a person. A record that names only the agent hides who asked; one that names only the person hides who acted. The example writes principal_id (the actor from the token's outermost act) and on_behalf_of (the token's sub). In a chain of actors, record the whole nested chain as history, while access control used only the outermost actor. Also record refusals of model proposals, as the example does for record 21: they are the only evidence that an injection was attempted and stopped, and the pattern of refused proposals over many runs says more about an agent's exposure than any single run.

Audit records contain personal data tooRetention and access are a privacy decision as well as a security one

Principal ids, tenants, resources and times describe what people did; that is personal data in many jurisdictions. The trail needs a retention period chosen deliberately, access limited to the people with a purpose, and a way to answer requests about it without breaking integrity (for example, pseudonymous principal ids resolved through a separately controlled directory). Integrity and erasure pull in opposite directions; deciding which records are kept for how long, and on what legal basis, belongs to the organization's governance, not to the logging library's defaults.

In one breath#

A diagnostic trace says how a request ran, a security event says something needs attention now, and a durable audit record says who did what to which record under which policy; they have different readers and lifetimes, and only traces may be sampled. Correlation ids join them: request_id and trace_id per inbound request, run_id across an agent's requests, principal_id with on_behalf_of for delegation, and authz_policy_version for the rules that decided. No token, secret, prompt or document content enters a record; a hash chain makes edits visible, storage outside the writer's control makes them hard, and only the auditor reads the result.

The trail now shows every refusal the earlier pages promised: cross-tenant reads, revoked and retired tokens, the agent's injected requests and the person's decision. Each was produced by a test written in advance. Threat modeling and testing turns to how those tests are chosen: the trust boundaries, the attacks worth proving impossible, and the negative tests that make each control evidence instead of intention.