Three months after run-7f3a, the auditor u-auditor-1 receives a question from the business: "The roof on PROP-2207 turned out to be fine and the broker complains about the inspection delay. Who decided to send an inspector on SUB-1004? Was it the AI? And did anything ever try to approve that submission without a person?"
The agent's prompts were never kept, and the tracing system's detailed spans expired long ago. What is left is whatever the system chose, at the time, to write down as a record of who did what.
If that record was designed well, the auditor answers in minutes: the decision was recorded at 09:12:02 by u-uw-w1, a person, through the cockpit, after a phishing-resistant step-up, under authorization policy authz-2026.11.1; the agent recommended refer; and at 09:05:32 the agent did try to call record_underwriter_decision with approve, an instruction it found in a broker's attachment, and was refused. If the record was designed badly, the answer is a guess, or the logs hold the underwriter's bearer token in plain text and the audit becomes an incident of its own.
This page is about the design that makes the first outcome normal.
Three records that look alike and are not#
Every request in the example produces output that could be called "a log". Three different things are hiding in that word, and the first job is to keep them apart.
A diagnostic trace says how a request ran: which services it passed through, how long each took, where it failed. Engineers read it to fix performance and bugs. Tiny example: the trace of req-0015 shows the tool server's span and its duration.
A security event says that something security-relevant happened, so that a person or a rule can react now. Tiny example: "six requests with bad signatures in ten seconds" is a set of security events; security operations should see them while they happen.
A durable audit record says who did what to which record, under which policy, with what result, and it is kept so that the question can be answered later. Tiny example: "the decision on SUB-1004 was recorded by u-uw-w1 under authz-2026.11.1". The series' word for one such entry is an audit record (Identity and trust primitives defines it); the ordered, protected collection of them is the audit trail.
| Diagnostic trace | Security event | Durable audit record | |
|---|---|---|---|
| Question it answers | how did this request run? | is something bad happening now? | who did what, to which record, under which policy? |
| Main reader | engineers on call | security operations, alert rules | auditors, investigators, the business |
| Typical content | spans, timings, errors, service names | failed sign-ins, denials, revocations, key changes | principal, on behalf of, action, resource, decision, rule, policy version |
| Lifetime | short, set by operations need; sampling is normal | long enough to correlate an attack | set by policy or regulation; never sampled |
| Integrity | best effort | protected in transit to the analysis system | tamper evident, append only, stored outside the writer's control |
| In the example | not modelled; a trace_id and span_id on every record point to it | authn.failure, authz.deny, token.revoked, key.rotated | authz.allow and authz.deny with rule and authz_policy_version; the human_decision |
The row that matters most is lifetime. A trace system that samples is doing its job; an audit trail that samples is broken. That is why the three must not share one pipeline with one retention setting.
Before AI. Backend teams already run all three, usually without naming the difference. The web server's access log, an APM tool's traces, and an audit table that a Spring service writes on every state change are three systems for a reason. Agents add new kinds of events (tool calls, delegations, refusals of model proposals) but not a new category.
The example's audit.py writes the second and third kinds into one append-only log, and leaves traces to the tracing system. Logs, metrics and traces as an observability discipline are taught on observe.sammuthu.com.
What counts as a security event#
A security event is any occurrence that changes or tests who can do what. The example's incident timeline (incident.py) runs its own services on a simulated clock and writes 27 records of seven kinds:
| Kind | Count in the timeline | Written by | Example record |
|---|---|---|---|
authn.failure | 8 | the gateway | six forged signatures (req-0001 to req-0006), a revoked token (req-0011), a retired key (req-0020) |
authn.success | 4 | the gateway | a broker's token validated before the quote service decides |
authz.allow | 7 | quote service, tool server, cockpit | the agent's roof read; the underwriter's decision |
authz.deny | 5 | quote service, tool server, cockpit | two cross-tenant reads (404), two refused tool calls, one decision without step-up (401) |
token.revoked | 1 | security operations | the token of u-broker-s1 after cross-tenant probing |
key.rotated | 1 | security operations | demo-2026a to demo-2026b, overlap begins |
key.retired | 1 | security operations | demo-2026a removed from the verifier key set |
Two kinds the spec of this site asks for are not in the timeline, and they belong in a real one. Token issuance: the example's token service does not write an event when it issues a token, so a forged token that was accepted could not be told apart from a legitimate one by its jti (Operations explains why that matters after a key compromise). Privileged elevation: the example's PAM module keeps its own trail (requested, approved, action, expired; see Privileged access), separate from this log. This site's recommendation: write issuance, elevation, approval and every tool call into the same security event stream, so one query sees them all.
Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "Where possible, always log" includes "Authentication successes and failures" and "Authorization (access control) failures". Guideline (OWASP Authorization Cheat Sheet): one of its headings is "Implement Appropriate Logging". Standard (A2A 1.0.0, §13): agents "SHOULD log security-relevant events (authentication failures, authorization denials, suspicious requests)". Three documents, one floor: failures and denials are always written.
The smallest primitive: one record#
Here is one record exactly as the example wrote it, the second cross-tenant read: the broker u-broker-s1 of tenant brk-south asking for Q-5531, which belongs to brk-north.
{
"seq": 10,
"time": "2026-11-18T09:01:32Z",
"kind": "authz.deny",
"principal_id": "u-broker-s1",
"decision": "deny",
"reason": "record belongs to another tenant: not found (existence not revealed)",
"request_id": "req-0008",
"run_id": null,
"trace_id": "390fda6d572fbd48f316d4b43cdb9e15",
"span_id": "f17a325ccec201a3",
"action": "quote.read",
"resource": "Q-5531",
"tenant": "brk-south",
"business_unit": null,
"rule": "R-AZ-03",
"authz_policy_version": "authz-2026.11.1",
"status": 404,
"prev_hash": "803ee0bb7a2a73152234f050e641d4bcc4ea1f1f4e6af0b17efc79d9b64279e2",
"hash": "3094a55624ce63e654889b62b9d4196b1484b477c5157107ff4f8f8d419f008d"
}
Read it as the answer to four questions. Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "The application logs must record "when, where, who and what" for each event." When: time. Who: principal_id and the caller's tenant. What: action on resource, with decision, status and reason. Where: the request_id and trace_id tie it to the edge request and to its trace. Under which rules: rule and authz_policy_version.
Note what is absent. There is no token, no Authorization header, no cookie, and no copy of the quote. The broker saw a 404 and the record says so; it does not record the quote's contents to prove the point. The last two fields, prev_hash and hash, make the record part of a chain; they come up below.
Correlation: which id joins what#
A single question ("did anything try to approve SUB-1004?") touches records written by different services at different times. A correlation id is an identifier carried across those records so they can be joined. The example carries several, and each joins something different:
| Field | What one value identifies | What it lets you join |
|---|---|---|
request_id | one inbound HTTP request at the edge | every record about that request, at every hop |
trace_id | one trace; on this site one trace per inbound request through every hop | the audit record and the diagnostic trace of the same request |
span_id | one hop of that request | the record and the exact span in the trace |
run_id | one execution of the agent workflow for a submission | every request the agent made in that run, across many traces |
principal_id | who acted (with on_behalf_of when delegated) | everything one principal did, across runs |
tenant or business_unit | the caller's tenant or unit | probing across tenants; activity per unit |
authz_policy_version | the authorization policy that decided | which decisions a policy change affects |
jti | one token | where a stolen or revoked token was used (logged as an id, never with the token) |
A request id is created at the edge, once per inbound request, and passed to every service that handles it. A trace id is the tracing system's identifier for the same journey; on this site each inbound request starts one trace, so request_id and trace_id correspond one to one. A run id is wider: run run-7f3a made six tool requests in the timeline, each its own inbound request with its own trace, so the run spans six traces. Agents are the reason run_id exists: without it, an agent's twelve requests look like twelve unrelated calls.
Production pitfall. Correlation ids must be created by your code at a trust boundary, not accepted from the caller. A client that sends its own request_id can make its requests look like someone else's in the trail. Accept an inbound id only as a separate, labelled field.
The spec of this site names tenant_id and policy_version; the example writes tenant (or business_unit) and authz_policy_version, the second on purpose: the business policy UW-GUIDE 2026.2 also has a version, and a record that says "policy version" without saying which policy invites the wrong answer (Policy as code explains the drift case UW-A09).
Guideline (OWASP Microservices Security Cheat Sheet, read 2026-11-18): structured logs with correlation IDs, and "Exclude secrets and unnecessary sensitive data before the microservice emits a log entry".
How the example writes a record#
AuditLog.append in audit.py builds the record with the correlation ids, redacts it, then chains it to the previous record. This is the part that matters, quoted from the file:
rec = redact(copy.deepcopy(rec))
prev = self.records[-1]["hash"] if self.records else GENESIS
rec["prev_hash"] = prev
rec["hash"] = _hash(prev, {k: v for k, v in rec.items() if k not in ("hash", "prev_hash")})
self.records.append(rec)
The order is deliberate. Redaction runs before hashing, so the chain protects the redacted form and nothing secret is ever hashed into a value someone might later try to reverse. _hash is SHA-256 over the previous hash plus the record serialized as canonical JSON (sorted keys, no spaces), so the same record always gives the same hash.
The snippet replays the incident timeline, checks the chain, scans everything written for token-shaped strings and key material, then plays an insider who edits one record. It demonstrates tamper evidence and redaction in the example's own log; it does not demonstrate a production audit store, which must also keep the records somewhere the writer cannot change.
import copy
import json
import audit
import incident
from audit import AuditLog, leaks, redact
out = incident.run()
log = AuditLog()
log.records = out["records"]
print("records:", len(log.records), "| chain:", log.verify_chain())
print("token or key strings found:", leaks(log.dump()))
# An insider edits record 10 (a denied cross-tenant read) to say "allow".
edited = AuditLog()
edited.records = copy.deepcopy(log.records)
edited.records[9]["decision"] = "allow"
print("edit record 10 only:", edited.verify_chain())
# The insider also recomputes record 10's own hash, so record 10 looks consistent.
r10 = edited.records[9]
r10["hash"] = audit._hash(r10["prev_hash"], {k: v for k, v in r10.items() if k not in ("hash", "prev_hash")})
print("edit and rehash 10: ", edited.verify_chain())
# Redaction runs on every record before it is written.
print(json.dumps(redact({"authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4In0.c2ln",
"reason": "client sent eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4In0.c2ln", "jti": "jti-d5ecb5057011"})))
Run in a scratch copy of examples/uw-auth on 2026-11-18 (the agent in the example is a scripted stand-in, so these results show how the access controls work, not how good any model is):
records: 27 | chain: {'intact': True, 'first_bad_seq': None}
token or key strings found: []
edit record 10 only: {'intact': False, 'first_bad_seq': 10}
edit and rehash 10: {'intact': False, 'first_bad_seq': 11}
{"authorization": "[REDACTED]", "reason": "client sent [REDACTED:token]", "jti": "jti-d5ecb5057011"}
The token string in the last line is a fabricated, unsigned fragment written for this page; the redactor removes it by shape.
Integrity: tamper evidence, not tamper proof#
Audit integrity is the property that a record, once written, cannot be changed or removed without the change being detectable. The example's chain gives a precise version of it. Each record's hash covers the previous record's hash, so changing record 10 breaks record 10's own hash check. An insider who also recomputes record 10's hash moves the break to record 11, whose prev_hash no longer matches. To hide the edit, the insider has to recompute every hash from record 10 to the end.
That last sentence is the limit. A chain stored next to the code that writes it detects careless tampering and accidental corruption. It does not stop someone who controls the whole log and recomputes it. The docstring of audit.py says it plainly: tamper evidence, not tamper proof. Three practices close the gap, and none is in the example:
- Ship records as they are written to storage the writing service cannot modify or delete (write-once storage, or a separate account the service can only append to).
- Publish the latest hash somewhere independent at intervals (another system, a signed checkpoint), so a recomputed chain no longer matches the published value.
- Record and monitor every access to the audit store itself.
Guideline (OWASP Logging Cheat Sheet, read 2026-11-18): "Build in tamper detection so you know if a record has been modified or deleted"; "Store or copy log data to read-only media as soon as possible"; "All access to the logs must be recorded and monitored (and may need prior approval)".
Redaction: what never enters a record#
Log redaction is removing or masking sensitive values before a record is written, so the trail never becomes a store of credentials or personal data. A token in a log is a token anyone with log access can use until it expires; a password in a log is a breach.
The example's redact works three ways. Values of sensitive keys (authorization, access_token, cookie, client_secret, code_verifier and others) become [REDACTED]. Anything shaped like a compact JWS or an opaque token handle, wherever it appears, becomes [REDACTED:token], which catches the token pasted into an error message. And demo key material becomes [REDACTED:key]. The token's jti stays: it is an identifier, not a credential, and it is what lets an investigator ask where one token was used. This site's own case UW-A24 scans every record of the run and the timeline and finds nothing (leaks=[]), with both hash chains intact.
Guideline (OWASP Logging Cheat Sheet): data that "should usually not be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed, or encrypted" includes "Session identification values", "Access tokens", "Authentication passwords" and "Encryption keys and other primary secrets".
Agents add a category of their own: unredacted user content. The broker's attachment A2 contained an injected instruction. The audit record of the refused call (record 21) names the action, the resource, the principal, on whose behalf, the reason and the policy version; it does not copy the attachment text, the prompt or the model's output. This site's recommendation: security events and audit records carry identifiers and decisions, never prompts, retrieved passages or document contents. If an investigation needs the content, keep it in a separate store with its own access control and retention, referenced by id from the record.
Security note. Redaction by key name alone fails the day a developer logsstr(request). Redact by shape as well, scan written records for token-shaped strings in a test (astest_negative.pydoes), and treat a hit as a production incident.
Who may read the audit trail#
An audit trail every engineer can read is a second copy of the business. Reading is its own permission. In the example, audit.read is a scope only the auditor holds, and rule R-AZ-10 grants reading audit records to auditors only. The snippet asks the policy decision point for three principals, then tries the tool server's read_audit_log tool.
import mcp_matrix
import pdp
from clock import START_EPOCH_S as NOW
from directory import subject_for
audit_log = {"type": "audit_log"}
subs = mcp_matrix.subjects(NOW)
for label, s in (("auditor u-auditor-1", subject_for("u-auditor-1", now=NOW)),
("underwriter u-uw-w1", subject_for("u-uw-w1", now=NOW)),
("underwriting agent", subs["underwriting_agent"])):
r = pdp.decide(s, "audit.read", audit_log, {"now": NOW})
print(f"{label:<20} {r['status']} {r['rule']}: {r['reason']}")
for label, key in (("auditor", "auditor"), ("underwriting agent", "underwriting_agent")):
r = mcp_matrix.execute(subs[key], "read_audit_log", {}, now=NOW)
print(f"read_audit_log as {label:<18} allowed={r['allowed']} check={r['check']}")
auditor u-auditor-1 200 R-AZ-10: auditor reads audit records
underwriter u-uw-w1 403 R-AZ-02: token scope lacks audit.read (insufficient_scope)
underwriting agent 403 R-AZ-02: token scope lacks audit.read (insufficient_scope)
read_audit_log as auditor allowed=True check=pdp
read_audit_log as underwriting agent allowed=False check=scope
The underwriter whose decision is in the trail cannot read it through the API, and the agent cannot read it at all. The auditor reads and writes nothing: u-auditor-1 holds no write scope, which is the separation of duties between the people who act and the person who checks (Governance). This site's recommendation: also log every read of the audit trail as a security event, and keep administrative access to the audit store (deleting, changing retention) under just-in-time elevation with a second approver (Privileged access).
The incident investigator#
The lab below holds the 27 records of the timeline. Filter by kind, decision, principal or id; reveal the answer to each of seven questions and see the browser recompute it from the records; edit record 10 and watch the chain break at 10, or at 11 if the insider also recomputes record 10's hash.
The recorded answers, with the fields that produce each one:
| Question | Answer | Answered from |
|---|---|---|
| Which requests failed authentication with a bad signature? | req-0001 to req-0006 | kind authn.failure, reason signature, request_id |
| Which requests were cross-tenant reads, and what did the caller see? | req-0008, req-0009 (u-broker-s1); 404, existence not revealed | kind authz.deny, status 404, tenant brk-south |
| How many traces does run run-7f3a span? | 6 (one trace per inbound request) | distinct trace_id where run_id is run-7f3a |
| Was a revoked token used again, and was it refused? | yes: req-0011, rejected by the gateway (revocation) | authn.failure with reason revocation, after token.revoked |
| Which request used a token signed with the retired key? | req-0020, rejected (kid demo-2026a retired at 09:22Z) | authn.failure with reason kid, after key.retired |
| Which tool calls of run-7f3a were refused? | record_underwriter_decision, record_recommendation (for SUB-2001) | decision deny, run_id run-7f3a, action |
| Who recorded the underwriting decision? | u-uw-w1, a person, through the cockpit, after step-up | action decision.record, decision allow, principal_id |
Chain status: intact (27 of 27 hashes recompute). No token, key or secret appears in any record.
The last two rows answer the auditor's question from the opening. The agent's attempt is record 21: principal_id wl-underwriting-agent, on_behalf_of u-uw-w1, action record_underwriter_decision, rule null because the tool server's scope check refused it before the policy decision point, reason insufficient_scope: uw.decide required. The decision is record 25, by u-uw-w1, rule R-AZ-09, acr phishing-resistant, policy authz-2026.11.1. The response to incidents built on these answers is on Operations.
Diagram: One refused tool call, three destinations · same ids, different contents and readers
The request is req-0015 of run run-7f3a: the underwriting agent, acting for u-uw-w1, proposes record_underwriter_decision on SUB-1004 and is refused at the scope check. Rows are the three destinations; columns are what each keeps, who reads it and how long it lives. All three carry the same request_id and trace_id, so they can be joined, but only the audit record is kept and never sampled.
- Diagnostic trace
- spans, timings, errors: The tool server's span and how long it took, under trace_id acfd2b61...
- engineers on call: Used to fix latency and bugs.
- short, sampled: Sampling is normal here, which is why it cannot be the audit trail.
- Security event
- authz.deny on a decision action: An agent principal proposed a tool it may not hold.
- security operations, SIEM rules: A rule can flag decision tools proposed by agents.
- long enough to correlate: Protected in transit to the analysis system; never sampled.
- Durable audit record
- principal, on behalf of, action, rule, version: wl-underwriting-agent for u-uw-w1, record_underwriter_decision on SUB-1004, deny, rule null, authz-2026.11.1.
- the auditor only: audit.read is held by u-auditor-1; no token or content is in the record.
- policy retention, hash-chained: Append only, tamper evident, shipped to storage the writer cannot change.
The diagram follows one request, the agent's refused call, into the three destinations. The important observation is that each destination keeps a different subset for a different reader, joined by the same ids.
Security events to a SIEM, and the alerts worth having#
A SIEM (security information and event management system) collects security events from many systems, stores them for search, and runs detection rules that raise alerts. The audit log is one of its sources; it does not replace the durable trail, which may live elsewhere with longer retention.
The timeline contains five patterns a rule could catch, each from fields already in the records:
| Pattern in the timeline | Rule (illustrative) | Records |
|---|---|---|
| Six bad signatures in ten seconds for one claimed subject | burst of authn.failure with reason signature | 1 to 6 |
| One principal getting 404 on another tenant's records | authz.deny, status 404, same principal, more than one resource | 10, 12 |
| A revoked token presented again | authn.failure with reason revocation | 17 |
| An agent run proposing a decision tool it may not hold | authz.deny on a decision action where the principal is an agent | 21 |
| A token signed with a retired key | authn.failure with reason kid, after key.retired | 27 |
The thresholds are design choices, not numbers from any standard. Signals can also arrive from outside. Standard (OpenID Shared Signals Framework 1.0 and CAEP 1.0, listed Final, read 2026-11-18): a transmitter streams events about subjects to receivers as Security Event Tokens (RFC 8417). Standard (RFC 9967, May 2026): a SCIM profile for Security Event Tokens, so identity changes such as a deactivation reach receivers without polling. Both bring events in; they do not replace your own records of what your services decided.
Diagram: The hash chain · what an edit breaks, and what only an outside copy catches
Each record's hash covers the previous hash and the record's canonical JSON. Editing record 10 breaks the check at 10. Recomputing record 10's hash moves the break to 11. Recomputing every hash to the end verifies again, which is why records are also shipped to storage the writer cannot change, or the latest hash is checkpointed somewhere independent.
- The log the service writes
- record 9: authn.success for u-broker-s1, req-0008.
- record 10: authz.deny, 404, cross-tenant read of Q-5531.
- record 11: Its prev_hash still holds the original h10.
- Outside the writer's control
- write-once copy: A copy the writing service cannot modify or delete.
- published checkpoint: The latest hash, recorded independently at intervals.
The second diagram shows the hash chain and the two edits from the snippet, and where an independent checkpoint would catch the third.
What goes wrong without these rules#
| Failure | What the auditor finds | Control |
|---|---|---|
| The bearer token is logged with the request | the audit store is now a token store; the audit is an incident | redaction by key and by shape; a leak scan in tests (UW-A24) |
| Denials go only to the trace system, which samples | the agent's refused call is gone | security events are never sampled |
No authz_policy_version on decisions | cannot tell whether the peer's decision was allowed under the old policy (UW-A09) | record the authorization policy version on every decision |
No run_id | the agent's six requests look unrelated | a run id on every request an agent makes |
No on_behalf_of | the agent's actions look like the underwriter's, or the reverse | record both principals on delegated requests |
| The writing service can rewrite the log | an edited denial is indistinguishable from the original | hash chain plus storage outside the writer's control |
| Everyone can read the trail | the trail leaks the business it records | audit.read for auditors only; reads are logged |
What this is not. This page does not teach tracing, metrics or log pipelines as an engineering discipline (observe.sammuthu.com), how to respond to an incident and repair access (Operations), or how to design negative tests (Threat modeling and testing). It is about which security-relevant facts to record, how to join and protect them, and who reads them.
Architecture choice: how much audit machinery#
| Situation | Simplest credible choice | Grow when | Alternatives |
|---|---|---|---|
| One application, few users | structured records for sign-in, denials and sensitive writes, with request id, principal and policy version; redaction; a separate table the app can only insert into | more than one service, or an auditor who must trust the records | the identity provider's own sign-in log plus application records |
| Several services | one security event stream with shared correlation ids; traces separate; a leak scan in CI | regulated data, or agents acting for people | per-service logs joined at query time |
| Agents acting for people | add run id, on_behalf_of, every tool call and every refused proposal; no prompts in records | many agents, external agents | a separate store for content, referenced by id |
| Regulated or multi-party | write-once storage, independent checkpoints of the chain, audited access to the trail, SIEM rules | disputes with partners | contractual log exchange with partners (Agent-to-agent trust) |
In a coding agent. A coding agent's actions show up in records it does not write: the repository's pull request and review history, the CI run log, and the cloud provider's record of which CI identity assumed which role. Because CI reaches the cloud through OIDC federation, the cloud's log names the repository and branch that asked, not a shared key; because a person merges, the merge record names that person. Join them with the pull request number and the CI run id. Depth on Coding agents.
Security notes#
- Failures and denials are always written; authorization decisions carry the rule and the authorization policy version.
- No token, password, key, session id, prompt or document content in any record; redact by key and by shape, and scan.
- Correlation ids are created at your trust boundary; caller-supplied ids are kept separately and labelled.
- The trail is tamper evident in the application and tamper resistant in storage the writer cannot change.
- Reading the trail is a permission (
audit.read), and every read is itself logged. - Security events are never sampled; traces may be.
Monday checklist#
- Search your logs for
Bearer,eyJand your API key prefixes. Every hit is a finding. - Pick one denied request from yesterday and find it in your security events. If it is only in traces, it can be sampled away.
- Check that decision records carry the authorization policy version, not only "policy version".
- For one agent run, list its requests by run id. If you cannot, add a run id.
- Find out who can delete or edit your audit store, and whether that action is logged.
- Write one SIEM rule from the table above against your own events.
Exercises#
Exercise 1. A developer adds the full HTTP request to every authz.deny record "to help debugging". What two problems does that create, and what do you keep instead?
AnswerTokens and user content in the trail; keep ids and decisions
The request carries the bearer token or cookie, and its body may carry user content. Keep the request id, trace id, action, resource, rule and reason; the trace system holds the debugging detail for its short lifetime.
Exercise 2. An insider recomputes every hash from record 10 to record 27. Does verify_chain notice? What would?
AnswerNo; an independent checkpoint or write-once copy would
A fully recomputed chain verifies. A copy shipped at write time to storage the insider cannot change, or a latest-hash checkpoint published elsewhere, no longer matches.
Exercise 3. The business asks for the prompts the agent saw on run-7f3a to be kept "for the audit". Where do they go, if anywhere?
AnswerA separate content store with its own access control, referenced by run id
Not in security events or audit records. If they must be kept, store them separately with restricted access and a retention decision, linked from the audit records by run_id, and redact credentials from them before storage.
Advanced perspective#
When the audit write failsThe record is part of the transaction for sensitive writes
If the decision on SUB-1004 is authorized but its audit record cannot be written, allowing the write leaves a decision nobody can attribute. The example's cockpit_record_decision appends the audit record before storing the decision, so an audit failure surfaces before the effect. For sensitive writes and privileged actions, refuse the action when no durable record can be written, locally or remotely; reads may continue with the gap recorded. The full fallback design, including a protected local buffer, is on Operations.
Logging every allowVolume, value and the decisions you will be asked about
Logging every denial is cheap because denials are rare. Logging every allow can be the largest data stream in the system. Two compromises are common. Log every allow for writes, privileged actions, delegated and agent actions and cross-unit reads, and aggregate plain reads (counts per principal, resource class and hour). Or log every allow with a short retention and keep the long-retention trail for the first set. What you must not do is sample decisions randomly: the one you dropped is the one someone will ask about. Policy engines often produce decision logs of their own; the trade-offs are on Policy as code.
Recording delegation honestlyTwo principals, one record, and the refused proposals too
An agent's request is made by the agent for a person. A record that names only the agent hides who asked; one that names only the person hides who acted. The example writes principal_id (the actor from the token's outermost act) and on_behalf_of (the token's sub). In a chain of actors, record the whole nested chain as history, while access control used only the outermost actor. Also record refusals of model proposals, as the example does for record 21: they are the only evidence that an injection was attempted and stopped, and the pattern of refused proposals over many runs says more about an agent's exposure than any single run.
Audit records contain personal data tooRetention and access are a privacy decision as well as a security one
Principal ids, tenants, resources and times describe what people did; that is personal data in many jurisdictions. The trail needs a retention period chosen deliberately, access limited to the people with a purpose, and a way to answer requests about it without breaking integrity (for example, pseudonymous principal ids resolved through a separately controlled directory). Integrity and erasure pull in opposite directions; deciding which records are kept for how long, and on what legal basis, belongs to the organization's governance, not to the logging library's defaults.
In one breath#
A diagnostic trace says how a request ran, a security event says something needs attention now, and a durable audit record says who did what to which record under which policy; they have different readers and lifetimes, and only traces may be sampled. Correlation ids join them: request_id and trace_id per inbound request, run_id across an agent's requests, principal_id with on_behalf_of for delegation, and authz_policy_version for the rules that decided. No token, secret, prompt or document content enters a record; a hash chain makes edits visible, storage outside the writer's control makes them hard, and only the auditor reads the result.
The trail now shows every refusal the earlier pages promised: cross-tenant reads, revoked and retired tokens, the agent's injected requests and the person's decision. Each was produced by a test written in advance. Threat modeling and testing turns to how those tests are chosen: the trust boundaries, the attacks worth proving impossible, and the negative tests that make each control evidence instead of intention.