At 09:03:37 security operations revokes the access token of the broker u-broker-s1. Shortly before, that token was used to ask for two quotes of another brokerage, Q-5531 and Q-5532; both requests got 404. The revocation is the right call, and it takes one line. Ten seconds later, at 09:03:47, the same token arrives at the gateway again.
What happens next depends on things nobody decided during the incident. If the gateway checks a revocation list, the request fails. If it checks only the signature and the expiry, the token works for another quarter of an hour. If the policy decision point cached an earlier answer, a stale allow may be served for minutes more. And if the audit service is down at that moment, nobody may ever know which of these happened. In the example, the gateway checks the list and refuses (request req-0011, rejected at "revocation"). Most real systems have places where it would not.
This page is about running an authentication and authorization system while it changes and while parts of it fail: rotating keys and secrets without an outage, revoking access in an emergency, invalidating cached decisions, choosing what fails closed, and writing down how fast access must disappear.
Operational lifecycles, in plain words#
Every credential and every decision in this site has a lifetime, and every lifetime ends either on schedule or in an emergency. An operational lifecycle is the planned path for each one: how it is replaced while the system keeps serving (rotation), how it is withdrawn early (revocation), and what the system does when the component that issues or checks it is unavailable (fallback). A tiny example: the identity provider's signing key demo-2026a will be replaced by demo-2026b; the replacement must not cause a single failed request, and the old key must stop working on a known day.
Before AI. Backend engineers know these jobs under older names: rotating a database password across twenty application servers without a restart, the certificate renewal cron job that someone forgot, an LDAP outage that locked everyone out of the admin console, a deny list for stolen session ids. The mechanics on this page are the same jobs, written down and tested before they are needed.
Rotation without downtime: the overlap#
A signing key cannot be swapped in one step. Tokens signed with the old key are still in flight, and verifiers learn about new keys only when they refresh their key set. Rotation is therefore an overlap: for a period, verifiers accept both keys. The order of the steps decides whether anyone notices. Zero-downtime rotation is that ordering.
The snippet signs one demo token with each key and verifies both against the three key sets a verifier holds over a rotation. It demonstrates why the order matters; it does not demonstrate production key handling. The demo keys are HMAC strings that say they are not secrets, published on purpose: use a maintained library and asymmetric keys from an identity provider in production.
import demo_keys
import tokens
from clock import START_EPOCH_S as NOW
old = tokens.issue_for("u-broker-n1", aud=tokens.API_AUD, now=NOW, kid="demo-2026a")
new = tokens.issue_for("u-broker-n1", aud=tokens.API_AUD, now=NOW, kid="demo-2026b")
stages = [("before: verifier knows a", ("demo-2026a",)),
("overlap: verifier knows a and b", ("demo-2026a", "demo-2026b")),
("retired: verifier knows b", ("demo-2026b",))]
for label, kids in stages:
ks = demo_keys.keyset(*kids)
res = []
for name, tok in (("a-signed", old), ("b-signed", new)):
v = tokens.verify(tok, keyset=ks, expected_issuer=tokens.ISSUER,
expected_audience=tokens.API_AUD, now=NOW)
res.append(name + "=" + ("valid" if v["valid"] else "fails at " + str(v["first_failed"])))
print(f"{label:<33} {' '.join(res)}")
Run in a scratch copy of the example on 2026-11-18 (the agent in the example is a scripted stand-in, so these results show how the access controls work, not how good any model is):
before: verifier knows a a-signed=valid b-signed=fails at kid
overlap: verifier knows a and b a-signed=valid b-signed=valid
retired: verifier knows b a-signed=fails at kid b-signed=valid
Read it as two possible outages. If the issuer starts signing with demo-2026b while some verifier is still in the first row, every new token fails at kid there. If the verifier drops demo-2026a while tokens signed with it are still alive, those fail. The safe sequence avoids both:
| Step | Who acts | What changes | Wait before the next step |
|---|---|---|---|
| 1. Publish | identity provider | the new public key appears in the key set (JWKS) next to the old one | at least the longest key-set cache time of any verifier |
| 2. Sign | identity provider | new tokens are signed with the new key | at least the longest token lifetime |
| 3. Retire | identity provider | the old key is removed from the key set | none |
| 4. Confirm | security operations | no verifier rejects new tokens; tokens signed with the old key are refused |
Standard (RFC 7517, read 2026-11-18): a JWK Set is "A JSON object that represents a set of JWKs" with a keys array, which is what makes step 1 possible: verifiers pick the key by kid from the set. The JWT anatomy and the verifier's checks are on Tokens; this page is about the timing.
The example's incident timeline (incident.py) performs this rotation inside a running system. Here are the records security operations wrote, with the request that later used an old-key token, from a scratch run on 2026-11-18:
13 09:02:37 key.rotated security-operations | signing key demo-2026a -> demo-2026b; verifiers accept both during the overlap | None
16 09:03:37 token.revoked security-operations | token of u-broker-s1 revoked after cross-tenant probing | None
17 09:03:47 authn.failure u-broker-s1 | gateway: revocation | req-0011
26 09:22:02 key.retired security-operations | demo-2026a removed from the verifier key set | None
27 09:22:22 authn.failure None | gateway: kid | req-0020
The overlap ran from 09:02:37 to 09:22:02, longer than the 900-second token lifetime used throughout the example (illustrative). After retirement, req-0020 presented a broker token signed with demo-2026a and failed at kid, the first check that can fail. That token had also passed its exp; with the checks in this order, the key check reports first.
Diagram: Rotating a signing key with no failed request · publish, wait, sign, wait, retire
Read left to right as one clock. The identity provider publishes demo-2026b next to demo-2026a and waits until every verifier has refreshed its key set. Then it signs with the new key and waits until every token signed with the old key has expired. Only then is demo-2026a retired. Skipping the first wait breaks new tokens at kid; skipping the second breaks tokens still in flight. After a key compromise the second wait is skipped on purpose.
- Identity provider
- publish b next to a: The new public key appears in the key set; nothing is signed with it yet.
- wait: longest key-set cache: Every verifier must have refreshed and know the new key before any token uses it.
- sign with b: New tokens carry kid demo-2026b. Tokens signed with demo-2026a are still alive.
- wait: longest token lifetime: Every token signed with the old key must expire before the key goes.
- Retire and confirm
- retire a: The old key is removed. Any token still signed with it fails at kid.
- confirm: Security operations checks that new tokens work and old-key tokens are refused.
The diagram lays the four steps on one clock. The two waits are the steps skipped under pressure, and each skip causes a different outage.
Secrets and certificates rotate on the same principle#
Not every credential is a signing key, but the overlap idea carries over. A database password or a vendor API key must be valid in both places, the secret store and the system it opens, during the switch. What three secret stores do for you is folded below.
How three secret stores handle rotationAWS rotates, Google notifies, Vault issues per lease
Product (AWS Secrets Manager, as documented on 2026-11-18): rotation comes as managed rotation ("doesn't use a Lambda function"), managed external secrets rotation for partners, and "Rotation by Lambda function" for other secrets, and it updates "the credentials in both the secret and the database or service." Product (Google Cloud Secret Manager, as documented on 2026-11-18): it does not rotate values itself; it sends a SECRET_ROTATE message to configured Pub/Sub topics on schedule, and your subscriber creates the new version. Product (HashiCorp Vault, as documented on 2026-11-18): the database secrets engine avoids long-lived passwords altogether by generating credentials per lease, "By default, TTL is 1 hour and max TTL is 24 hours".
The cheapest rotation is the one you never perform, because the credential is short-lived and issued per use. Federation in place of stored cloud keys is on Cloud IAM.
Certificates follow the same overlap: the new certificate is deployed and trusted before the old one expires. The certificate details, chains and renewal are on TLS, PKI and certificates. One operational fact changes revocation planning. Product (Let's Encrypt, as documented on 2026-11-18, SURPRISING and confirmed on a second fetch): it turned off its OCSP service in August 2025 and publishes revocation information "exclusively via Certificate Revocation Lists (CRLs)"; its six-day certificates were announced without OCSP or CRL URLs, on the reasoning that short-lived certificates "greatly reduce the potential compromise window". For public certificates, short lifetimes are replacing online revocation checks.
Emergency revocation: what each mechanism can reach#
Emergency revocation is withdrawing access before its scheduled end: a stolen token, a compromised key, a leaver who must be out now. The first thing to know is that "revoke" is not one operation. Each mechanism reaches a different place, and the gap between them is where access survives.
Return to the leaver of the governance page, u-uw-w2, deactivated at 10:00Z in this site's own case UW-A17. The lab below starts at the moment HR records the departure. Step forward and watch four things answer the same question differently: the directory, a self-contained token, an introspected opaque token, and the policy cache.
The recorded run (tokens 15 minutes, cache 5 minutes; checks are u-uw-w2 reading commercial-west after deactivation at 10:00Z):
| Time | Directory | Self-contained token | Introspection | Policy cache |
|---|---|---|---|---|
| 10:00Z, right after deactivation | denies | still accepted, 15 min left | inactive | hit, stale (serves allow) |
| 10:05Z | denies | still accepted, 10 min left | inactive | miss, denies |
| 10:15Z | denies | expired, not accepted | inactive | miss, denies |
The directory is right at once, but no request asks it; the token every request carries is the slowest to die.
| Mechanism | Who acts | What it can revoke | What it cannot revoke |
|---|---|---|---|
Deactivate the account (SCIM active: false) | identity governance | new sign-ins; new tokens from the identity provider | sessions, refresh tokens and access tokens already issued, unless the application acts on it |
| Revoke the server-side session | the BFF or application | the next request that uses the session cookie | tokens the session already handed to other services |
| Revoke a refresh token (RFC 7009) | client or identity provider | new access tokens from that grant; where supported, the server "SHOULD also invalidate all access tokens based on the same authorization grant" | self-contained access tokens at resource servers that never ask |
| Opaque token with introspection (RFC 7662) | resource server asks the issuer | the token, on the next introspection | a cached introspection result, until the cache expires (never beyond the token's exp) |
jti deny list | issuer distributes, resource server checks | the listed token, wherever the list has arrived | servers that have not received the list yet |
| Remove the signing key | identity provider | every token that key signed, at once | nothing it signed survives, including every legitimate token (the right trade after a key compromise) |
| Policy or directory change | policy administration | the next fresh decision | cached decisions, until invalidated or expired |
| Revoke a certificate | the certificate authority | clients that check the CRL | clients that do not; a short-lived certificate limits the window instead |
Shared Signals event (CAEP session-revoked) | identity provider as transmitter | sessions at receivers that subscribed and act on it | receivers that do not implement it |
| Back-channel logout (OpenID Connect) | identity provider posts a logout token | relying-party sessions that registered a backchannel_logout_uri | access tokens already at APIs |
Standard (RFC 7009, read 2026-11-18): servers MUST support revocation of refresh tokens and SHOULD support revocation of access tokens. Standard (RFC 7662): an introspection response "MAY be cached" and "MUST NOT be cached beyond" the token's exp. Standard (OpenID Connect Back-Channel Logout 1.0, Final): the identity provider POSTs a logout_token, a JWT that "MUST contain either a sub or a sid Claim". Standard (OpenID Shared Signals Framework 1.0 and CAEP 1.0, listed Final, read 2026-11-18): a transmitter streams events about subjects to receivers; CAEP's session-revoked "signals that the session identified by the subject has been revoked". Each of these is a message to a receiver; none reaches a component that does not listen. How a self-contained token survives revocation, and how a deny list should be keyed, is on Tokens.
Diagram: Where each revocation acts · at the source, or where requests are checked
Rows are three kinds of revocation; columns are where they act. Mechanisms at the source stop new credentials. Only mechanisms where requests are checked stop a credential that already exists, and only on the components that ask. That is why deactivating a leaver in the directory is instant and the self-contained token still works for 15 minutes in UW-A17.
- A person
- SCIM active = false: The directory denies at once and the identity provider issues nothing new.
- session revoked, CAEP or logout event: The application ends the session; receivers that subscribe hear a session-revoked or back-channel logout event.
- self-contained token, stale cache: Accepted until exp, served from cache until the entry expires.
- A token
- revoke at issuer: The issuer marks the token or its refresh grant revoked.
- deny list or introspection: A resource server that checks the list, or introspects, refuses it.
- servers that never ask: A resource server that checks only signature and exp keeps accepting the token.
- A key or certificate
- stop signing, revoke certificate: No new token or connection uses the key.
- remove from key set, CRL check: Verifiers refuse everything the key signed; clients that check the CRL refuse the certificate.
- clients that do not check revocation: A short certificate lifetime limits the window instead.
The diagram groups the mechanisms by where they act. Only those that act where requests are checked stop a credential that already exists.
Cached decisions and their invalidation#
The policy decision point is often called on every request, so services cache its answers. A cached decision is a copy of an authorization answer, and like any copy it goes stale. The example's lifecycle module caches for 300 seconds (illustrative): at 10:00Z, right after deactivation, the leaver's earlier allow is still in the cache and is served, marked STALE.
Policy cache invalidation is how a change in policy or in the facts it reads reaches the cached answers. There are three ways, and most systems combine them.
Expire. Every entry lives at most a fixed time. Simple, and the lifetime is the worst-case staleness.
Version. Each decision is cached with the version of the policy and of the subject's facts it used (authz-2026.11.1, a directory change counter). A change bumps the version and every older entry misses. The example's decisions already carry authz_policy_version for audit; the same field can key a cache.
Push. The directory or policy store sends an invalidation event, and caches drop matching entries. Fast, and it adds a delivery path that can fail, so pair it with a short expiry as a backstop.
Production pitfall. A cache keyed only on subject and action, with no resource, can serve one record's allow for another record. The example's key joins user, action and unit. Include whatever the decision depended on in the key, or do not cache it. Caching in the enforcement architecture is on Enforcement.
Fallback modes: when the identity provider or the policy decision point is down#
Every external dependency of a request can be unavailable. The design question is what the service does then, and the answer must be written down before the outage: an availability choice taken during an incident tends to become "let everything through".
The example answers it in code. When the policy decision point is unreachable, the enforcement point fails closed: pep_decide returns 503 with rule PEP-FAIL-CLOSED, "never allow". When the identity provider is down, no new authentication can be obtained, so an action that needs fresh strong authentication cannot proceed; but a token issued before the outage is still validated locally against the cached key set. This is case UW-A23:
import services
import tokens
from clock import START_EPOCH_S as NOW
from directory import ACR_STRONG
plat = services.Platform(NOW)
tok = tokens.issue_for("u-broker-n1", aud=tokens.API_AUD, now=NOW)
strong = {"principal_id": "u-uw-w1", "acr": ACR_STRONG, "auth_time": NOW}
weak = {"principal_id": "u-uw-w1", "acr": "urn:insurer.example:acr:password", "auth_time": NOW - 600}
rows = [
("PDP down, decision write", plat.cockpit_record_decision(strong, "SUB-1004", "refer_to_inspection", "x", "r1", pdp_available=False)),
("PDP down, quote read", plat.read_quote(tok, "Q-5531", "r2", pdp_available=False)),
("IdP down, write needs step-up", plat.cockpit_record_decision(weak, "SUB-1004", "refer_to_inspection", "x", "r3")),
("IdP down, read, valid token", plat.read_quote(tok, "Q-5531", "r4")),
]
for label, r in rows:
print(f"{label:<30} {r['status']} {r['decision']['rule']}: {r['decision']['reason']}")
print("decisions written:", len(plat.decisions))
PDP down, decision write 503 PEP-FAIL-CLOSED: policy decision point unavailable: fail closed
PDP down, quote read 503 PEP-FAIL-CLOSED: policy decision point unavailable: fail closed
IdP down, write needs step-up 401 R-AZ-09: step-up required: strong authentication within 300 s
IdP down, read, valid token 200 R-AZ-03: broker of tenant brk-north on its own quote
decisions written: 0
The four rows are not symmetric. During an identity provider outage, the underwriter's decision on SUB-1004 cannot be written (401: the step-up it needs cannot happen), while the broker's read of Q-5531 with a still-valid token continues (200): the quote service validates the token itself. During a policy decision point outage, both the write and the read fail closed with 503. Nothing was written in any row.
Common confusion. "Fail closed" does not mean "go down". During the identity provider outage in UW-A23, reads continue for everyone who already holds a valid token; only actions that need what is missing stop. The service stays partly available and never invents an allow.
Whether reads should also continue during a policy decision point outage is a real design choice. A service could serve reads from cached decisions within their normal lifetime, recorded as "served from cache". It must never turn "no answer" into "allow". This site's recommendation: writes and privileged actions fail closed with no exception; reads may use an explicit, short, recorded cache of earlier decisions; nothing allows by default. The example takes the strictest version: both fail.
When the audit log cannot be written#
The audit service is a dependency too. Suppose the underwriter's decision on SUB-1004 is authorized, but the audit record cannot be written. Allowing the write leaves a decision nobody can later attribute; refusing it stops underwriting. The example does not simulate an audit outage, but its code shows the order that matters: in cockpit_record_decision the audit record is appended before the decision is stored, so an audit failure surfaces before the effect.
This site's recommendation:
- For sensitive writes and privileged actions, the audit record is part of the transaction. If it cannot be durably written, either locally or to the audit service, the action is refused, like any other failed dependency.
- A durable local buffer (a write-ahead file or a queue on the same host) lets the service keep working through a short audit-service outage, as long as the buffer is itself protected, alerted and drained.
- Reads may continue with buffered or reduced logging, and the gap is recorded when the service returns.
- An audit outage is a security event in itself: it is the moment an insider would choose.
Redaction, correlation ids and hash-chained integrity are on Audit.
Incident response: contain, rotate, investigate, repair#
Incident response for access controls follows the same arc as any incident, with auth-specific steps. The example's incident timeline holds 27 redacted, hash-chained records that the example's own services wrote, and the investigator lab asks seven questions of them.
The recorded answers, for a reader without JavaScript:
| Question | Answer |
|---|---|
| Which requests failed authentication with a bad signature? | req-0001 to req-0006 |
| Which requests were cross-tenant reads, and what did the caller see? | req-0008, req-0009 (u-broker-s1); 404, existence not revealed |
| How many traces does run run-7f3a span? | 6 (one trace per inbound request) |
| Was a revoked token used again, and was it refused? | yes: req-0011, rejected by the gateway (revocation) |
| Which request used a token signed with the retired key? | req-0020, rejected (kid demo-2026a retired at 09:22Z) |
| Which tool calls of run-7f3a were refused? | record_underwriter_decision, record_recommendation (for SUB-2001) |
| Who recorded the underwriting decision? | u-uw-w1, a person, through the cockpit, after step-up |
Chain status: intact (27 of 27 hashes recompute). No token, key or secret appears in any record.
The timeline maps onto four steps. Contain: revoke the specific credential (the broker token, 09:03:37) and confirm the refusal (req-0011). Rotate: if a key or secret may be exposed, rotate it; for a suspected signing-key compromise, retire the old key without waiting for in-flight tokens to expire, because invalidating them is the goal. Investigate: follow request, trace and run ids from the first failure to the last success, and check that no other tenant's record returned data (both probes got 404). Repair: fix the cause and narrow what was too broad.
That last step has a name worth keeping. Least-privilege repair is the work after an incident or a review that removes access found to be broader than needed: the group u-uw-old should not have had, the export rights svc-legacy-export should not have kept, a scope that an agent never used. It is where incidents turn into smaller permissions instead of longer runbooks. Decisions in the example carry the authorization policy version, so a repair can be shown to apply from a known version onward.
For an outage that takes down more than access control, the recovery of the whole system, backups and failover belong to reliable.sammuthu.com.
The access-removal target: how fast must access disappear?#
An access-removal target, often written as an access-removal SLA, is the maximum time an organization accepts between the decision "this person or credential must lose access" and the moment no request succeeds with it. The number is an organizational choice: no standard read for this page sets one, and the right value depends on what the access protects. What engineering can do is compute what a design actually delivers, and compare.
The worst case is the sum of the slowest paths. In the example, with its illustrative settings:
| Path | Example value | Where it comes from |
|---|---|---|
| HR event to directory deactivation | 0 s (same minute) | the scripted timeline; real feeds take longer |
| Self-contained access token still accepted | up to 900 s | token lifetime; zero where every service introspects or checks a deny list |
| Cached decision still served | up to 300 s | policy cache lifetime, overlapping the token window |
| Worst case in this example | 900 s (15 minutes) | the longest path, since the cache window falls inside the token window here |
If the target is shorter than what the design delivers, one of the rows has to change: shorter token lifetimes, introspection or deny lists on the services that hold sensitive data, push invalidation of caches, or event-driven deprovisioning. This site's recommendation: write the target down per class of access (privileged, sensitive data, ordinary), compute the design's worst case for each, and test it with a leaver drill like the lab above.
Diagram: When a dependency is down · reads, writes, and what the example does
Rows are the dependency that is down; columns are reads, writes and the example's evidence. Nothing turns no answer into allow. An identity provider outage stops actions that need fresh authentication and lets reads with a still-valid token continue. A PDP outage fails both closed in the example. An audit outage is not simulated in the example; the recommendation is that sensitive writes need a durable record.
- Identity provider down
- continue with a still-valid token: The quote service validates the token itself against its cached key set; the PDP is up.
- stop if fresh authentication is needed: The decision needs step-up within 300 s, which cannot happen during the outage.
- UW-A23: Case UW-A23, idp_down rows.
- PDP down
- fail closed (or a recorded cache): The example returns 503. A bounded, recorded cache of earlier decisions is a design option; allow by default is not.
- fail closed: The enforcement point never allows without an answer.
- UW-A23: Case UW-A23, pdp_down rows.
- Audit service down
- may continue, gap recorded: Reads continue with buffered or reduced logging; the gap is recorded on recovery.
- refuse without a durable record: The audit record is part of the transaction for sensitive writes; a protected local buffer can bridge a short outage.
- not simulated: The example has no audit-outage switch; this row is a recommendation, not a run.
The last diagram puts the fallback choices from the previous sections on one page.
Architecture choice: how much operational machinery#
| Situation | Simplest credible choice | Grow when | Alternatives |
|---|---|---|---|
| One API, one identity provider | Short-lived JWT access tokens; rotation by publish, wait, sign, wait, retire; fail closed on the policy decision point | logout or removal must take effect faster than the token lifetime | opaque tokens with introspection |
| Several services, sensitive data | Introspection or a deny list on the services that hold sensitive data; versioned or pushed cache invalidation; a written access-removal target | many relying parties need to hear about revocations | short tokens everywhere, at the cost of more refreshes |
| Many relying parties, many products | Event-driven signals: SCIM events, Shared Signals (CAEP), back-channel logout, where products support them | events are missed or delayed | scheduled reconciliation as a backstop |
| Regulated environment | All of the above plus rehearsed drills: key-compromise rotation, leaver within target, identity provider and audit outages |
In a coding agent. A stored cloud key in a CI system is a rotation job that someone must remember. CI federation to the cloud through short-lived OIDC tokens removes the job: there is no long-lived key to rotate or revoke, and revoking access means changing the trust policy for one repository and branch. A coding agent's sandbox holds no production credential, so an incident in an agent run is contained by ending the run, not by rotating production keys. Depth on Coding agents.
What this is not. This page does not teach how a JWT is validated or why a self-contained token survives revocation (Tokens), how certificates chain and renew (TLS, PKI and certificates), or how to recover a whole system after a failure (reliable.sammuthu.com). It is about the timing and the failure behaviour of the access controls themselves.
Security notes#
- Rotation is rehearsed on a schedule, so the emergency rotation is a familiar procedure.
- Signing-key compromise: retire the old key at once, accepting that every token it signed fails.
- Every dependency of a request has a written fallback, and no fallback allows by default.
- Writes and privileged actions require a durable audit record; an audit outage raises an alert.
- Cache keys include everything the decision depended on, including the resource.
- The access-removal target is written down and tested, not assumed.
Monday checklist#
- Find the longest access-token lifetime and the longest decision-cache lifetime in your system; add them. Is that within your access-removal target?
- Run a leaver drill: deactivate a test user and measure when the last request succeeds.
- Write down your signing-key rotation steps with the two waits, and when they last ran.
- Add a negative test: the policy decision point unreachable gives 503 for writes, never 200.
- Find what happens to a sensitive write when the audit service is down. If the answer is "it succeeds silently", fix that first.
- List which relying parties receive revocation signals (introspection, deny list, back-channel logout, Shared Signals) and which hear nothing.
Exercises#
Exercise 1. An identity provider starts signing with a new key the same minute it publishes it. Some resource servers cache the key set for an hour. What fails, and for how long?
AnswerNew tokens fail at kid for up to an hour
Resource servers whose cached key set lacks the new key reject every new token at kid until they refresh, up to an hour. Publish first and wait at least the longest key-set cache time before signing. A verifier that refetches the key set on an unknown kid shortens the gap but should rate-limit those refetches.
Exercise 2. The policy decision point is slow during a deploy, and a developer proposes "if the PDP times out, allow reads". What would you agree to instead?
AnswerRecorded, bounded cache; never allow by default
Serve reads from earlier cached decisions for the same subject, action and resource, within their normal lifetime, marked in the audit trail as served from cache. A request with no cached decision fails closed (503). Writes always fail closed.
Exercise 3. The access-removal target for underwriters is five minutes. Tokens live 15 minutes, the decision cache lives 5 minutes. Name two changes that meet the target.
AnswerShorter tokens or introspection, plus a shorter or pushed cache
Either shorten access tokens to a few minutes or introspect (or check a deny list) on the services that hold underwriting data; and shorten the decision cache or invalidate it by push or version when the directory changes. The cache window alone already uses the whole target, so it must shrink too.
Advanced perspective#
Signing-key compromise: the rotation you do differentlySkip the second wait on purpose, then deal with the fallout
A planned rotation waits for in-flight tokens to expire before retiring the old key. After a suspected key compromise you do the opposite: anyone holding the key can mint valid tokens, so every token signed with it must stop working now. Publish the new key, start signing with it, and retire the old key in the same change. Every legitimate session that holds an old-key token fails at kid and must re-authenticate; plan for that load. Then answer the question the rotation cannot: which tokens did the attacker mint? Tokens carry jti and the audit trail records issuance, so a token that was never issued but was accepted is the signature of a forged one. This is why issuance events belong in the audit log (see Audit).
Push beats poll, until the push is lostEvent-driven revocation needs a reconciliation backstop
Shared Signals, CAEP, SCIM events (RFC 9967) and back-channel logout all move revocation from "the next time someone asks" to "now". They share one failure: a receiver that missed an event does not know it missed it. Two practices close the gap. Keep a periodic reconciliation that compares the receiver's state with the source (active users, live sessions) and fixes differences. And keep the time-based backstops (token lifetimes, cache lifetimes) short enough that a lost event still ends within the access-removal target. Events make the common case fast; lifetimes make the worst case bounded. Standard (RFC 9967, May 2026): SCIM security events let receivers react to changes such as prov:deactivate without polling; whether a given product implements it is a question for that product's documentation.
Refresh tokens: the long tail of every sessionRotation with reuse detection turns theft into a signal
An access token lives minutes; the refresh token behind it can live days. Revoking access therefore means revoking the refresh token at the authorization server. Standard (RFC 7009): servers MUST support revoking refresh tokens. Standard (RFC 9700, read 2026-11-18): refresh tokens for public clients "MUST be sender-constrained or use refresh token rotation"; with rotation the server issues a new refresh token on every refresh and "MUST utilize one of these methods to detect refresh token replay". Reuse detection is an operational gift: when an old refresh token comes back, either the client or a thief has a copy, and the server can revoke the whole grant. In the broker portal the BFF holds the refresh token server-side (see Sessions), which keeps it out of the browser and makes logout a server operation.
Drills: the only evidence that a fallback worksRehearse the four bad days
A fallback that has never run is a hypothesis. Four drills cover most of this page: a planned key rotation (expect zero failed requests), an emergency key retirement (expect every old-key token to fail and sessions to recover through re-authentication), a leaver within the access-removal target (measure the last successful request), and an outage of each dependency in turn: identity provider, policy decision point, audit service (expect the written fallback, and no allow by default). The example's tests are a small version of this: UW-A17 for the leaver, UW-A23 for the outages, the incident timeline for rotation and retirement. Running them in a test environment proves the logic; running them against production-like infrastructure proves the timing. Neither proves the other.
In one breath#
Every credential and every cached decision ends, on schedule or in an emergency, and operations is deciding both endings in advance. Rotation is an overlap with two waits, publish then sign, then retire; skipping the second wait is right only for a compromised key. Revocation is not one operation: each mechanism reaches only the components that ask, so the access-removal target is the sum of the slowest paths, and it can be computed. When a dependency is down, writes and privileged actions fail closed, reads may degrade in a recorded, bounded way, and nothing ever turns "no answer" into "allow".
The running example now has a full lifecycle: people join and leave, administrators elevate for minutes, keys rotate, revoked tokens are refused and outages fail closed, all with evidence. What remains is to name the standards, libraries, products and engines behind each mechanism precisely, with what each one does and what stays in your code. Standards, frameworks and products gives each one its own entry.