At 10:00Z the underwriting agent stops working. Every call it makes to the quote service fails before a single byte of the request is read. The run run-7f3a for submission SUB-1004 stalls at its first tool call, and the underwriter u-uw-w1 sees "assessment unavailable" in the cockpit.
The on-call engineer finds the cause in a minute: the agent's client certificate, agent-2026a, was valid until 10:00Z. Nobody renewed it. A teammate posts the fastest fix in the incident channel: "Turn off certificate verification on the quote service until we sort it out."
Hold that suggestion. While it verified, the quote service knew one thing for certain about every caller: it held a private key a trusted authority had tied to the name spiffe://insurer.example/ns/uw/sa/underwriting-agent. With verification off, any program that can reach the port becomes "the agent". That is this site's own case UW-A20 in its first form (an expired certificate fails the validity check), and the fix that looks fastest would reopen UW-A04, the forged caller from Primitives.
This page is about what a certificate actually proves, how a verifier checks it, why it expires, and how to replace it without an outage and without lowering the bar.
What a certificate says, and what it does not#
A certificate is a signed statement from one party about another: "the holder of this public key is called this name, from this time until that time." The party that signs it is a certificate authority (CA). The party it describes holds the matching private key and never sends it anywhere.
A tiny example. The demo workload CA signs: the key k-agent-2026a belongs to spiffe://insurer.example/ns/uw/sa/underwriting-agent, from 2026-11-17 10:00Z to 2026-11-18 10:00Z. The quote service trusts the root that signed that CA, so a caller who shows the statement and proves it holds the key is known to be that workload.
That is the whole claim. The certificate does not say what the agent may do. It does not say the agent may record a decision, or read submission SUB-1007, or act for any particular underwriter. A certificate names a key holder, not a business permission. Every later section keeps those two questions apart.
Plain English. A certificate is a name tag that a trusted office signed, plus a secret only the wearer can use to prove it is theirs. The name tag answers "who is this?". It never answers "may they do this?".
Before AI. Java teams know this as the keystore and the truststore. The keystore holds the service's own private key and certificate; the truststore holds the CA certificates it accepts from others. A servlet container configured to require client certificates asks every caller for one. None of that changes the controller code, which still has to decide what the caller may do.
The smallest primitive: a real chain, made and verified#
The example module mtls.py uses synthetic certificate records, small Python dicts, so the checks can be read line by line. They are not X.509 and they are not TLS. To show the real objects once, the commands below build a throwaway three-level chain with OpenSSL on the author's machine, verify it four ways, and delete every key. A successful run proves that the chain verifies against the right root and fails against a wrong root and after expiry. It does not prove anything about a production CA, which needs protected keys, issuance policy and audit. Use a real CA for real workloads.
The script make_pki.sh (run in the session scratch folder on 2026-11-18 with OpenSSL 3.6.3) creates an EC P-256 root, an intermediate signed by the root with CA:TRUE, pathlen:0, and a one-day leaf signed by the intermediate, with an empty subject and one critical URI SAN. The leaf extension file it used:
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=critical,URI:spiffe://insurer.example/ns/uw/sa/underwriting-agent
The output, pasted as it ran:
== leaf: subject, issuer, SAN, EKU
subject=
issuer=CN=Demo Workload CA (insurer.example)
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name: critical
URI:spiffe://insurer.example/ns/uw/sa/underwriting-agent
== 1. verify against the trusted root, intermediate supplied
leaf.pem: OK
== 2. verify against an unrelated root
CN=Demo Workload CA (insurer.example)
error 20 at 1 depth lookup: unable to get local issuer certificate
error leaf.pem: verification failed
== 3. verify at a time two days from now (after the leaf expires)
error 10 at 0 depth lookup: certificate has expired
error leaf.pem: verification failed
== 4. verify for the purpose sslclient (client authentication)
leaf.pem: OK
== keys and certificates deleted
Each line of that output is one idea of this page.
The chain is the list leaf, intermediate, root, where each certificate is signed by the next. The root is self-signed and is trusted because someone put it in the verifier's trust store on purpose, not because of anything inside it. The intermediate does the daily signing, so the root's key can stay offline. The leaf describes the workload. Run 1 passed because the chain led to a root the verifier trusted; run 2 failed at depth 1 because the verifier could not find a trusted issuer for the intermediate.
The subject alternative name (SAN) is where the name lives. Standard (RFC 5280, read 2026-11-18): "all parts of the subject alternative name MUST be verified by the CA", and if the identity is only in the SAN, "the subject name MUST be an empty sequence and the subjectAltName extension MUST be critical". That is why the leaf above prints subject= with nothing after it.
The validity window is checked against the verifier's clock. Run 3 asked OpenSSL to pretend it was two days later, and the one-day leaf failed with error 10. That is the 10:00Z incident in one line.
The extended key usage (EKU) limits what the key may be used for. Standard (RFC 5280): id-kp-clientAuth means "TLS WWW client authentication", and when the extension is present "the certificate MUST only be used for one of the purposes indicated". Run 4 passed the client-authentication purpose because the leaf lists both server and client authentication, as Standard (X.509-SVID, read 2026-11-18) asks of SPIFFE leaf certificates: when the EKU is included, "id-kp-serverAuth and id-kp-clientAuth MUST be set".
How a verifier walks the chain#
The verifier is whichever side receives a certificate: the browser checking a server, or the quote service checking the agent. Standard (RFC 5280, section 6.1): path validation checks, for each certificate, that "The signature on the certificate can be verified", "The certificate validity period includes the current time", and "At the current time, the certificate is not revoked." CA certificates carry cA TRUE; a leaf does not.
Chain and validity tell the verifier that some trusted CA vouched for this key now. They do not yet say that it is the key it wanted. A valid certificate for the quote service is still the wrong certificate when the verifier expected the agent. So the verifier compares the name in the certificate with the name it expected. On the public web that comparison is hostname validation: the browser expected api.uw.insurer.example because that is what it connected to.
Standard (RFC 9525, "Service Identity in TLS", November 2023, read 2026-11-18): it "obsoletes RFC 6125". Names are matched against SAN entries of type dNSName (a DNS-ID) or uniformResourceIdentifier (a URI-ID). "The Common Name RDN MUST NOT be used to identify a service"; the advice is "Do not include or check strings that look like domain names in the subject's Common Name." Wildcards are constrained: "There is only one wildcard character", and it "appears only as the complete content of the left-most label."
Between workloads the expected name is usually a URI SAN, a SPIFFE ID. The quote service expects the agent's id, or one of a set of ids, rather than a hostname. The page Workload identity explains those ids.
Common confusion. "The certificate is valid" is four facts, not one: trusted chain, current validity, expected name, proven key. Checking only the first two accepts a valid certificate for the wrong service.
Key ownership: why a copied certificate is useless#
A certificate is public; every server sends it to every client. If showing it were enough, anyone could replay one. What makes the handshake an authentication is the fourth check: the peer signs something fresh with the private key.
Standard (RFC 9846, TLS 1.3, July 2026, read 2026-11-18): the CertificateVerify message is "A signature over the entire handshake using the private key corresponding to the public key in the Certificate message." The handshake is new each time, so an old signature does not fit it. An attacker who copied agent-2026a from a log can present it but cannot produce that signature.
The example models this as the key check in verify_peer. On versions: RFC 9846 is "a minor update to TLS 1.3 that retains the same version number"; it obsoletes RFC 8446 and RFC 5246 (TLS 1.2), so cite RFC 9846 for new work.
Server TLS and mutual TLS#
Most TLS on the internet authenticates one side. The broker's browser checks the portal's certificate; the portal learns nothing about the browser from TLS, and authenticates the broker with a session instead. Standard (RFC 9846): the server sends a CertificateRequest "if certificate-based client authentication is desired", and a client omits its Certificate "if the server did not send CertificateRequest". Client authentication is optional in the protocol.
Mutual TLS (mTLS) is TLS where the server also asks for, and verifies, a client certificate. Both sides then know which key holder is on the other end. It suits service-to-service calls, where there is no person to type a password and no browser to hold a cookie.
| Question | Server TLS | Mutual TLS |
|---|---|---|
| Who presents a certificate | the server | the server and the client |
| Who verifies a name | the client checks the server's DNS-ID | both; the server also checks the client's SAN, often a SPIFFE ID |
| What the server learns about the caller | nothing; it authenticates the caller another way (session, token) | which workload holds the client key |
| What it never settles | whether the caller may do the action | whether the caller may do the action |
The last row is identical on purpose. Neither form of TLS answers the authorization question.
Diagram: Mutual TLS at the quote service · the handshake names the caller, two later checks decide
Time runs down the page. The agent opens a connection; the quote service presents its certificate and asks for the client's with CertificateRequest. The agent sends its chain and a CertificateVerify signature made with its private key. The quote service verifies chain, validity, SAN and key, and ends the handshake with a verified SPIFFE ID. Only then does the request arrive. The quote service asks its allowlist whether this workload may call this operation, and the policy decision point whether this subject may act on this record. For decision.record the allowlist already says no to the agent.
- Underwriting agent (application) to Quote service: ClientHello. The agent starts the TLS 1.3 handshake.
- Quote service to Underwriting agent (application): ServerHello + server certificate + CertificateRequest. The server authenticates itself and asks the client for a certificate. Without CertificateRequest, the client sends none.
- Underwriting agent (application) to Quote service: client certificate chain + CertificateVerify. The chain names the agent; the signature over the whole handshake proves it holds the key.
- Quote service: verify chain, validity, SAN, key. Any failure ends the connection before a request is read.
- Underwriting agent (application) to Quote service: request: decision.record SUB-1004. Arrives over the authenticated connection, from a verified peer.
- Quote service to Caller allowlist: may this SPIFFE ID call decision.record?. A coarse check on the operation, using the verified peer id, never a header.
- Caller allowlist to Quote service: no. Handshake succeeded, operation refused.
- Quote service to Underwriting agent (application): 403. The model recommends; a person decides. Only the cockpit BFF carries decisions.
- Quote service to Policy decision point: for allowed callers: may this subject act on this record?. For the cockpit BFF the allowlist says yes, and the policy decision point still checks the underwriter's assignment and step-up.
Read the diagram from top to bottom. The first four messages are the handshake: the quote service ends it knowing a verified SPIFFE ID, or it closes the connection. Only then does a request arrive, and the quote service asks two more questions the handshake cannot answer: is this workload on the allowlist for this operation, and does the policy allow this subject on this record?
In our insurance example: the agent's certificate at the quote service#
The quote service checks the agent's certificate with mtls.py verify_peer, in the order chain, validity, SAN, key. The snippet below runs it for six situations at the simulated clock. It demonstrates which check catches which fault. A successful run proves the order and the separation of checks in this teaching model; it does not exercise any real TLS library.
import mtls
from clock import START_EPOCH_S as NOW
H = 3600
def show(label, cert, san, t, key=None, trust=mtls.TRUST_BUNDLE):
v = mtls.verify_peer(mtls.chain_for(cert), trust, san, t, presented_key_id=key)
failed = [c["name"] for c in v["checks"] if not c["passed"]]
print(f"{label:<13} handshake_ok={v['handshake_ok']!s:<5} failed={failed}")
for c in v["checks"]:
if not c["passed"]:
print(" ", c["detail"])
return v
print("chain presented by agent-2026a:")
for c in mtls.chain_for("agent-2026a"):
print(f" {c['id']:<12} subject={c['subject']:<40} issuer={c['issuer']} is_ca={c['is_ca']}")
print()
show("valid", "agent-2026a", mtls.AGENT_ID, NOW)
show("stolen-cert", "agent-2026a", mtls.AGENT_ID, NOW, key="k-attacker")
show("wrong-san", "quote-server", mtls.AGENT_ID, NOW)
show("untrusted-ca", "rogue-agent", mtls.AGENT_ID, NOW)
show("expired", "agent-2026a", mtls.AGENT_ID, NOW + 2 * H)
show("rotated", "agent-2026b", mtls.AGENT_ID, NOW + 2 * H)
Run in a scratch copy of examples/uw-auth on 2026-11-18 (the agent here is a scripted stand-in, so these results show how the access controls work, not how good any model is):
chain presented by agent-2026a:
agent-2026a subject=CN=underwriting-agent issuer=CN=Demo Workload CA (insurer.example) is_ca=False
ca-workload subject=CN=Demo Workload CA (insurer.example) issuer=CN=Demo Root CA (insurer.example) is_ca=True
ca-root subject=CN=Demo Root CA (insurer.example) issuer=CN=Demo Root CA (insurer.example) is_ca=True
valid handshake_ok=True failed=[]
stolen-cert handshake_ok=False failed=['key']
peer could not prove possession of the private key
wrong-san handshake_ok=False failed=['san']
URI SAN spiffe://insurer.example/ns/uw/sa/quote does not match spiffe://insurer.example/ns/uw/sa/underwriting-agent
untrusted-ca handshake_ok=False failed=['chain']
issuer CN=Unknown CA is not in the trust bundle
expired handshake_ok=False failed=['validity']
certificate agent-2026a is outside its validity window (not_after 1794996000, now 1794999600)
rotated handshake_ok=True failed=[]
Each failure names one check. stolen-cert has a genuine certificate and fails only at key possession. wrong-san is the quote service's own valid certificate presented where the agent's was expected. untrusted-ca carries the right SPIFFE ID, signed by a CA nobody trusts: a SAN means nothing without the chain. expired is the 10:00Z incident; rotated, the agent's next certificate at the same moment, is the fix.
Diagram: A certificate chain and the four checks · three look at documents, one looks at the live connection
Read left to right. The root is trusted because the quote service put it in its trust bundle on purpose. The root signed the workload CA, and the workload CA signed the agent's leaf certificate, whose SAN holds the SPIFFE ID. The verifier then runs four checks in order: the chain ends at a trusted root, every certificate is inside its validity window, the SAN is the name it expected, and the peer proved it holds the private key in this handshake. Only the last check involves the live peer. Passing all four yields a verified name and nothing else: no permission.
- The chain the agent presents
- root CA: Trusted because the verifier chose to put it in its trust bundle, not because of anything it says about itself.
- intermediate CA: Signed by the root, signs workload certificates. Its compromise is serious; the root's would be worse.
- leaf certificate: Binds the agent's public key to its name for 24 hours in the example. Not a CA: it cannot sign others.
- What the quote service checks
- chain: Each certificate is signed by the next, and the last one is in the trust bundle.
- validity: Every certificate in the chain is inside its window now.
- SAN: The URI SAN is the SPIFFE ID the service expected (a DNS name for a public server).
- key possession: The peer signs the handshake with the private key (CertificateVerify). A copied certificate cannot do this.
- What it yields
- a verified name: The service knows which key holder is calling. Whether it may perform the operation is the allowlist's and the policy's question.
The diagram puts the chain and the four checks side by side. Notice that three of the four checks look at the certificate and one looks at the live connection. A verifier that skips the live check is verifying a document, not a peer.
The mTLS lab#
Change the certificate, clock, pinned SAN, key proof and operation, and watch the handshake checks and, separately, the operation decision. The table holds the same recorded results.
| Scenario | Certificate | Clock | Operation | Handshake | Operation decision |
|---|---|---|---|---|---|
| valid-read | agent-2026a | 09:00Z | submission.read | established | allow 200 |
| allowlist | agent-2026a | 09:00Z | decision.record | established | deny 403: not on the allowlist for decision.record |
| cockpit-decide | cockpit-2026 | 09:00Z | decision.record | established | allow 200 |
| expired | agent-2026a | 11:00Z | submission.read | refused at validity (expired 10:00Z) | deny 401 |
| rotated | agent-2026b | 11:00Z | submission.read | established | allow 200 |
| overlap | agent-2026b | 09:00Z | submission.read | established (both agent certificates valid 08:00Z to 10:00Z) | allow 200 |
| wrong-san | quote-server | 09:00Z | submission.read | refused at SAN | deny 401 |
| untrusted-ca | rogue-agent | 09:00Z | submission.read | refused at chain (CN=Unknown CA not trusted) | deny 401 |
The allowlist of the example: submission.read for the underwriting agent and the cockpit BFF; decision.record for the cockpit BFF only. The demo certificates are synthetic and the clock is simulated.
A certificate names a key holder, not a permission#
Look at the allowlist row. The handshake is fine. The quote service knows, with cryptographic confidence, that the caller is the underwriting agent. And it still refuses decision.record, because the agent's identity is not on the list of callers allowed to perform that operation. The model recommends; a person decides, so the only workload allowed to carry a decision is the cockpit BFF, where a stepped-up underwriter acts.
The snippet runs authorize_peer, the receiving service's decision on a verified peer id, for three calls. It demonstrates that the decision is a separate function that takes the verified id as input. It does not show the record-level policy, which the policy decision point adds afterwards.
import mtls
from clock import START_EPOCH_S as NOW
for cert, op in [("agent-2026a", "submission.read"), ("agent-2026a", "decision.record"), ("cockpit-2026", "decision.record")]:
v = mtls.verify_peer(mtls.chain_for(cert), mtls.TRUST_BUNDLE, None, NOW)
a = mtls.authorize_peer(v["peer_id"], op, mtls.ALLOWLIST)
print(f"{cert:<13} {op:<16} handshake_ok={v['handshake_ok']} -> {a['status']} {a['reason']}")
agent-2026a submission.read handshake_ok=True -> 200 spiffe://insurer.example/ns/uw/sa/underwriting-agent is on the allowlist for submission.read
agent-2026a decision.record handshake_ok=True -> 403 handshake succeeded, but spiffe://insurer.example/ns/uw/sa/underwriting-agent is not on the allowlist for decision.record
cockpit-2026 decision.record handshake_ok=True -> 200 spiffe://insurer.example/ns/uw/sa/cockpit-bff is on the allowlist for decision.record
Even the third line is not a permission to decide; it says the cockpit BFF may carry decisions at all. Whether this underwriter may decide on this submission is the policy decision point's job, requiring the assigned underwriter, the uw.decide entitlement, and a phishing-resistant login within 300 seconds (rule R-AZ-09). Three layers, three questions: which key holder, which caller may call this operation, which subject may act on this record.
What this is not. The allowlist and SPIFFE IDs belong to Workload identity; the record-level decision belongs to Authorization and Enforcement. This page stops at "the handshake produced a verified name".
Revocation: the check that is often not there#
Certificates can be withdrawn before they expire, for example when a key leaks. Certificate revocation is that withdrawal, and the hard part is telling every verifier in time.
Standard (RFC 5280): a certificate revocation list (CRL) is a periodically issued, signed list of revoked serial numbers; nextUpdate "indicates the date by which the next CRL will be issued", so a verifier caching a CRL can accept a revoked certificate until then. The Online Certificate Status Protocol (OCSP) asks a responder about one certificate at a time.
The public web has moved away from OCSP. Product (Let's Encrypt, as documented on 2026-11-18): in a post dated 6 August 2025, "Today we turned off our Online Certificate Status Protocol (OCSP) service"; it now publishes revocation "exclusively via Certificate Revocation Lists (CRLs)", citing OCSP's "considerable risk to privacy". The other answer is short lifetimes: its shortlived profile issues 160-hour certificates, which "greatly reduce the potential compromise window". Guideline (CA/Browser Forum Baseline Requirements 2.3.1, read 2026-11-18): maximum public certificate validity falls to 200 days from 2026-03-15, 100 days from 2027-03-15 and 47 days from 2029-03-15.
Inside one organization the example's agent certificates live 24 hours, so a leaked key is useful for at most a day, and the CA can stop renewing a compromised workload. This site's recommendation: for workload mTLS, prefer short lifetimes and automated renewal over revocation checks you cannot guarantee; keep a CRL for the intermediate and for long-lived certificates you cannot avoid.
Production pitfall. Whether a TLS library checks revocation at all, and whether it accepts the peer when the CRL cannot be fetched, is a property of that library and its configuration; the sources read for this page do not settle it for any one library. Test what yours does before you count revocation as a control. Guideline (OWASP Microservices Security Cheat Sheet, read 2026-11-18): "The main challenges of using mTLS are key provisioning and trust bootstrap, certificate revocation, and key rotation."
Renewal, rotation and expiry recovery#
Back to 10:00Z. The certificate did not fail; the process around it did. A certificate with a 24-hour life needs a renewal every day, and a renewal process needs three properties.
Overlap. The next certificate must be valid before the current one expires, so the workload can switch while both work. In the example, agent-2026b became valid at 08:00Z and agent-2026a expired at 10:00Z: a two-hour window in which either one passes. Certificate rotation is that switch: new key, new certificate, then retire the old one, with no moment where the workload has nothing valid.
Renew early, from the workload's side. The sources read here fix no universal renewal point, so the fraction of lifetime is a team choice. Trust bundles rotate too: verifiers accept a new intermediate before the CA signs with it.
Alert before expiry. Monitor the remaining lifetime of every certificate a service presents or accepts.
Diagram: Rotation with overlap · the next certificate is valid before the current one expires
Read left to right along the simulated morning. At 08:00Z the agent's next certificate, agent-2026b, becomes valid while agent-2026a still has two hours left. Between 08:00Z and 10:00Z either certificate passes, so the agent can switch at any moment. At 10:00Z agent-2026a expires; a workload that has not switched fails its handshake. The recovery is a new certificate from the normal CA, never turning verification off.
- Before 08:00Z
- agent-2026a in use: The current certificate, issued with a 24-hour life.
- renewal job runs: The workload or an agent beside it asks the CA for the next certificate well before expiry, with a new key.
- 08:00Z to 10:00Z: overlap
- both valid: Either certificate passes the validity check, so the switch causes no failed call.
- agent switches: New connections use agent-2026b; old connections drain.
- From 10:00Z
- agent-2026a expired: Any workload still presenting it fails the handshake: 401 before a request is read.
- recovery: issue, do not disable: Issue an emergency certificate from the normal CA with an approval and an audit record. Turning verification off would admit every caller.
The diagram shows the two agent certificates on one timeline. The whole point is the shaded overlap: rotation without downtime needs a moment where both are valid, and the renewal job must finish inside it.
When expiry happens anyway, the recovery is the same in every design: issue a new certificate, do not lower verification. Turning verification off, adding a "temporary" trust for a self-signed certificate, or rolling the verifier's clock back each removes authentication for every caller, not only for the one whose certificate expired. This site's recommendation: keep a runbook that issues an emergency certificate for one workload from the normal CA, with an approval and an audit record, and test it. The teammate's suggestion in the opening is the mistake this runbook exists to prevent.
Public CA or internal CA: client certificates after 8 July 2026#
Who should sign workload certificates? For servers that browsers reach, a public CA is the default, because every browser already trusts it. For mTLS between workloads, the answer changed this year.
Product (Let's Encrypt, as documented on 2026-11-18): Let's Encrypt no longer issues certificates with the TLS Client Authentication EKU. The default classic profile dropped it on 11 February 2026, and the temporary tlsclient profile was retired on 8 July 2026: "no further certificates with the Client Authentication EKU will be issued." The reason it gives is the Chrome root program's "June 2026 deadline to split TLS Client and Server Authentication into separate PKIs." Its own advice: "Many uses of client authentication are better served by a private certificate authority."
The implication for mTLS is direct: such a certificate fails a strict client-authentication check (run 4 of the OpenSSL script). And a public CA vouches for domain control, which says little about which internal workload is calling. An internal CA under your control can issue SPIFFE-style names, short lifetimes and both purposes.
| Situation | Simplest credible choice | Grow when |
|---|---|---|
| A browser reaches your server | public CA certificate with automated renewal | more names and hosts: automate per host, never share keys |
| A handful of internal services call each other | internal CA (root offline, one intermediate) issuing short-lived client and server certificates; allowlist at each receiver | many teams or clusters: a workload identity system such as SPIRE |
| Partners call your API with certificates | a partner CA you operate, or certificates pinned per partner; RFC 8705 tls_client_auth at the token endpoint | many partners: federation of trust bundles, per-partner allowlists |
| A scheduled job, no person | the job's own workload certificate | never a person's certificate for a job |
In a coding agent. A coding agent's sandbox needs no certificate for production services at all. CI that deploys reaches the cloud through OIDC federation, not a stored client key, and a person with merge rights approves the pull request that changes certificate configuration. Coding agents owns that design.
Security notes#
- Verify all four things: chain to a root you chose, validity window, the expected name in the SAN, and key possession in the handshake. Never accept "any certificate from our CA" where a specific workload is expected.
- Never disable verification to recover from expiry. Issue a new certificate through the normal CA, with an approval and an audit record.
- Keep the root key offline and sign with an intermediate. Constrain the intermediate (
pathlen:0) so it cannot mint further CAs. - Treat revocation as best effort unless you have tested that your verifiers check it and fail closed. Short lifetimes are the stronger control for workloads.
- Never put a private key in a repository, a container image, a prompt or a log. Generate it where it is used and let it never leave.
- A successful handshake grants nothing. Every operation still needs the receiving service's allowlist and the record-level policy.
Monday checklist#
- List every certificate each service presents and accepts, with its expiry. Add an alert that fires well before the earliest one.
- For one mTLS client and one server, confirm in code or configuration which name is checked (DNS-ID or URI SAN) and that the check cannot be skipped.
- Find any
verify=False,InsecureSkipVerify, trust-allTrustManageror-kin code and scripts, and remove it or document why it is safe. - Check that the next certificate is issued before the current one expires, and measure the overlap on a real rotation.
- Find out whether your TLS stack checks revocation, and whether it fails open when the CRL is unreachable.
- If you used public certificates as mTLS client certificates, plan the move to an internal CA: since 8 July 2026 Let's Encrypt issues none with the client-authentication EKU.
Exercises#
1. The quote service receives a client certificate that chains to the trusted root, is inside its validity window, and carries the SAN spiffe://insurer.example/ns/uw/sa/quote. The quote service expected the underwriting agent. What happens, and which check decides?
AnswerThe SAN check refuses it
The handshake is refused at the SAN check (the wrong-san scenario, 401). A valid certificate for a different workload is still the wrong certificate.
2. A teammate proposes renewing certificates at exactly their expiry time, "so we never waste validity". What breaks?
AnswerThere is no overlap
Any delay in renewal or in distributing the new certificate leaves the workload with nothing valid: the 10:00Z incident. Rotation without downtime needs an overlap window, like 08:00Z to 10:00Z in the example.
3. The cockpit BFF completes mTLS with the quote service and is on the allowlist for decision.record. Can the peer underwriter u-uw-w2 now record a decision on SUB-1004 through the cockpit?
AnswerNo: the record decides
No. mTLS and the allowlist say the cockpit BFF may carry decisions. The policy decision point still checks assignment on the record: u-uw-w2 is not assigned to SUB-1004, so the answer is 403 (UW-A09 under authz-2026.11.1).
Advanced perspective#
Where TLS ends: proxies, load balancers and forwarded certificatesmTLS proves the last hop only
mTLS authenticates the two ends of one TCP connection. If a load balancer terminates TLS and opens a new connection to the quote service, the quote service authenticates the load balancer, not the agent. A common pattern forwards the client certificate in an HTTP header from the proxy to the service. That header is exactly as forgeable as x-app-id unless two conditions hold: the service accepts it only from the proxy, verified by the proxy's own mTLS identity, and the proxy strips any incoming header of the same name. Otherwise any caller that reaches the service directly can type a certificate into a header. Prefer end-to-end mTLS to the workload, or a sidecar on the same host whose connection to the application cannot be reached from the network. Record which design you run, because the audit question "who called?" has a different answer in each.
Certificate-bound tokens: one step past the handshakeRFC 8705 ties a token to the client certificate
mTLS proves the caller at the connection; a bearer token proves nothing about who presents it. RFC 8705 joins the two: the authorization server puts a hash of the client's certificate into the token, and the resource server checks that the certificate on the connection matches. Standard (RFC 8705, read 2026-11-18): the hash "SHOULD be represented using the x5t#S256 confirmation method member" inside cnf; the resource server "MUST verify that the certificate matches the certificate associated with the access token", and a mismatch is 401 invalid_token. The same RFC defines tls_client_auth and self_signed_tls_client_auth, two ways a client authenticates to the token endpoint with a certificate. A token copied out of a log is then useless without the key. Plan for rotation: tokens bound to agent-2026a stop working when the agent switches to agent-2026b, so token lifetime must fit inside the overlap. Tokens compares this with DPoP; Workload identity shows where it fits on a call between services.
Limiting what a stolen CA key can doOffline roots, constrained intermediates, separate trust domains
Every verifier trusts whatever the CA signs, so a stolen intermediate key can mint a certificate for any workload name. Four controls shrink that. Keep the root offline, used only to sign intermediates. Give each intermediate pathlen:0 so it cannot create sub-CAs, and where the library supports it, name constraints so it can sign only names under one trust domain (RFC 5280 notes that name constraints "apply only when the specified name form is present", so test with URI SANs, not only DNS). Use separate roots for production and staging; the SPIFFE concepts page advises separate trust domains for environments such as production and staging. And make intermediate rotation a rehearsed operation: publish the new intermediate in every trust bundle first, switch signing, then withdraw the old one. The quote service's allowlist is the last line here: a forged certificate for the agent still cannot record a decision, because the agent is not on that list.
Revocation designs comparedCRL, OCSP, short life, and stopping renewal
A CRL is simple and private (the verifier downloads one list, revealing nothing about which peer it is checking), but it is stale up to nextUpdate and grows with every revocation. OCSP is fresher per certificate but adds a live dependency at every handshake and tells the responder who connects to whom, which is the privacy cost Let's Encrypt gave for ending its service. Short-lived certificates turn revocation into "stop renewing": the window is the remaining lifetime, known in advance. For workload identity inside one organization, the last option is usually the right default; the residual risk is the remaining lifetime, so choose it with the access-removal time your Operations page promises in mind. Whatever you choose, decide explicitly whether a verifier that cannot reach revocation data fails closed or open, and write that into the service's runbook.
Clocks are part of the trust modelSkew, simulated time and expiry testing
Every validity check compares against the verifier's clock. A container with a drifting clock rejects fresh certificates as "not yet valid" or accepts expired ones. Short lifetimes make this sharper: a one-day certificate leaves little room for minutes of skew at either end. Run time synchronization on every host, allow a small, documented tolerance for not_before, and never allow tolerance on not_after large enough to matter. In tests, inject the clock instead of waiting: the example's clock is simulated from 2026-11-18T09:00:00Z, which is why expired and rotated are deterministic, and the OpenSSL run used -attime for the same reason. Add a test for each boundary: one second before expiry passes, one second after fails.
In one breath#
A certificate is a CA's signed statement that a public key belongs to a name, for a time. A verifier trusts it only after four checks: the chain ends at a root it chose, every certificate is current, the name is the one it expected, and the peer proved it holds the private key in this handshake. mTLS applies those checks to the caller as well as the server, and still decides nothing about what the caller may do. Certificates expire on purpose, so renew early with overlap, and recover from expiry by issuing a new certificate, never by turning verification off.
The quote service now knows, by a verified certificate, that the underwriting agent is calling. It does not yet know how the agent got that certificate without a secret baked into its image, how two agents in one cluster keep distinct names, or how the agent's identity travels together with the underwriter it acts for. Workload identity answers those three questions.