All chapters
  1. Start hereA valid token is not a permission
  2. Foundations · 1Identity and trust primitives: who is calling, and who checks
  3. Foundations · 2People and identity proofing: accounts, passwords and recovery
  4. Foundations · 3Sessions and browser security: cookies, CSRF, XSS and the BFF
  5. Foundations · 4Authenticators and MFA: passwords, passkeys and phishing resistance
  6. Foundations · 5API keys and application credentials: what a key proves
  7. Foundations · 6Tokens: bearer, JWT anatomy, validation and revocation
  8. Foundations · 7OAuth, OpenID Connect and federation: delegation and login
  9. Authorization · 8Authorization from first principles: subject, action, resource, context
  10. Authorization · 9Access control models: RBAC, ABAC, ReBAC and ownership
  11. Authorization · 10The enforcement architecture: PEP, PDP, PIP and PAP
  12. Authorization · 11Policy as code: versions, tests, engines and decision logs
  13. Authorization · 12Tenant and data isolation: one platform, many brokerages
  14. Authorization · 13Identity governance and lifecycle: joiner, mover, leaver
  15. Authorization · 14Privileged access: standing rights, just-in-time elevation and break-glass
  16. Workloads · 15TLS, PKI and certificates: proving a host, and a client
  17. Workloads · 16Workload and service identity: SPIFFE, mTLS and caller allowlists
  18. Workloads · 17API gateways and zero trust: the edge checks, the service decides
  19. Workloads · 18Event-driven identity: Kafka principals, ACLs and replay
  20. Workloads · 19Cloud IAM and secretless deployment: roles, federation and no stored keys
  21. Workloads · 20External SaaS, webhooks and trust: calls that cross company lines
  22. Workloads · 21Private controlled environments: what private networking does and does not secure
  23. AI-native · 22Authenticating to model providers: keys, gateways and cloud IAM
  24. AI-native · 23MCP identity and authorization: discovery is not permission
  25. AI-native · 24RAG and retrieval authorization: only what this caller may read
  26. AI-native · 25Agents as principals and delegates: whose authority is this?
  27. AI-native · 26Agent-to-agent trust: identity, capabilities and forwarded authority
  28. AI-native · 27Coding agents and CI: repository scopes, OIDC and no production keys
  29. Operations · 28Auditing and observability of access: decisions you can prove later
  30. Operations · 29Threat modeling and testing: the negative tests that prove access control
  31. Operations · 30Operational lifecycles: rotation, revocation and outages
  32. Reference · 31Standards, frameworks and products: what each one is, and what it is not
  33. Reference · 32Architecture decision playbooks: choosing an auth strategy
  34. ReferenceGlossary
  35. AboutAbout this site and its sources
Workloads · 15

TLS, PKI and certificates: proving a host, and a client

Root, intermediate and leaf certificates, subject alternative names, hostname checks, server TLS versus mutual TLS, revocation limits, renewal, rotation and expiry recovery. A certificate names a key holder; it does not grant a business action.

You will read
A certificate chain from root to leaf, and the four checks a verifier makes on it: chain, validity, name, key.
You will separate
What a TLS handshake proves (who holds the key) from what the receiving service must still decide (what that holder may do).
You will run
Renewal with overlap, and an expiry recovery that never turns verification off.

At 10:00Z the underwriting agent stops working. Every call it makes to the quote service fails before a single byte of the request is read. The run run-7f3a for submission SUB-1004 stalls at its first tool call, and the underwriter u-uw-w1 sees "assessment unavailable" in the cockpit.

The on-call engineer finds the cause in a minute: the agent's client certificate, agent-2026a, was valid until 10:00Z. Nobody renewed it. A teammate posts the fastest fix in the incident channel: "Turn off certificate verification on the quote service until we sort it out."

Hold that suggestion. While it verified, the quote service knew one thing for certain about every caller: it held a private key a trusted authority had tied to the name spiffe://insurer.example/ns/uw/sa/underwriting-agent. With verification off, any program that can reach the port becomes "the agent". That is this site's own case UW-A20 in its first form (an expired certificate fails the validity check), and the fix that looks fastest would reopen UW-A04, the forged caller from Primitives.

This page is about what a certificate actually proves, how a verifier checks it, why it expires, and how to replace it without an outage and without lowering the bar.

What a certificate says, and what it does not#

A certificate is a signed statement from one party about another: "the holder of this public key is called this name, from this time until that time." The party that signs it is a certificate authority (CA). The party it describes holds the matching private key and never sends it anywhere.

A tiny example. The demo workload CA signs: the key k-agent-2026a belongs to spiffe://insurer.example/ns/uw/sa/underwriting-agent, from 2026-11-17 10:00Z to 2026-11-18 10:00Z. The quote service trusts the root that signed that CA, so a caller who shows the statement and proves it holds the key is known to be that workload.

That is the whole claim. The certificate does not say what the agent may do. It does not say the agent may record a decision, or read submission SUB-1007, or act for any particular underwriter. A certificate names a key holder, not a business permission. Every later section keeps those two questions apart.

Plain English. A certificate is a name tag that a trusted office signed, plus a secret only the wearer can use to prove it is theirs. The name tag answers "who is this?". It never answers "may they do this?".
Before AI. Java teams know this as the keystore and the truststore. The keystore holds the service's own private key and certificate; the truststore holds the CA certificates it accepts from others. A servlet container configured to require client certificates asks every caller for one. None of that changes the controller code, which still has to decide what the caller may do.

The smallest primitive: a real chain, made and verified#

The example module mtls.py uses synthetic certificate records, small Python dicts, so the checks can be read line by line. They are not X.509 and they are not TLS. To show the real objects once, the commands below build a throwaway three-level chain with OpenSSL on the author's machine, verify it four ways, and delete every key. A successful run proves that the chain verifies against the right root and fails against a wrong root and after expiry. It does not prove anything about a production CA, which needs protected keys, issuance policy and audit. Use a real CA for real workloads.

The script make_pki.sh (run in the session scratch folder on 2026-11-18 with OpenSSL 3.6.3) creates an EC P-256 root, an intermediate signed by the root with CA:TRUE, pathlen:0, and a one-day leaf signed by the intermediate, with an empty subject and one critical URI SAN. The leaf extension file it used:

basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=critical,URI:spiffe://insurer.example/ns/uw/sa/underwriting-agent

The output, pasted as it ran:

== leaf: subject, issuer, SAN, EKU
subject=
issuer=CN=Demo Workload CA (insurer.example)
X509v3 Extended Key Usage: 
    TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name: critical
    URI:spiffe://insurer.example/ns/uw/sa/underwriting-agent
== 1. verify against the trusted root, intermediate supplied
leaf.pem: OK
== 2. verify against an unrelated root
CN=Demo Workload CA (insurer.example)
error 20 at 1 depth lookup: unable to get local issuer certificate
error leaf.pem: verification failed
== 3. verify at a time two days from now (after the leaf expires)

error 10 at 0 depth lookup: certificate has expired
error leaf.pem: verification failed
== 4. verify for the purpose sslclient (client authentication)
leaf.pem: OK
== keys and certificates deleted

Each line of that output is one idea of this page.

The chain is the list leaf, intermediate, root, where each certificate is signed by the next. The root is self-signed and is trusted because someone put it in the verifier's trust store on purpose, not because of anything inside it. The intermediate does the daily signing, so the root's key can stay offline. The leaf describes the workload. Run 1 passed because the chain led to a root the verifier trusted; run 2 failed at depth 1 because the verifier could not find a trusted issuer for the intermediate.

The subject alternative name (SAN) is where the name lives. Standard (RFC 5280, read 2026-11-18): "all parts of the subject alternative name MUST be verified by the CA", and if the identity is only in the SAN, "the subject name MUST be an empty sequence and the subjectAltName extension MUST be critical". That is why the leaf above prints subject= with nothing after it.

The validity window is checked against the verifier's clock. Run 3 asked OpenSSL to pretend it was two days later, and the one-day leaf failed with error 10. That is the 10:00Z incident in one line.

The extended key usage (EKU) limits what the key may be used for. Standard (RFC 5280): id-kp-clientAuth means "TLS WWW client authentication", and when the extension is present "the certificate MUST only be used for one of the purposes indicated". Run 4 passed the client-authentication purpose because the leaf lists both server and client authentication, as Standard (X.509-SVID, read 2026-11-18) asks of SPIFFE leaf certificates: when the EKU is included, "id-kp-serverAuth and id-kp-clientAuth MUST be set".

How a verifier walks the chain#

The verifier is whichever side receives a certificate: the browser checking a server, or the quote service checking the agent. Standard (RFC 5280, section 6.1): path validation checks, for each certificate, that "The signature on the certificate can be verified", "The certificate validity period includes the current time", and "At the current time, the certificate is not revoked." CA certificates carry cA TRUE; a leaf does not.

Chain and validity tell the verifier that some trusted CA vouched for this key now. They do not yet say that it is the key it wanted. A valid certificate for the quote service is still the wrong certificate when the verifier expected the agent. So the verifier compares the name in the certificate with the name it expected. On the public web that comparison is hostname validation: the browser expected api.uw.insurer.example because that is what it connected to.

Standard (RFC 9525, "Service Identity in TLS", November 2023, read 2026-11-18): it "obsoletes RFC 6125". Names are matched against SAN entries of type dNSName (a DNS-ID) or uniformResourceIdentifier (a URI-ID). "The Common Name RDN MUST NOT be used to identify a service"; the advice is "Do not include or check strings that look like domain names in the subject's Common Name." Wildcards are constrained: "There is only one wildcard character", and it "appears only as the complete content of the left-most label."

Between workloads the expected name is usually a URI SAN, a SPIFFE ID. The quote service expects the agent's id, or one of a set of ids, rather than a hostname. The page Workload identity explains those ids.

Common confusion. "The certificate is valid" is four facts, not one: trusted chain, current validity, expected name, proven key. Checking only the first two accepts a valid certificate for the wrong service.

Key ownership: why a copied certificate is useless#

A certificate is public; every server sends it to every client. If showing it were enough, anyone could replay one. What makes the handshake an authentication is the fourth check: the peer signs something fresh with the private key.

Standard (RFC 9846, TLS 1.3, July 2026, read 2026-11-18): the CertificateVerify message is "A signature over the entire handshake using the private key corresponding to the public key in the Certificate message." The handshake is new each time, so an old signature does not fit it. An attacker who copied agent-2026a from a log can present it but cannot produce that signature.

The example models this as the key check in verify_peer. On versions: RFC 9846 is "a minor update to TLS 1.3 that retains the same version number"; it obsoletes RFC 8446 and RFC 5246 (TLS 1.2), so cite RFC 9846 for new work.

Server TLS and mutual TLS#

Most TLS on the internet authenticates one side. The broker's browser checks the portal's certificate; the portal learns nothing about the browser from TLS, and authenticates the broker with a session instead. Standard (RFC 9846): the server sends a CertificateRequest "if certificate-based client authentication is desired", and a client omits its Certificate "if the server did not send CertificateRequest". Client authentication is optional in the protocol.

Mutual TLS (mTLS) is TLS where the server also asks for, and verifies, a client certificate. Both sides then know which key holder is on the other end. It suits service-to-service calls, where there is no person to type a password and no browser to hold a cookie.

QuestionServer TLSMutual TLS
Who presents a certificatethe serverthe server and the client
Who verifies a namethe client checks the server's DNS-IDboth; the server also checks the client's SAN, often a SPIFFE ID
What the server learns about the callernothing; it authenticates the caller another way (session, token)which workload holds the client key
What it never settleswhether the caller may do the actionwhether the caller may do the action

The last row is identical on purpose. Neither form of TLS answers the authorization question.

Diagram: Mutual TLS at the quote service · the handshake names the caller, two later checks decide

Time runs down the page. The agent opens a connection; the quote service presents its certificate and asks for the client's with CertificateRequest. The agent sends its chain and a CertificateVerify signature made with its private key. The quote service verifies chain, validity, SAN and key, and ends the handshake with a verified SPIFFE ID. Only then does the request arrive. The quote service asks its allowlist whether this workload may call this operation, and the policy decision point whether this subject may act on this record. For decision.record the allowlist already says no to the agent.

  1. Underwriting agent (application) to Quote service: ClientHello. The agent starts the TLS 1.3 handshake.
  2. Quote service to Underwriting agent (application): ServerHello + server certificate + CertificateRequest. The server authenticates itself and asks the client for a certificate. Without CertificateRequest, the client sends none.
  3. Underwriting agent (application) to Quote service: client certificate chain + CertificateVerify. The chain names the agent; the signature over the whole handshake proves it holds the key.
  4. Quote service: verify chain, validity, SAN, key. Any failure ends the connection before a request is read.
  5. Underwriting agent (application) to Quote service: request: decision.record SUB-1004. Arrives over the authenticated connection, from a verified peer.
  6. Quote service to Caller allowlist: may this SPIFFE ID call decision.record?. A coarse check on the operation, using the verified peer id, never a header.
  7. Caller allowlist to Quote service: no. Handshake succeeded, operation refused.
  8. Quote service to Underwriting agent (application): 403. The model recommends; a person decides. Only the cockpit BFF carries decisions.
  9. Quote service to Policy decision point: for allowed callers: may this subject act on this record?. For the cockpit BFF the allowlist says yes, and the policy decision point still checks the underwriter's assignment and step-up.

Read the diagram from top to bottom. The first four messages are the handshake: the quote service ends it knowing a verified SPIFFE ID, or it closes the connection. Only then does a request arrive, and the quote service asks two more questions the handshake cannot answer: is this workload on the allowlist for this operation, and does the policy allow this subject on this record?

In our insurance example: the agent's certificate at the quote service#

The quote service checks the agent's certificate with mtls.py verify_peer, in the order chain, validity, SAN, key. The snippet below runs it for six situations at the simulated clock. It demonstrates which check catches which fault. A successful run proves the order and the separation of checks in this teaching model; it does not exercise any real TLS library.

import mtls
from clock import START_EPOCH_S as NOW

H = 3600


def show(label, cert, san, t, key=None, trust=mtls.TRUST_BUNDLE):
    v = mtls.verify_peer(mtls.chain_for(cert), trust, san, t, presented_key_id=key)
    failed = [c["name"] for c in v["checks"] if not c["passed"]]
    print(f"{label:<13} handshake_ok={v['handshake_ok']!s:<5} failed={failed}")
    for c in v["checks"]:
        if not c["passed"]:
            print("              ", c["detail"])
    return v


print("chain presented by agent-2026a:")
for c in mtls.chain_for("agent-2026a"):
    print(f"  {c['id']:<12} subject={c['subject']:<40} issuer={c['issuer']}  is_ca={c['is_ca']}")
print()
show("valid", "agent-2026a", mtls.AGENT_ID, NOW)
show("stolen-cert", "agent-2026a", mtls.AGENT_ID, NOW, key="k-attacker")
show("wrong-san", "quote-server", mtls.AGENT_ID, NOW)
show("untrusted-ca", "rogue-agent", mtls.AGENT_ID, NOW)
show("expired", "agent-2026a", mtls.AGENT_ID, NOW + 2 * H)
show("rotated", "agent-2026b", mtls.AGENT_ID, NOW + 2 * H)

Run in a scratch copy of examples/uw-auth on 2026-11-18 (the agent here is a scripted stand-in, so these results show how the access controls work, not how good any model is):

chain presented by agent-2026a:
  agent-2026a  subject=CN=underwriting-agent                    issuer=CN=Demo Workload CA (insurer.example)  is_ca=False
  ca-workload  subject=CN=Demo Workload CA (insurer.example)    issuer=CN=Demo Root CA (insurer.example)  is_ca=True
  ca-root      subject=CN=Demo Root CA (insurer.example)        issuer=CN=Demo Root CA (insurer.example)  is_ca=True

valid         handshake_ok=True  failed=[]
stolen-cert   handshake_ok=False failed=['key']
               peer could not prove possession of the private key
wrong-san     handshake_ok=False failed=['san']
               URI SAN spiffe://insurer.example/ns/uw/sa/quote does not match spiffe://insurer.example/ns/uw/sa/underwriting-agent
untrusted-ca  handshake_ok=False failed=['chain']
               issuer CN=Unknown CA is not in the trust bundle
expired       handshake_ok=False failed=['validity']
               certificate agent-2026a is outside its validity window (not_after 1794996000, now 1794999600)
rotated       handshake_ok=True  failed=[]

Each failure names one check. stolen-cert has a genuine certificate and fails only at key possession. wrong-san is the quote service's own valid certificate presented where the agent's was expected. untrusted-ca carries the right SPIFFE ID, signed by a CA nobody trusts: a SAN means nothing without the chain. expired is the 10:00Z incident; rotated, the agent's next certificate at the same moment, is the fix.

Diagram: A certificate chain and the four checks · three look at documents, one looks at the live connection

Read left to right. The root is trusted because the quote service put it in its trust bundle on purpose. The root signed the workload CA, and the workload CA signed the agent's leaf certificate, whose SAN holds the SPIFFE ID. The verifier then runs four checks in order: the chain ends at a trusted root, every certificate is inside its validity window, the SAN is the name it expected, and the peer proved it holds the private key in this handshake. Only the last check involves the live peer. Passing all four yields a verified name and nothing else: no permission.

  1. The chain the agent presents
    1. root CA: Trusted because the verifier chose to put it in its trust bundle, not because of anything it says about itself.
    2. intermediate CA: Signed by the root, signs workload certificates. Its compromise is serious; the root's would be worse.
    3. leaf certificate: Binds the agent's public key to its name for 24 hours in the example. Not a CA: it cannot sign others.
  2. What the quote service checks
    1. chain: Each certificate is signed by the next, and the last one is in the trust bundle.
    2. validity: Every certificate in the chain is inside its window now.
    3. SAN: The URI SAN is the SPIFFE ID the service expected (a DNS name for a public server).
    4. key possession: The peer signs the handshake with the private key (CertificateVerify). A copied certificate cannot do this.
  3. What it yields
    1. a verified name: The service knows which key holder is calling. Whether it may perform the operation is the allowlist's and the policy's question.

The diagram puts the chain and the four checks side by side. Notice that three of the four checks look at the certificate and one looks at the live connection. A verifier that skips the live check is verifying a document, not a peer.

The mTLS lab#

Change the certificate, clock, pinned SAN, key proof and operation, and watch the handshake checks and, separately, the operation decision. The table holds the same recorded results.

ScenarioCertificateClockOperationHandshakeOperation decision
valid-readagent-2026a09:00Zsubmission.readestablishedallow 200
allowlistagent-2026a09:00Zdecision.recordestablisheddeny 403: not on the allowlist for decision.record
cockpit-decidecockpit-202609:00Zdecision.recordestablishedallow 200
expiredagent-2026a11:00Zsubmission.readrefused at validity (expired 10:00Z)deny 401
rotatedagent-2026b11:00Zsubmission.readestablishedallow 200
overlapagent-2026b09:00Zsubmission.readestablished (both agent certificates valid 08:00Z to 10:00Z)allow 200
wrong-sanquote-server09:00Zsubmission.readrefused at SANdeny 401
untrusted-carogue-agent09:00Zsubmission.readrefused at chain (CN=Unknown CA not trusted)deny 401

The allowlist of the example: submission.read for the underwriting agent and the cockpit BFF; decision.record for the cockpit BFF only. The demo certificates are synthetic and the clock is simulated.

A certificate names a key holder, not a permission#

Look at the allowlist row. The handshake is fine. The quote service knows, with cryptographic confidence, that the caller is the underwriting agent. And it still refuses decision.record, because the agent's identity is not on the list of callers allowed to perform that operation. The model recommends; a person decides, so the only workload allowed to carry a decision is the cockpit BFF, where a stepped-up underwriter acts.

The snippet runs authorize_peer, the receiving service's decision on a verified peer id, for three calls. It demonstrates that the decision is a separate function that takes the verified id as input. It does not show the record-level policy, which the policy decision point adds afterwards.

import mtls
from clock import START_EPOCH_S as NOW

for cert, op in [("agent-2026a", "submission.read"), ("agent-2026a", "decision.record"), ("cockpit-2026", "decision.record")]:
    v = mtls.verify_peer(mtls.chain_for(cert), mtls.TRUST_BUNDLE, None, NOW)
    a = mtls.authorize_peer(v["peer_id"], op, mtls.ALLOWLIST)
    print(f"{cert:<13} {op:<16} handshake_ok={v['handshake_ok']}  -> {a['status']} {a['reason']}")
agent-2026a   submission.read  handshake_ok=True  -> 200 spiffe://insurer.example/ns/uw/sa/underwriting-agent is on the allowlist for submission.read
agent-2026a   decision.record  handshake_ok=True  -> 403 handshake succeeded, but spiffe://insurer.example/ns/uw/sa/underwriting-agent is not on the allowlist for decision.record
cockpit-2026  decision.record  handshake_ok=True  -> 200 spiffe://insurer.example/ns/uw/sa/cockpit-bff is on the allowlist for decision.record

Even the third line is not a permission to decide; it says the cockpit BFF may carry decisions at all. Whether this underwriter may decide on this submission is the policy decision point's job, requiring the assigned underwriter, the uw.decide entitlement, and a phishing-resistant login within 300 seconds (rule R-AZ-09). Three layers, three questions: which key holder, which caller may call this operation, which subject may act on this record.

What this is not. The allowlist and SPIFFE IDs belong to Workload identity; the record-level decision belongs to Authorization and Enforcement. This page stops at "the handshake produced a verified name".

Revocation: the check that is often not there#

Certificates can be withdrawn before they expire, for example when a key leaks. Certificate revocation is that withdrawal, and the hard part is telling every verifier in time.

Standard (RFC 5280): a certificate revocation list (CRL) is a periodically issued, signed list of revoked serial numbers; nextUpdate "indicates the date by which the next CRL will be issued", so a verifier caching a CRL can accept a revoked certificate until then. The Online Certificate Status Protocol (OCSP) asks a responder about one certificate at a time.

The public web has moved away from OCSP. Product (Let's Encrypt, as documented on 2026-11-18): in a post dated 6 August 2025, "Today we turned off our Online Certificate Status Protocol (OCSP) service"; it now publishes revocation "exclusively via Certificate Revocation Lists (CRLs)", citing OCSP's "considerable risk to privacy". The other answer is short lifetimes: its shortlived profile issues 160-hour certificates, which "greatly reduce the potential compromise window". Guideline (CA/Browser Forum Baseline Requirements 2.3.1, read 2026-11-18): maximum public certificate validity falls to 200 days from 2026-03-15, 100 days from 2027-03-15 and 47 days from 2029-03-15.

Inside one organization the example's agent certificates live 24 hours, so a leaked key is useful for at most a day, and the CA can stop renewing a compromised workload. This site's recommendation: for workload mTLS, prefer short lifetimes and automated renewal over revocation checks you cannot guarantee; keep a CRL for the intermediate and for long-lived certificates you cannot avoid.

Production pitfall. Whether a TLS library checks revocation at all, and whether it accepts the peer when the CRL cannot be fetched, is a property of that library and its configuration; the sources read for this page do not settle it for any one library. Test what yours does before you count revocation as a control. Guideline (OWASP Microservices Security Cheat Sheet, read 2026-11-18): "The main challenges of using mTLS are key provisioning and trust bootstrap, certificate revocation, and key rotation."

Renewal, rotation and expiry recovery#

Back to 10:00Z. The certificate did not fail; the process around it did. A certificate with a 24-hour life needs a renewal every day, and a renewal process needs three properties.

Overlap. The next certificate must be valid before the current one expires, so the workload can switch while both work. In the example, agent-2026b became valid at 08:00Z and agent-2026a expired at 10:00Z: a two-hour window in which either one passes. Certificate rotation is that switch: new key, new certificate, then retire the old one, with no moment where the workload has nothing valid.

Renew early, from the workload's side. The sources read here fix no universal renewal point, so the fraction of lifetime is a team choice. Trust bundles rotate too: verifiers accept a new intermediate before the CA signs with it.

Alert before expiry. Monitor the remaining lifetime of every certificate a service presents or accepts.

Diagram: Rotation with overlap · the next certificate is valid before the current one expires

Read left to right along the simulated morning. At 08:00Z the agent's next certificate, agent-2026b, becomes valid while agent-2026a still has two hours left. Between 08:00Z and 10:00Z either certificate passes, so the agent can switch at any moment. At 10:00Z agent-2026a expires; a workload that has not switched fails its handshake. The recovery is a new certificate from the normal CA, never turning verification off.

  1. Before 08:00Z
    1. agent-2026a in use: The current certificate, issued with a 24-hour life.
    2. renewal job runs: The workload or an agent beside it asks the CA for the next certificate well before expiry, with a new key.
  2. 08:00Z to 10:00Z: overlap
    1. both valid: Either certificate passes the validity check, so the switch causes no failed call.
    2. agent switches: New connections use agent-2026b; old connections drain.
  3. From 10:00Z
    1. agent-2026a expired: Any workload still presenting it fails the handshake: 401 before a request is read.
    2. recovery: issue, do not disable: Issue an emergency certificate from the normal CA with an approval and an audit record. Turning verification off would admit every caller.

The diagram shows the two agent certificates on one timeline. The whole point is the shaded overlap: rotation without downtime needs a moment where both are valid, and the renewal job must finish inside it.

When expiry happens anyway, the recovery is the same in every design: issue a new certificate, do not lower verification. Turning verification off, adding a "temporary" trust for a self-signed certificate, or rolling the verifier's clock back each removes authentication for every caller, not only for the one whose certificate expired. This site's recommendation: keep a runbook that issues an emergency certificate for one workload from the normal CA, with an approval and an audit record, and test it. The teammate's suggestion in the opening is the mistake this runbook exists to prevent.

Public CA or internal CA: client certificates after 8 July 2026#

Who should sign workload certificates? For servers that browsers reach, a public CA is the default, because every browser already trusts it. For mTLS between workloads, the answer changed this year.

Product (Let's Encrypt, as documented on 2026-11-18): Let's Encrypt no longer issues certificates with the TLS Client Authentication EKU. The default classic profile dropped it on 11 February 2026, and the temporary tlsclient profile was retired on 8 July 2026: "no further certificates with the Client Authentication EKU will be issued." The reason it gives is the Chrome root program's "June 2026 deadline to split TLS Client and Server Authentication into separate PKIs." Its own advice: "Many uses of client authentication are better served by a private certificate authority."

The implication for mTLS is direct: such a certificate fails a strict client-authentication check (run 4 of the OpenSSL script). And a public CA vouches for domain control, which says little about which internal workload is calling. An internal CA under your control can issue SPIFFE-style names, short lifetimes and both purposes.

SituationSimplest credible choiceGrow when
A browser reaches your serverpublic CA certificate with automated renewalmore names and hosts: automate per host, never share keys
A handful of internal services call each otherinternal CA (root offline, one intermediate) issuing short-lived client and server certificates; allowlist at each receivermany teams or clusters: a workload identity system such as SPIRE
Partners call your API with certificatesa partner CA you operate, or certificates pinned per partner; RFC 8705 tls_client_auth at the token endpointmany partners: federation of trust bundles, per-partner allowlists
A scheduled job, no personthe job's own workload certificatenever a person's certificate for a job
In a coding agent. A coding agent's sandbox needs no certificate for production services at all. CI that deploys reaches the cloud through OIDC federation, not a stored client key, and a person with merge rights approves the pull request that changes certificate configuration. Coding agents owns that design.

Security notes#

Monday checklist#

  1. List every certificate each service presents and accepts, with its expiry. Add an alert that fires well before the earliest one.
  2. For one mTLS client and one server, confirm in code or configuration which name is checked (DNS-ID or URI SAN) and that the check cannot be skipped.
  3. Find any verify=False, InsecureSkipVerify, trust-all TrustManager or -k in code and scripts, and remove it or document why it is safe.
  4. Check that the next certificate is issued before the current one expires, and measure the overlap on a real rotation.
  5. Find out whether your TLS stack checks revocation, and whether it fails open when the CRL is unreachable.
  6. If you used public certificates as mTLS client certificates, plan the move to an internal CA: since 8 July 2026 Let's Encrypt issues none with the client-authentication EKU.

Exercises#

1. The quote service receives a client certificate that chains to the trusted root, is inside its validity window, and carries the SAN spiffe://insurer.example/ns/uw/sa/quote. The quote service expected the underwriting agent. What happens, and which check decides?

AnswerThe SAN check refuses it

The handshake is refused at the SAN check (the wrong-san scenario, 401). A valid certificate for a different workload is still the wrong certificate.

2. A teammate proposes renewing certificates at exactly their expiry time, "so we never waste validity". What breaks?

AnswerThere is no overlap

Any delay in renewal or in distributing the new certificate leaves the workload with nothing valid: the 10:00Z incident. Rotation without downtime needs an overlap window, like 08:00Z to 10:00Z in the example.

3. The cockpit BFF completes mTLS with the quote service and is on the allowlist for decision.record. Can the peer underwriter u-uw-w2 now record a decision on SUB-1004 through the cockpit?

AnswerNo: the record decides

No. mTLS and the allowlist say the cockpit BFF may carry decisions. The policy decision point still checks assignment on the record: u-uw-w2 is not assigned to SUB-1004, so the answer is 403 (UW-A09 under authz-2026.11.1).

Advanced perspective#

Where TLS ends: proxies, load balancers and forwarded certificatesmTLS proves the last hop only

mTLS authenticates the two ends of one TCP connection. If a load balancer terminates TLS and opens a new connection to the quote service, the quote service authenticates the load balancer, not the agent. A common pattern forwards the client certificate in an HTTP header from the proxy to the service. That header is exactly as forgeable as x-app-id unless two conditions hold: the service accepts it only from the proxy, verified by the proxy's own mTLS identity, and the proxy strips any incoming header of the same name. Otherwise any caller that reaches the service directly can type a certificate into a header. Prefer end-to-end mTLS to the workload, or a sidecar on the same host whose connection to the application cannot be reached from the network. Record which design you run, because the audit question "who called?" has a different answer in each.

Certificate-bound tokens: one step past the handshakeRFC 8705 ties a token to the client certificate

mTLS proves the caller at the connection; a bearer token proves nothing about who presents it. RFC 8705 joins the two: the authorization server puts a hash of the client's certificate into the token, and the resource server checks that the certificate on the connection matches. Standard (RFC 8705, read 2026-11-18): the hash "SHOULD be represented using the x5t#S256 confirmation method member" inside cnf; the resource server "MUST verify that the certificate matches the certificate associated with the access token", and a mismatch is 401 invalid_token. The same RFC defines tls_client_auth and self_signed_tls_client_auth, two ways a client authenticates to the token endpoint with a certificate. A token copied out of a log is then useless without the key. Plan for rotation: tokens bound to agent-2026a stop working when the agent switches to agent-2026b, so token lifetime must fit inside the overlap. Tokens compares this with DPoP; Workload identity shows where it fits on a call between services.

Limiting what a stolen CA key can doOffline roots, constrained intermediates, separate trust domains

Every verifier trusts whatever the CA signs, so a stolen intermediate key can mint a certificate for any workload name. Four controls shrink that. Keep the root offline, used only to sign intermediates. Give each intermediate pathlen:0 so it cannot create sub-CAs, and where the library supports it, name constraints so it can sign only names under one trust domain (RFC 5280 notes that name constraints "apply only when the specified name form is present", so test with URI SANs, not only DNS). Use separate roots for production and staging; the SPIFFE concepts page advises separate trust domains for environments such as production and staging. And make intermediate rotation a rehearsed operation: publish the new intermediate in every trust bundle first, switch signing, then withdraw the old one. The quote service's allowlist is the last line here: a forged certificate for the agent still cannot record a decision, because the agent is not on that list.

Revocation designs comparedCRL, OCSP, short life, and stopping renewal

A CRL is simple and private (the verifier downloads one list, revealing nothing about which peer it is checking), but it is stale up to nextUpdate and grows with every revocation. OCSP is fresher per certificate but adds a live dependency at every handshake and tells the responder who connects to whom, which is the privacy cost Let's Encrypt gave for ending its service. Short-lived certificates turn revocation into "stop renewing": the window is the remaining lifetime, known in advance. For workload identity inside one organization, the last option is usually the right default; the residual risk is the remaining lifetime, so choose it with the access-removal time your Operations page promises in mind. Whatever you choose, decide explicitly whether a verifier that cannot reach revocation data fails closed or open, and write that into the service's runbook.

Clocks are part of the trust modelSkew, simulated time and expiry testing

Every validity check compares against the verifier's clock. A container with a drifting clock rejects fresh certificates as "not yet valid" or accepts expired ones. Short lifetimes make this sharper: a one-day certificate leaves little room for minutes of skew at either end. Run time synchronization on every host, allow a small, documented tolerance for not_before, and never allow tolerance on not_after large enough to matter. In tests, inject the clock instead of waiting: the example's clock is simulated from 2026-11-18T09:00:00Z, which is why expired and rotated are deterministic, and the OpenSSL run used -attime for the same reason. Add a test for each boundary: one second before expiry passes, one second after fails.

In one breath#

A certificate is a CA's signed statement that a public key belongs to a name, for a time. A verifier trusts it only after four checks: the chain ends at a root it chose, every certificate is current, the name is the one it expected, and the peer proved it holds the private key in this handshake. mTLS applies those checks to the caller as well as the server, and still decides nothing about what the caller may do. Certificates expire on purpose, so renew early with overlap, and recover from expiry by issuing a new certificate, never by turning verification off.

The quote service now knows, by a verified certificate, that the underwriting agent is calling. It does not yet know how the agent got that certificate without a secret baked into its image, how two agents in one cluster keep distinct names, or how the agent's identity travels together with the underwriter it acts for. Workload identity answers those three questions.