All chapters
  1. Start hereA valid token is not a permission
  2. Foundations · 1Identity and trust primitives: who is calling, and who checks
  3. Foundations · 2People and identity proofing: accounts, passwords and recovery
  4. Foundations · 3Sessions and browser security: cookies, CSRF, XSS and the BFF
  5. Foundations · 4Authenticators and MFA: passwords, passkeys and phishing resistance
  6. Foundations · 5API keys and application credentials: what a key proves
  7. Foundations · 6Tokens: bearer, JWT anatomy, validation and revocation
  8. Foundations · 7OAuth, OpenID Connect and federation: delegation and login
  9. Authorization · 8Authorization from first principles: subject, action, resource, context
  10. Authorization · 9Access control models: RBAC, ABAC, ReBAC and ownership
  11. Authorization · 10The enforcement architecture: PEP, PDP, PIP and PAP
  12. Authorization · 11Policy as code: versions, tests, engines and decision logs
  13. Authorization · 12Tenant and data isolation: one platform, many brokerages
  14. Authorization · 13Identity governance and lifecycle: joiner, mover, leaver
  15. Authorization · 14Privileged access: standing rights, just-in-time elevation and break-glass
  16. Workloads · 15TLS, PKI and certificates: proving a host, and a client
  17. Workloads · 16Workload and service identity: SPIFFE, mTLS and caller allowlists
  18. Workloads · 17API gateways and zero trust: the edge checks, the service decides
  19. Workloads · 18Event-driven identity: Kafka principals, ACLs and replay
  20. Workloads · 19Cloud IAM and secretless deployment: roles, federation and no stored keys
  21. Workloads · 20External SaaS, webhooks and trust: calls that cross company lines
  22. Workloads · 21Private controlled environments: what private networking does and does not secure
  23. AI-native · 22Authenticating to model providers: keys, gateways and cloud IAM
  24. AI-native · 23MCP identity and authorization: discovery is not permission
  25. AI-native · 24RAG and retrieval authorization: only what this caller may read
  26. AI-native · 25Agents as principals and delegates: whose authority is this?
  27. AI-native · 26Agent-to-agent trust: identity, capabilities and forwarded authority
  28. AI-native · 27Coding agents and CI: repository scopes, OIDC and no production keys
  29. Operations · 28Auditing and observability of access: decisions you can prove later
  30. Operations · 29Threat modeling and testing: the negative tests that prove access control
  31. Operations · 30Operational lifecycles: rotation, revocation and outages
  32. Reference · 31Standards, frameworks and products: what each one is, and what it is not
  33. Reference · 32Architecture decision playbooks: choosing an auth strategy
  34. ReferenceGlossary
  35. AboutAbout this site and its sources
Workloads · 18

Event-driven identity: Kafka principals, ACLs and replay

Producers and consumers are principals too. SASL SCRAM, OAUTHBEARER and mTLS, topic and consumer group ACLs, schema registry access, provisioning, event provenance without bearer tokens, and authorization on replay and dead letter topics.

You will understand
Why a producer and a consumer are separate principals, and how the broker answers "who is this client?" before "may it do this?".
You will be able to read
A Kafka ACL: literal or prefixed, allow or deny, on a topic, a consumer group or a transactional id.
You will keep apart
Event provenance (who caused this event) from a bearer token, and replaying an event from being allowed to act on it.

The broker's quote Q-5531 has become submission SUB-1004 (the portal creates the quote_id, intake maps it one to one to the submission_id, and every decision is made on the record both point to). The intake service publishes a "submission created" event to the topic uw.submissions.v1. The underwriting workflow reads that topic and starts run run-7f3a, in which the underwriting agent gathers evidence for series case UW-004.

In the design review, an engineer points out a gap. The underwriting workflow later calls the quote API, and the quote API decides per broker and per tenant. So why not put the broker's access token into the event? The consumer could then call the API "as the broker", and every downstream check would see the right person.

That proposal is the mistake this page starts from. An event is not a request: it is copied to every consumer, kept after it is read, and read again on replay. A bearer token inside it becomes a credential that anyone with read access to the topic can lift and use until it expires, and the consumer that uses it is no longer itself; it is pretending to be the broker. The questions of this page follow from that scene: who is this client to the broker, what may it do to this topic, and what should an event say about who caused it, without carrying anyone's power?

Producers and consumers are principals#

On the broker side, every connection belongs to a principal. A Kafka principal is the name the broker gives an authenticated client, written as a type and a name. A tiny example: the intake service's producer connects with its credential, and the broker calls it User:intake-producer; the underwriting workflow's consumer is User:underwriting-consumer. Product (Apache Kafka 4.4, as documented on 2026-11-18): principals are written PrincipalType:name, and the authorizer decides operations for that principal.

The example keeps the two roles apart on purpose. The intake service writes submissions; it never reads them back. The underwriting workflow reads submissions and writes recommendations; it never writes submissions. If both used one shared client identity, a bug or a compromised consumer could publish fake submissions, and the topic's audit would show "the platform" did it.

Plain English. Every program that talks to Kafka needs its own name, proved to the broker. The broker then checks a list that says which names may write or read which topics. Two different questions, two different mechanisms.
Before AI. This is the Kafka you may already run: a client keystore per service, a CN in a certificate, kafka-acls.sh in a provisioning script, and a ticket to "get produce rights on the orders topic". Nothing about an agent changes the mechanism. What changes is that more workloads (intake agents, replay tools, evaluation jobs) want to read and write, and each one needs its own principal.

Proving who the client is: SSL, SCRAM, OAUTHBEARER#

The broker cannot check a permission until it knows the name, so authentication comes first. Product (Apache Kafka 4.4, as documented on 2026-11-18): Kafka authenticates connections from clients, brokers and tools "using either SSL or SASL"; the SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. SASL (Simple Authentication and Security Layer) is a framework that lets a protocol plug in different login mechanisms; the broker enables a list with sasl.enabled.mechanisms, and each client picks one with sasl.mechanism.

Three choices fit a service like the intake producer:

Mutual TLS. The client presents a certificate during the TLS handshake. Product (Kafka 4.4): ssl.client.auth=required means "client authentication is required"; requested lets clients without certificates connect anyway and is discouraged because it "provides a false sense of security". By default the principal is the certificate's full distinguished name, such as CN=writeuser,OU=Unknown,..., and ssl.principal.mapping.rules maps it to a short name, first matching rule wins. Certificates, chains and rotation are explained on TLS and PKI.

SASL/SCRAM. SCRAM (Salted Challenge Response Authentication Mechanism) is a username and password login in which the password never crosses the wire; the client proves it knows it. Product (Kafka 4.4): "The default SCRAM implementation in Kafka stores SCRAM credentials in the metadata log", and kafka-configs.sh creates and updates them. Simple to start; it is still a long-lived shared secret per client.

SASL/OAUTHBEARER. OAUTHBEARER lets a client authenticate with an OAuth access token. The client gets the token from the identity provider (for a service, with client credentials, see OAuth and OIDC); the broker validates it. Product (Kafka 4.4): the default implementation uses unsecured JWTs and "is suitable only for non-production use"; for production the broker sets sasl.oauthbearer.jwks.endpoint.url, sasl.oauthbearer.expected.audience and sasl.oauthbearer.expected.issuer, and with the JWKS URL set it "fails at startup unless" audience and issuer are configured. sasl.oauthbearer.sub.claim.name names the claim that becomes the principal. On the client, sasl.oauthbearer.token.endpoint.url points at the token endpoint, and the default retriever uses client credentials for an HTTP(S) URL.

Common confusion. OAUTHBEARER puts a bearer token on the connection to the broker, which validates it once at login for that client. That is the recommended use. The labelled mistake of this page is different: a person's token copied into the event payload, where every reader can take it.

Product (Kafka 4.4): "Security is off by default." A fresh cluster accepts unauthenticated PLAINTEXT connections, and with no authorizer configured, "all authenticated principals have full access". The security model adds: "Production deployments must explicitly configure authentication, authorization, and transport encryption", and "mTLS is the recommended mechanism for broker-to-broker and controller-to-broker traffic." The example models a client that connected without authenticating as the principal User:ANONYMOUS; the 4.4 pages read here do not name that principal, so treat the name as the example's convention and check your broker.

The smallest primitive: one ACL and one check#

Once the broker knows the name, the authorizer decides each operation. Product (Kafka 4.4): with the KRaft authorizer org.apache.kafka.metadata.authorizer.StandardAuthorizer, an ACL has the documented shape: principal P is allowed or denied operation O from host H on any resource R matching resource pattern RP. A Kafka ACL is one such line. Resource types include Topic, Group, Cluster and TransactionalId; operations include Read, Write, Create, Describe, IdempotentWrite and All. A pattern is literal (the exact name, the default) or prefixed (every name that starts with the string).

What a producer and a consumer need comes from the documented operations table. Product (Kafka 4.4): a producer needs Write on the topic, Describe on the topic for metadata, IdempotentWrite on the cluster for idempotent produce, and Write on the TransactionalId when transactional.id is set. A consumer needs Read on the topic and Read on its consumer group: the named set of consumers that share a topic's partitions and commit their read positions together. A tiny example: the underwriting workflow consumes as group uw-underwriting; a second program that joins that group would take over part of its work and move its committed position, which is why joining a group is a permission of its own.

Two defaults decide most surprises. Product (Kafka 4.4): if no ACL matches a resource, "only super users are allowed to access it". The setting allow.everyone.if.no.acl.found=true opens resources that have no ACLs at all and "does not weaken or skip ACL checks when any ACL exists for that resource"; the documentation says to keep the default false in production, and the security model calls the setting a migration aid that "should not remain set in steady state". Super users bypass ACLs (super.users=User:Bob;User:Alice, separated by semicolons because SSL names may contain commas); the security model says to treat that list "as you would a root password".

The third rule is the one people argue about: does a DENY beat an ALLOW? The Apache Kafka 4.4 page read here does not state it in one sentence. Product (Confluent Platform 8.4, as documented on 2026-11-18): "deny ACLs take precedence over allow ACLs". The example infers the same rule from the 4.4 page's own --deny-principal example, which grants everyone Read on a topic and then denies one principal from one host; that example only makes sense if the matching deny wins. The example implements deny-wins and says in every result that a deny decided it.

The example module examples/uw-auth/kafka_acl.py is a teaching model of those documented semantics, not Kafka: no broker, no client library. Its decision function, quoted verbatim:

def authorize(principal: str, operation: str, resource_type: str, resource_name: str, acls: list[dict[str, Any]],
              *, host: str = "*", super_users: list[str] | None = None, allow_if_no_acl: bool = False) -> dict[str, Any]:
    """One authorization check, as the model of the documented semantics. Returns {allow, reason, matched}."""
    if principal in (super_users or []):
        return {"allow": True, "reason": principal + " is a super user", "matched": []}
    on_resource = [a for a in acls if _resource_match(a, resource_type, resource_name)]
    if not on_resource:
        if allow_if_no_acl:
            return {"allow": True, "reason": "no ACL matches " + resource_type + ":" + resource_name + " and allow.everyone.if.no.acl.found=true", "matched": []}
        return {"allow": False, "reason": "no ACL matches " + resource_type + ":" + resource_name + ": only super users may access it", "matched": []}
    hits = [a for a in on_resource
            if a["principal"] in (principal, "User:*") and a["operation"] in (operation, "All") and a["host"] in ("*", host)]
    denies = [a["id"] for a in hits if a["permission"] == "DENY"]
    if denies:
        return {"allow": False, "reason": "DENY " + ",".join(denies) + " matches; deny takes precedence (inferred from the documented deny-principal example)", "matched": denies}
    allows = [a["id"] for a in hits if a["permission"] == "ALLOW"]
    if allows:
        return {"allow": True, "reason": "ALLOW " + ",".join(allows) + " matches " + operation + " on " + resource_type + ":" + resource_name, "matched": allows}
    return {"allow": False, "reason": "no ALLOW ACL for " + principal + " " + operation + " on " + resource_type + ":" + resource_name, "matched": []}

Read it in order: super users first; then "does any ACL exist for this resource?" (if none, deny unless the migration flag is on); then a matching deny; then a matching allow; otherwise deny. A successful run proves the model follows the rules as written above. It does not prove anything about a real broker: the model leaves out hosts beyond *, implicit Describe, and every protocol detail.

The insurance example: seven ACLs for three clients#

The example provisions three clients and seven ACLs (kafka_acl.ACLS):

ACLPrincipalPermissionOperationResourcePattern
acl-1User:intake-producerALLOWWriteTopic uw.submissions.v1literal
acl-2User:underwriting-consumerALLOWReadTopic uw.submissions.v1prefixed
acl-3User:underwriting-consumerDENYReadTopic uw.submissions.v1.dlqliteral
acl-4User:underwriting-consumerALLOWReadGroup uw-underwritingliteral
acl-5User:underwriting-consumerALLOWWriteTopic uw.recommendations.v1literal
acl-6User:replay-toolALLOWReadTopic uw.submissions.v1.dlqliteral
acl-7User:replay-toolALLOWReadGroup uw-replayliteral

Look at acl-2 and acl-3 together. A prefixed pattern uw.submissions.v1 also matches uw.submissions.v1.dlq, the dead letter topic: the topic where a consumer parks events it could not process, so that the main flow keeps moving and a person can look at the failures later. Without acl-3, the underwriting consumer could read every failed submission, including ones it was never meant to retry. acl-3 denies that, and deny precedence makes the denial stick.

Diagram: Two questions at the broker · who is this client, then may it do this

Read each lane left to right. The listener authenticates the connection and produces a principal: a certificate name, a SCRAM user or a token subject. Only then does the authorizer check the ACL table, once per resource the operation touches. The intake producer passes one check (Write on the topic). The underwriting consumer passes two (Read on its group and on the topic). A client that did not authenticate is modelled as User:ANONYMOUS and finds no ACL, so it is refused. Authentication never grants an operation by itself.

  1. Intake service · producer
    1. connects with mTLS or SASL: The intake service's producer proves its identity on the listener: a client certificate, a SCRAM login or an OAuth token.
    2. principal User:intake-producer: The broker names the authenticated client. Mapping rules or a token claim decide the name.
    3. Write on Topic uw.submissions.v1: The authorizer finds a literal ALLOW for Write on the topic.
  2. Underwriting workflow · consumer
    1. principal User:underwriting-consumer: The consumer has its own principal, never the producer's.
    2. Read on Group uw-underwriting: JoinGroup and offset commits need Read on the group.
    3. Read on Topic uw.submissions.v1: The prefixed allow matches the topic. Both checks must pass for the consumer to read.
  3. Unauthenticated client
    1. plaintext listener, no credential: A listener that accepts unauthenticated connections lets any client in. Kafka's security is off by default.
    2. principal User:ANONYMOUS: The example models the unauthenticated client as User:ANONYMOUS (its own convention; the 4.4 pages read do not name it); no ACL names it.
    3. refused on group and topic: No matching ALLOW, so both checks fail. Authentication succeeded in a sense (the connection exists), authorization did not.

The diagram follows each client through the two questions. Authentication on the listener produces a principal, or User:ANONYMOUS, or no connection at all when the handshake fails. Only then does the authorizer read the ACL table, once per resource the operation touches: a consumer is checked on the group and on the topic, and both must pass.

Run the example#

The commands below ran in a scratch copy of examples/uw-auth on 2026-11-18. The first is this site's own case UW-A15 ("Kafka: consumer write and DLQ replay refused"); the second prints every lab scenario and four extra checks from a small driver script that calls kafka_acl.authorize, produce and consume.

/opt/homebrew/bin/python3 run_cases.py --case UW-A15 --verbose
case    result title                                                first check
UW-A15  PASS   Kafka: consumer write and DLQ replay refused         consumer_write=false
    ok  consumer_write: expected false, actual false
    ok  consumer_read_dlq: expected false, actual false
    ok  replay_read_dlq: expected true, actual true
    ok  replay_write_back: expected false, actual false

1/1 cases match their expected block. (Scripted stand-in agent: this shows the access controls, not model quality.)
intake-produce              allow=True  Write Topic:uw.submissions.v1: ALLOW acl-1 matches Write on Topic:uw.submissions.v1
uw-consume                  allow=True  Read Group:uw-underwriting: ALLOW acl-4 matches Read on Group:uw-underwriting | Read Topic:uw.submissions.v1: ALLOW acl-2 matches Read on Topic:uw.submissions.v1
uw-write-submissions        allow=False Write Topic:uw.submissions.v1: no ALLOW ACL for User:underwriting-consumer Write on Topic:uw.submissions.v1
uw-produce-recommendation   allow=True  Write Topic:uw.recommendations.v1: ALLOW acl-5 matches Write on Topic:uw.recommendations.v1
uw-read-dlq-denied          allow=False Read Group:uw-underwriting: ALLOW acl-4 matches Read on Group:uw-underwriting | Read Topic:uw.submissions.v1.dlq: DENY acl-3 matches; deny takes precedence (inferred from the documented deny-principal example)
replay-read-dlq             allow=True  Read Group:uw-replay: ALLOW acl-7 matches Read on Group:uw-replay | Read Topic:uw.submissions.v1.dlq: ALLOW acl-6 matches Read on Topic:uw.submissions.v1.dlq
replay-write-back-refused   allow=False Write Topic:uw.submissions.v1: no ALLOW ACL for User:replay-tool Write on Topic:uw.submissions.v1
consumer-wrong-group        allow=False Read Group:uw-replay: no ALLOW ACL for User:underwriting-consumer Read on Group:uw-replay | Read Topic:uw.submissions.v1: ALLOW acl-2 matches Read on Topic:uw.submissions.v1
anonymous                   allow=False Read Group:uw-underwriting: no ALLOW ACL for User:ANONYMOUS Read on Group:uw-underwriting | Read Topic:uw.submissions.v1: no ALLOW ACL for User:ANONYMOUS Read on Topic:uw.submissions.v1

default deny, no ACL on topic: no ACL matches Topic:uw.audit.v1: only super users may access it
same, allow.everyone.if.no.acl.found=true: no ACL matches Topic:uw.audit.v1 and allow.everyone.if.no.acl.found=true
flag does not open a topic that has ACLs: no ALLOW ACL for User:intake-producer Read on Topic:uw.submissions.v1
super user: User:kafka-admin is a super user
without acl-3, consumer reads DLQ: True

Three lines carry the lesson. uw-write-submissions: a principal that may read a topic may not write it, so a compromised consumer cannot inject submissions. consumer-wrong-group: the topic check passes but the group check fails, so the consumer cannot join the replay tool's group. The last line removes acl-3 and shows the prefixed allow leaking the dead letter topic. The uw-auth example has no model in this flow; elsewhere the agent is a scripted stand-in, so these results show how the access controls work, not how good any model is.

Try it in the lab#

The lab lets you pick a client, how it authenticates, an operation and a topic, and edit the ACL table. Start with the default, then load the dead letter presets.

The same recorded run, for a reader without JavaScript (provisioned ACLs acl-1 to acl-7):

ScenarioPrincipalOperationResultWhy
intake-produceUser:intake-producerwrite uw.submissions.v1allowedALLOW acl-1 (Write, literal)
uw-consumeUser:underwriting-consumerread uw.submissions.v1, group uw-underwritingallowedALLOW acl-4 on the group and acl-2 on the topic
uw-write-submissionsUser:underwriting-consumerwrite uw.submissions.v1refusedno ALLOW ACL for Write on that topic
uw-produce-recommendationUser:underwriting-consumerwrite uw.recommendations.v1allowedALLOW acl-5
uw-read-dlq-deniedUser:underwriting-consumerread uw.submissions.v1.dlqrefusedDENY acl-3 wins over the prefixed ALLOW acl-2 (deny precedence inferred from the documentation's example)
replay-read-dlqUser:replay-toolread uw.submissions.v1.dlq, group uw-replayallowedALLOW acl-7 on the group and acl-6 on the topic
replay-write-back-refusedUser:replay-toolwrite uw.submissions.v1refusedthe replay tool has no Write ACL; it cannot re-inject events as the producer
consumer-wrong-groupUser:underwriting-consumerread uw.submissions.v1, group uw-replayrefusedno ALLOW ACL on group uw-replay (the topic check alone passes)
anonymousUser:ANONYMOUSread uw.submissions.v1, group uw-underwritingrefusedan unauthenticated client gets no ACL at all

Source of the semantics: the Apache Kafka 4.4 Authorization and ACLs page, read 2026-11-18. Implicit Describe is not modelled. A teaching model of the documented semantics, not Kafka.

Event provenance without a bearer token#

Return to the proposal from the design review. The underwriting workflow does need to know who caused the event: which broker, which tenant, which quote. That information is event provenance: facts in the event that say who or what caused it and where it came from. Kafka gives a place to put it. Product (Kafka 4.4, as documented on 2026-11-18): each record can carry headers, each a string key with a byte-array value, and "The order of headers in a record is preserved when producing and consuming." Kafka does not define provenance or user identity inside records; that is an application pattern.

Pattern: the producer writes provenance headers made of identifiers, never credentials. For SUB-1004 the intake service would write something like this (illustrative; the header names are this site's choice):

{
  "event_type": "submission.created",
  "quote_id": "Q-5531",
  "submission_id": "SUB-1004",
  "tenant": "brk-north",
  "business_unit": "commercial-west",
  "caused_by": "u-broker-n1",
  "produced_by": "spiffe://insurer.example/ns/uw/sa/intake-agent",
  "request_id": "req-example-01",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "authz_policy_version": "authz-2026.11.1"
}

Nothing in it can be presented to an API. caused_by names the broker for audit and for the consumer's own decisions; it does not let the consumer act as the broker. When the underwriting workflow later calls the quote API, the underwriting agent uses its own workload credential, narrowed by token exchange to the delegated task (see Agents), and the quote service decides on the record. A valid token is not a permission, and a header that names a broker is not one either.

Why trust the headers at all? Because only User:intake-producer may write uw.submissions.v1 (acl-1, and uw-write-submissions refused above). The consumer's trust in tenant: brk-north rests on the ACL that limits who can write the topic, not on the header itself. That is a real dependency: anyone who obtains the intake producer's credential can write any header they like. Signing the provenance closes that gap at a cost, discussed in the folded notes.

Diagram: Replay from the dead letter topic · three permissions kept apart

Top lane: the labelled mistake. The replay tool gets Write on the submissions topic and becomes a second, unvalidated producer whose writes look like intake's events. Bottom lane: the example's design. The underwriting consumer is denied the DLQ (acl-3 beats the prefixed allow). The replay tool reads the DLQ in its own group but cannot write submissions (UW-A15). The event goes back through the intake service, which validates it, adds replay provenance and produces it as itself. The consumer then decides on the record against current state.

  1. Rejected: replay tool writes back
    1. replay tool gets Write on submissions: A convenience grant that makes an operator tool a producer of business events.
    2. events look like intake's events: Consumers cannot tell a replayed event from a new one, and intake's checks never ran.
  2. The example: replay through the producer
    1. consumer denied the DLQ: The prefixed allow acl-2 would match the DLQ; the explicit deny contains it.
    2. replay tool reads DLQ, own group: The replay tool has its own principal and group, read-only on the dead letter topic.
    3. intake re-validates, produces as itself: The only writer of submissions stays the intake producer; the event keeps its original provenance plus who replayed it and under which approval.
    4. consumer decides on current state: Authorization at processing time: if the submission was withdrawn or reassigned, the old event does not carry the old permission.

Failure mode: replay and the dead letter topic#

The second half of the scene happens a week later. One submission event failed processing and went to uw.submissions.v1.dlq. An operator fixes the cause and wants to replay it.

The tempting design, shown here as a labelled mistake, gives the replay tool Write on uw.submissions.v1 so it can put the event back. That makes the replay tool a second producer of submissions with none of intake's validation, and its writes look like intake's events to every consumer. Case UW-A15 refuses it: replay-write-back-refused.

The design the example models keeps three permissions apart. The underwriting consumer may not read the dead letter topic (acl-3). The replay tool may read it, in its own group uw-replay (acl-6, acl-7). Nobody but the intake producer writes submissions. This site's recommendation: a replay goes back through the producer's own path. The replay tool hands the event to the intake service (or to a replay topic that intake consumes); intake re-validates it and produces it again as itself, adding replayed_by and the approval id to the headers while keeping the original provenance. The consumer then decides on the record at processing time, against current state: if SUB-1004 was withdrawn or reassigned since the event was first produced, the decision made at produce time no longer holds. Authorization is a decision about now, not a property the event carried from the day it was written.

Provisioning: who creates the principal and the ACLs#

Someone has to create User:intake-producer, give it a credential, and write acl-1. Product (Kafka 4.4): kafka-acls.sh manages ACLs; its --producer convenience generates ACLs "that allows WRITE, DESCRIBE and CREATE on topic", and --consumer grants "READ, DESCRIBE on topic and READ on consumer-group". Documented shape, with the example's names substituted; not run here:

# Documented shape (Apache Kafka 4.4 authorization page); names from the example; not run here.
bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --allow-principal User:intake-producer --producer --topic uw.submissions.v1

bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --allow-principal User:underwriting-consumer --consumer --topic uw.submissions.v1 --group uw-underwriting

bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --deny-principal User:underwriting-consumer --operation Read --topic uw.submissions.v1.dlq

Notice that --producer includes Create. If topics are created by the platform team and never by producers, grant operations one by one with --operation instead of the convenience flag. --command-config names "A property file containing configs to be passed to Admin Client", which holds the administrator's own credential: provisioning is itself a privileged identity.

How the request reaches that command is an organizational choice. Pattern (self-service certificate portal): a team registers a topic and a role (produce or consume) in a portal; the portal records an owner, issues a client certificate whose name becomes the principal, writes the matching ACLs, and renews the certificate automatically before it expires. It is one common way to run Kafka at scale, and nothing in Kafka requires it. Alternatives are an infrastructure-as-code repository where ACL changes arrive as reviewed pull requests, or OAUTHBEARER with workload identities, where there is no certificate to distribute and the principal comes from the token's subject claim. The goal behind all three is the same one the cloud chapter pursues: no long-lived secret that a person copies by hand, and an owner for every principal so that a review can find it.

In a coding agent. ACLs kept as code are a good fit for a coding agent: it can propose the pull request that adds acl-5 for a new recommendations topic. A person with merge rights on the provisioning repository reviews and merges, and the pipeline applies the change with its own short-lived credential. The agent's sandbox holds no Kafka administrator credential. See Coding agents.

Schema registry access#

A consumer parses an event with a schema. Schema registry is a service that stores versioned schemas for topics, so that producers and consumers agree on the format. Apache Kafka itself does not include one; the commonly used registry is a Confluent product. Product (Confluent Schema Registry, Confluent Platform 8.4, as documented on 2026-11-18): it supports HTTP Basic authentication, and the documentation warns that "the Basic protocol passes credentials in plain text", so use HTTPS; "Starting with Confluent Platform 7.7, the Schema Registry server supports Open Authentication (OAuth)"; authorization goes through the Schema Registry Security Plugin with an ACL authorizer or RBAC, and "The Schema Registry security plugins require a Confluent Enterprise license."

This site's recommendation: treat "register a new schema version for uw.submissions.v1" as a permission separate from Write on the topic. A principal that can change the schema can change what every consumer believes a field means, without writing a single event. Give registration to the producing team's pipeline, reading to consumers, and nothing to the replay tool.

What this is not. Certificates and mTLS mechanics live on TLS and PKI; SPIFFE identities for these services on Workload identity. Delivery guarantees, retries and recovery of the event flow as a system belong to Reliability Engineering at https://reliable.sammuthu.com.

Architecture choice: how to authenticate clients and run ACLs#

SituationSimplest credible optionGrow whenAlternatives
One team, a few services, one clusterSASL/SCRAM over TLS, one principal per service, literal ACLs in a reviewed scriptMore teams, rotation by hand becomes riskymTLS with an internal CA
Many teams, a platform team runs KafkamTLS from an internal CA with automated renewal, ACLs as codeTeams already have workload tokens from an identity providerSelf-service certificate portal (pattern)
Services already get tokens from an identity providerSASL/OAUTHBEARER with JWKS validation, expected audience and issuerCross-organization accessmTLS for broker-to-broker either way
A tool that replays failuresIts own principal, read-only on the dead letter topic, its own groupReplays need approvalReplay through the producer's validation path

The trigger to grow is usually rotation and ownership, not throughput. A dozen SCRAM passwords are manageable; two hundred, owned by teams that reorganize, are not. This site's recommendation: whatever the mechanism, one principal per workload and per role (producer or consumer), literal ACLs where you can, prefixed ACLs only with explicit denies for the names they must not cover, allow.everyone.if.no.acl.found=false, and a short super user list that is reviewed like root access.

Security notes#

Security note. Never put a bearer token, session cookie or API key in an event payload or header. Provenance names principals; credentials stay on connections.
Security note. ssl.client.auth=requested lets clients without certificates connect. Use required on any listener that carries client data.
Production pitfall. A prefixed ACL written for convenience covers every future topic that shares the prefix, including dead letter, retry and audit topics created later. List the topics a prefix matches whenever a new one is created.
Security note. Kafka's authorizer decisions go to the log4j logger kafka.authorizer.logger, but, as documented, "there is no built-in tamper-evident audit trail". Ship those logs to a store you control; see Audit.

Monday checklist#

  1. List every principal on the cluster with its owner, credential type and expiry; delete principals without an owner.
  2. Confirm the authorizer is configured and allow.everyone.if.no.acl.found is false.
  3. For each prefixed ACL, list the topics it matches today; add explicit denies for dead letter and audit topics.
  4. Check that no producer and consumer share a principal, and that no consumer holds Write on the topic it reads.
  5. Grep event schemas and sample records for token-shaped values (eyJ, Bearer, key prefixes).
  6. Review the super user list and the identities that hold the provisioning credential.

Exercises#

  1. A new fraud-scoring service needs to read uw.submissions.v1. The team asks to join group uw-underwriting "to save setup". What do you answer?
AnswerOwn principal, own group

No. Joining the group would split the underwriting workflow's partitions and move its committed positions. Give the service its own principal and its own group, with Read on the topic and on that group.

  1. The OAUTHBEARER broker config sets the JWKS URL but the team adds sasl.oauthbearer.allow.unverified.audience because startup failed. What is now true?
AnswerTokens for other APIs log in

The broker accepts a token minted for any audience from that issuer, so a token stolen from another service can authenticate to Kafka. The documentation calls the escape hatch insecure; set the expected audience instead.

  1. Why does the example deny the consumer on the dead letter topic instead of changing acl-2 to literal?
AnswerBoth work; the deny shows precedence

A literal acl-2 is the cleaner fix in practice. The example keeps the prefixed allow plus an explicit deny to show how a prefix overreaches and how deny precedence (stated by Confluent, inferred from the Apache example) contains it.

Advanced perspective#

Deny precedence: stated, inferred, and testedThree different levels of evidence for one rule

Three sources give three kinds of evidence. Confluent's ACL overview states the rule in words for Confluent Platform. The Apache 4.4 page shows an example (--allow-principal User:'*' plus --deny-principal User:BadBob --deny-host 198.51.100.3 --operation Read) whose purpose only works if a deny wins, but does not state the rule. The example in this repository implements deny-wins as a model, which proves only that the model does what it says.

For a real cluster, the evidence that counts is a test against it: create a prefixed allow and a literal deny on a test topic, and confirm the client is refused. Put that test in the cluster's acceptance suite, so an authorizer change or a vendor distribution with different behavior fails loudly. The same applies to implicit Describe: the 4.4 page lists Describe among producer and consumer needs, and the model grants nothing implicitly; check your broker before relying on either reading.

mTLS or OAUTHBEARER for client identityCertificate distribution against token validation

mTLS puts identity in the transport. Strengths: no extra service in the login path once certificates are issued; the same mechanism as broker-to-broker traffic, which Kafka's security model recommends for mTLS. Costs: an internal CA, renewal before expiry on every client, principal mapping rules that turn distinguished names into short names, and revocation that is weak in practice (see TLS and PKI). Short-lived certificates with automated renewal make revocation matter less.

OAUTHBEARER puts identity in a token from the identity provider the services already use. Strengths: the same workload identity reaches Kafka and HTTP APIs; tokens expire on their own; no certificates to distribute. Costs: the broker depends on the JWKS endpoint, the audience and issuer must be configured or the broker refuses to start, and the principal name now comes from a claim, so the identity provider's naming becomes part of your ACLs. Both are credible; the choice usually follows which identity system the platform team already runs well.

Signed provenance and its costMoving trust from the topic ACL into the event

Unsigned provenance is only as trustworthy as the set of principals that can write the topic. If that set is one producer, guarded well, that is often enough. When events cross clusters (mirroring), pass through intermediate processors that rewrite them, or are kept for years as audit evidence, the ACL that guarded the original topic is no longer visible to the reader.

Signing the provenance headers (for example a JWS over a canonical form of the header set, made by the producer's workload key) lets any reader check them later without trusting the path. The cost is real: key distribution and rotation for every producer, canonicalization rules that every language implements identically, and a verification step in every consumer. Signing proves who wrote the headers; it does not prove the facts in them are correct, and it does not grant the consumer any authority. This site's recommendation: start with topic ACLs and identifier-only headers, and add signatures where events leave the topic's trust boundary.

Operating identities on a clusterRotation, super users, and what the broker does not log

Rotation differs by mechanism. SCRAM credentials can be updated with kafka-configs.sh while brokers run, but a client keeps its connection until it reconnects, so plan the overlap. Certificates rotate by issuing a new one before the old expires and reloading the client's keystore. OAUTHBEARER tokens rotate on their own; the identity provider's signing keys rotate through JWKS. Delegation tokens, documented as a lightweight complement to SASL/SSL, need delegation.token.secret.key to match across brokers and controllers.

Kafka's security model also says that any host able to authenticate on inter-broker or controller listeners "is effectively part of the cluster's trust boundary", that configuring a plugin "is equivalent to running its code", and that Kafka "does not encrypt data at rest". Those three sentences are why the provisioning identity, the super user list and the plugin directory deserve the same review as production database administrator access. Leaver handling applies too: when a team is dissolved, its principals and ACLs must go in the same change that removes its people (see Governance).

In one breath#

Kafka asks two questions in order: authentication on the listener turns a certificate, a SCRAM login or an OAuth token into a principal, and ACLs on topics, groups and transactional ids decide each operation for that principal, denying by default. A producer and a consumer are separate principals, a reader is never a writer, a prefixed ACL needs explicit denies for the names it must not cover, and a replay goes back through the producer's own validation. Events carry provenance as identifiers, never as anyone's bearer token, and the consumer decides on the record when it processes the event.

The event platform now knows which program wrote and read each event. Those programs run somewhere: the assessment service on a cloud runtime, the intake service in a container, CI pipelines that deploy them. Each needs to reach cloud APIs and read secrets such as the roof vendor's key, and the obvious way is a stored access key. How a workload gets cloud permissions without any stored key is the subject of Cloud IAM.