The underwriting agent's prototype worked on a developer's laptop. For SUB-1004 it read the roof assessment, found the application saying 9 years and the provider saying 22, searched the guidance, and drafted a recommendation that cited R-ROOF-03. The model calls behind that draft used an API key the developer created in a personal account and exported as an environment variable. Nobody wrote that down, because it worked.
Now the agent is moving to the private controlled environment, and the review asks four questions about that one line of configuration. Whose account is billed when the agent drafts a recommendation at night? Who can read the prompts, which contain submission details and retrieved guidance? Who stops a runaway loop from spending the month's budget in an hour? And when the developer leaves, does the agent stop working?
The prototype's answers are: the developer's account, the developer and the provider, nobody, and yes. Each answer is a problem, and none of them is about the model. They are about which principal authenticates to the model provider, and with what credential. A second proposal makes it worse: "The agent already holds a delegated token for the underwriter. Send that to the model too, so the provider knows who asked." This site's own case UW-A22 shows the token service refusing exactly that request.
What authenticating to a model provider means#
A model call is an HTTP request from an application to a provider's API. The provider authenticates the caller, applies that caller's limits and model permissions, bills an account, and may log the request. The question "who is calling?" therefore decides cost, visibility and control, the same way it does for any API.
Plain English. The model provider needs to know which account to bill and which limits to apply. It does not need to know which underwriter asked, and it should never receive a credential that could act on the insurer's records.
Before AI. Backend engineers have made this decision for every shared paid service: an SMTP relay, a payment processor, a geocoding API, a database connection pool. The mature answer was always the same. One service owns the credential, applications call that service with their own identity, and the service enforces quotas and writes the bill per application. An LLM gateway is that answer applied to model calls.
The smallest primitive: one request, one credential header#
The smallest thing to look at is the request itself. Below is the documented shape of a request to the Claude API, written from its documentation read today (documented shape; not run here). It shows where the credential travels: one header. It does not show which principal the credential belongs to, which is the whole question of this page.
POST /v1/messages HTTP/1.1
Host: api.anthropic.com
Authorization: Bearer <API key, or a short-lived token from Workload Identity Federation>
anthropic-version: 2026-10-21
content-type: application/json
(body omitted: the model name and the messages)
Product (Claude API, as documented on 2026-11-18): Authorization carries "Bearer <token>, where <token> is your API key or a short-lived access token obtained from POST /v1/oauth/token through Workload Identity Federation"; x-api-key is a "Legacy fallback for Authorization, still supported"; anthropic-workspace-id is required with an API key not scoped to a workspace. Responses carry request-id, anthropic-organization-id and anthropic-workspace-id. Those response headers say what the provider can attribute a call to: an organization and a workspace. Nothing in the request names the underwriter, the agent run run-7f3a, or the submission, unless the caller puts it in the prompt.
The same call, four ways#
There are four common ways to supply that header, and each moves the secret, the bill and the logs to a different place.
1. A personal API key. A developer creates a key in their own account and the application sends it. A provider API key is a credential the model provider issues that identifies an account (or a person or service account inside it) on every request. With a personal key, the developer's identity is the caller. Product (Claude API, as documented on 2026-11-18): a personal key "acts as its user", while a service account key "keeps working if the creating user leaves". A personal key is the right tool for a developer trying a model with synthetic data, and the wrong one for anything an organization depends on.
2. A service credential. The organization creates a key or identity for the application, not for a person, and stores it in a secrets manager. The application is now the caller, and the key survives staff changes. Each application holds its own provider credential, so each one is a place the key can leak from, and the provider sees each application as a separate account or workspace. Product (Claude API, as documented on 2026-11-18): keys can be scoped to a single workspace; each workspace can have a monthly spend limit and rate limits, which "can keep a workspace below your organization's limits, but they can't raise those limits".
3. An enterprise LLM gateway with per-application identity. An LLM gateway is an internal service that every application calls instead of the provider. It authenticates each calling application with that application's own workload identity, applies the organization's model policy and budgets, logs what the organization decided to log, and calls the provider with the only provider credential. In the example that service is svc-llm-gateway, "the only holder of the model provider credential". Applications hold no provider secret at all. The gateway is a pattern, not a product or a standard; several products implement it, and teams also build their own.
4. A cloud-managed model through IAM. The model runs as a service in the organization's own cloud account, and the caller authenticates with its cloud identity (a role, a service account, a managed identity), with no stored key. Model access becomes a cloud permission like any other. Product (Amazon Bedrock, as documented on 2026-11-18): invocation "is authorized like any other bedrock:InvokeModel request, against the policies that apply to the principal making it". Product (Google Cloud, as documented on 2026-11-18): the predefined role roles/aiplatform.user is now titled "Agent Platform User" and "Grants access to use all resource in Agent Platform"; the documentation notes that predefined roles "often contain more permissions than you need". Product (Azure OpenAI, classic portal page, as documented on 2026-11-18): the Cognitive Services OpenAI User role can "Make inference API calls with Microsoft Entra ID", and roles can be assigned to a user, group, service principal or managed identity. Options 3 and 4 combine well: the gateway calls a cloud-managed model with its own cloud identity, and nothing anywhere holds a static key.
Diagram: One model call, four ways · where the secret, the bill and the prompt go
Each lane is one design for the same model call. Follow the credential: on the developer's laptop, inside each application, inside the LLM gateway alone, or nowhere because a cloud identity replaces it. Wherever the credential sits, the prompt passes through, and that is where it can be logged and where a budget can be enforced. The first lane is a labelled mistake for anything beyond a prototype.
- 1 · Personal API key
- developer's key in an environment variable: The key identifies the developer. The Claude documentation says a personal key acts as its user.
- provider bills the personal account: Spend and prompts fall under the developer's account; the insurer has no record and no limit of its own.
- 2 · Service credential per application
- app key in a secrets manager: A key for the application, workspace-scoped, read only by the application's code.
- provider workspace limits: Workspace spend and rate limits stay below the organization's limits; they cannot raise them.
- 3 · LLM gateway with per-application identity
- app authenticates as itself: The agent application calls the gateway with its own workload credential, never a provider key.
- gateway: model allowlist + budget: The gateway decides which models this application may call and whether its budget allows the call, before calling.
- only provider credential: The gateway alone holds the provider credential, or a cloud identity in place of one.
- 4 · Cloud-managed model via IAM
- cloud role or managed identity: The caller authenticates with its cloud identity; invoking a model is a cloud permission.
- model permission in cloud policy: Predefined roles may grant more than inference; scope to the models and actions needed.
The diagram puts the four designs in four lanes. Follow the credential in each: on the laptop, in each application, in the gateway alone, nowhere (replaced by a cloud identity). Then follow the prompt: wherever the credential is, the prompt passes through, and that place is where it can be logged.
Now the comparison, with the terms in hand. Model permission means the rule that says which caller may use which model; budget enforcement means a limit on spend or tokens that stops calls when reached; prompt logging means storing the text sent to and returned by the model. No prices appear here; "bears the cost" names whose account is billed.
| Question | Personal API key | Service credential | LLM gateway, per-app identity | Cloud-managed model via IAM |
|---|---|---|---|---|
| Who holds the secret | The developer | Each application (in a secrets manager) | The gateway alone (svc-llm-gateway) | No static secret: the caller's cloud identity |
| Who bears the cost | The developer's account | The organization, per workspace or account | The organization, attributed per application by the gateway | The organization's cloud account, per role or project |
| Who sees prompts and logs | The developer and the provider | The application team and the provider | The gateway's log store (as configured) and the provider | The cloud account's logs (as configured) and the model service |
| Who enforces budgets | The personal account's limits | Provider workspace limits | The gateway, per application, before the call | Cloud quotas and policies; per-application only if roles are per application |
| Model permission | Whatever the account allows | Per workspace or key | Gateway allowlist per application | Cloud policy on the model resource |
| Survives a leaver | No | Yes | Yes | Yes |
| Typical fit | Prototype, synthetic data | One application, one team | Many applications sharing models | Organization standardised on one cloud |
Read the last two columns together. They are not rivals: the gateway answers "which application, which model, which budget" and the cloud identity answers "how does the gateway authenticate without a key". The first column is fine for the first afternoon of a prototype; the second is fine for one application; once three teams share models, the questions in rows four and five decide the design.
Common confusion. "We use the cloud provider's model service, so there is no key to protect." True for the key, not for authority: the cloud role that may invoke models is now the credential, and a predefined role may allow far more than inference. Scope it to the models and actions the caller needs.
In the insurance example: the gateway holds the only key#
In the example the underwriting agent never holds a provider credential. It calls the LLM gateway at https://llm-gw.internal.example with its own workload identity, wl-underwriting-agent (spiffe://insurer.example/ns/uw/sa/underwriting-agent). The gateway decides whether that workload may use the requested model and has budget left, and only the gateway talks to the provider. The person's authority, the underwriter's delegated token, is for the insurer's own tools and records, not for the model.
The example has no gateway service, but it does have the token service and the validator, so three facts can be shown with its own code. The script below, run in a scratch copy of examples/uw-auth, asks the token-exchange stand-in for a delegated token aimed at the gateway, then checks what the gateway's first validation step would make of the agent's own workload token and of its delegated tool token. A successful run proves the example's audience rules; it says nothing about a real gateway product. The tokens are demo tokens signed with HMAC from the standard library: teaching code, so use a maintained library in production.
import agent
import demo_keys
import tokens
from clock import SimClock
NOW = SimClock().now()
dt = agent.delegated_token(SimClock())
# 1. Token exchange: ask for a delegated token whose audience is the LLM gateway.
r = tokens.token_exchange(dt["subject_token"], dt["actor_token"], tokens.LLM_GW_AUD, "uw.read", now=NOW)
print("exchange for LLM gateway:", r["status"], r["error"], "-", r["error_description"])
# 2. The agent's own workload token, issued for the LLM gateway audience.
own = tokens.issue_for("wl-underwriting-agent", aud=tokens.LLM_GW_AUD, now=NOW, seed="llm")
v = tokens.verify(own, keyset=demo_keys.keyset(), expected_issuer=tokens.ISSUER,
expected_audience=tokens.LLM_GW_AUD, now=NOW)
c = v["claims"]
print("agent workload token at LLM gateway: valid", v["valid"], "| sub", c["sub"], "| principal_type",
c["principal_type"], "| spiffe_id", c["spiffe_id"])
# 3. The delegated MCP token presented to the LLM gateway.
mcp_tok = dt["exchange"]["response"]["access_token"]
v2 = tokens.verify(mcp_tok, keyset=demo_keys.keyset(), expected_issuer=tokens.ISSUER,
expected_audience=tokens.LLM_GW_AUD, now=NOW)
print("delegated MCP token at LLM gateway: valid", v2["valid"], "| first_failed", v2["first_failed"])
exchange for LLM gateway: 400 invalid_target - no token can be issued for audience https://llm-gw.internal.example
agent workload token at LLM gateway: valid True | sub wl-underwriting-agent | principal_type agent | spiffe_id spiffe://insurer.example/ns/uw/sa/underwriting-agent
delegated MCP token at LLM gateway: valid False | first_failed aud
Line one is case UW-A22: the token service refuses to mint a delegated token for the gateway, because tokens.py lists the gateway outside EXCHANGE_TARGETS with the comment "the LLM gateway is never a delegation target". Line two is the intended design: the agent authenticates to the gateway as itself. Line three closes the other door: the delegated token the agent uses for its tools carries the audience https://mcp.uw.insurer.example, so a gateway that checks audience rejects it at the first claim. The underwriter's authority cannot leak to the model path, and the model path's credential cannot act on records. The agent here is a scripted stand-in, so these results show how the access controls work, not how good any model is.
Diagram: One model call from run-7f3a · the agent's identity to the gateway, the gateway's to the provider
Time runs down the page. The model writes the text of a request; the agent application sends it with the agent's own workload token. The gateway validates the token's audience, checks the model allowlist and budget for this application, records metadata, and calls the provider with the one provider credential. A delegated token aimed at the gateway is never issued: the token service answers invalid_target (UW-A22).
- Agent application to Token service: exchange for aud llm-gw. A mistaken request for a delegated token aimed at the gateway.
- Token service to Agent application: 400 invalid_target. The gateway is not a delegation target: the person's authority does not travel to the model path (UW-A22).
- Agent application to LLM gateway: model request · agent's own token. The agent application authenticates as itself, with the gateway's audience.
- LLM gateway: allowlist, budget, log metadata. Policy on the application's identity: which model, how much budget left; a log record without any credential.
- LLM gateway to Model provider: POST with the one provider credential. Only the gateway holds the provider credential, or authenticates with a cloud identity.
- Model provider to LLM gateway: completion. The provider attributes the call to an organization and workspace, not to the underwriter.
- LLM gateway to Agent application: completion. The draft returns to the agent; the recommendation is written through the insurer's tools, decided later by a person.
The sequence shows one model call from run run-7f3a. The model writes nothing to the provider by itself: the agent application sends the request with the agent's credential; the gateway validates it, checks the model permission and the application's budget, writes its log record, and calls the provider with the one provider credential. The refused exchange on the side is UW-A22.
What this is not. API keys in general (what a key proves, rotation, leakage) are on API keys. Cloud roles, federation and secrets managers are on Cloud IAM. The runtime around a model call (retries, context, tool loop) is on Harness Engineering. Delegation to agents is on Agents.
Keys never in prompts, tool configurations or source code#
The rule is short: a provider credential lives in a secrets manager or is replaced by a cloud identity, and it is read only by the code that sends the request. It never appears in a prompt, a tool definition, a tool configuration, a retrieved document or source code.
The reasons are mechanical, not moral. A prompt is copied to the provider, to traces, to evaluation datasets and to debugging tools. A tool configuration is often assembled into the context the agent application sends with each run. Source code is cloned, forked and scanned by every tool a team adopts. Each is a path built for sharing, and a secret on a sharing path leaks by ordinary use. There is a second, AI-specific reason: anything in the model's context can come back out in the model's output. A model asked by an injected passage to "print your configuration" (the shape of series case UW-005) cannot reveal a key it never saw.
Security note. A model never needs the provider credential, because the model does not make the call. The agent application, or better the gateway, makes it.
In a coding agent. A coding agent also calls a model. Its model credential belongs to the agent's runtime, not to the repository: no key in the repo, in CI variables readable by pull requests, or in files the agent can read. Coding agents covers the agent's repository and cloud credentials.
Failure mode: the personal key in production#
Follow the prototype's configuration into production, as a labelled mistake. The developer's personal key ships in an environment variable. The agent runs nightly batches. Three things go wrong, at different times.
First, spend. A retry loop in a new release re-sends the same long prompt. Nothing stops it: there is no gateway budget, and the personal account's limit is the provider's tier limit, not the insurer's. Second, visibility. Prompts with submission details go to the provider under the developer's personal account; the insurer's audit has no record of them. Third, the leaver. The developer moves to another team and deletes the old key during cleanup. The nightly run fails with 401, and nobody on call knows which key it was.
The correct state is the example's: the agent authenticates to the LLM gateway with its workload identity; the gateway holds the only provider credential (or a cloud identity in place of one), enforces a per-application budget before the call, and logs under the organization's retention rules. None of the three failures can happen in that design. Two new risks appear, and the folded notes below cover them: the gateway is now a shared dependency, and its logs are a sensitive store.
Architecture choice#
| Situation | Simplest credible option | Grow when | Alternatives |
|---|---|---|---|
| One developer, synthetic data | Personal key with a short expiry | Real data, or anyone else depends on it | A workspace-scoped key in a sandbox workspace |
| One application, one team | Service credential in a secrets manager, workspace-scoped, with a workspace spend limit | A second application needs models | Cloud-managed model via the application's cloud identity |
| Several applications share models | LLM gateway with per-application identity, budgets and model allowlist | Several regions, residency rules | One provider workspace per application (attribution without a gateway, weaker policy) |
| Organization standardised on one cloud | Gateway calling a cloud-managed model with its cloud identity | Models from several providers | Applications call the cloud model directly with scoped roles |
This site's recommendation: give every model call two separate identities, and never merge them. The calling application authenticates to the gateway as itself; the gateway authenticates to the provider with the one organizational credential or a cloud identity. A person's delegated authority stays with the insurer's tools and records, where the policy decision point can use it.
Products for this chapter#
Product (as documented on 2026-11-18, none run here): Claude API overview and authentication (Bearer header, x-api-key legacy, Workload Identity Federation, personal and service account keys; workspace keys "(legacy)"); Claude workspaces (workspace-scoped keys, spend and rate limits); Amazon Bedrock model access and API keys; Google Cloud access control for Agent Platform (formerly titled Vertex AI); Azure OpenAI RBAC, classic portal. Pattern: the LLM gateway. Standard (RFC 8693, token exchange): the shape the example's refusal invalid_target follows; see OAuth and OIDC. What each provider stores from a request and for how long is set out in its data-use terms, which were not read for this page.
Security notes#
Security note. Block models you have not approved where the platform allows it. Product (Amazon Bedrock, as documented on 2026-11-18): in commercial Regions most models are enabled on first use (given AWS Marketplace permissions), and "Denyingaws-marketplace:Subscribealone will not block the first model invocation"; to block access, denybedrock:InvokeModelat the organization (SCP) or account level.
Security note. Turn off or inventory key-based access where identity-based access exists. Product (Azure OpenAI classic page): "using API keys will bypass the Azure RBAC access control".
Production pitfall. A gateway that authenticates applications by a header such as x-app-id cannot attribute cost or enforce budgets: any caller can claim any application. Use the application's verified workload identity (Workload identity).
Monday checklist#
- Find every provider credential in use: who created it, personal or service, which application, expiry, last use.
- Search prompts, tool configurations, trace stores and repositories for key-shaped strings; rotate any hit.
- Replace personal keys in shared or scheduled workloads with a service credential or the gateway.
- Write down, per application: which models it may call, its budget, and who sees its prompts.
- Check whether model access in your cloud is denied by default or enabled on first use, and set a policy either way.
- Classify the gateway's prompt logs and set their retention like the data they hold.
Exercises#
- The cockpit team wants the model to "know which underwriter asked" and proposes sending the underwriter's access token to the LLM gateway. What happens in the example, and what do you propose instead?
AnswerRefused at audience; pass attribution as data
The token is for the quote API audience and the gateway rejects it; the exchange service refuses to mint one for the gateway (UW-A22, invalid_target). If attribution is needed, the cockpit application authenticates as itself and sends the underwriter's principal id as a logged attribute, never a credential.
- A team says: "We moved to the cloud model service, so budgets are handled." Are they?
AnswerOnly per role or account, not per application, unless designed
Cloud quotas and policies apply to the roles, projects or accounts calling the model. Per-application budgets need a role per application or a gateway that counts per application.
- Which of the four designs survives the developer who wrote the prototype leaving the company?
AnswerAll but the personal key
The Claude documentation says a personal key acts as its user, while a service account key keeps working if the creating user leaves; the gateway and cloud identities do not depend on one person.
Advanced perspective#
The gateway as a confused deputyOne powerful credential, many callers
A gateway that holds the organization's only provider credential is a deputy for every application. If it authenticates callers weakly, or attributes requests by a claim the caller makes, then the least trusted application can use the most expensive model under another team's budget. That is the confused deputy problem from MCP in a different place: the deputy must decide on the caller's authority, not its own.
Three rules keep it honest. Authenticate each application with a verified workload identity (mTLS with an internal CA, or a token with the gateway's audience, as in the example's line two). Make the model allowlist and the budget a policy decision on that identity, with a reason and a policy version in the decision log. And never let a caller choose the provider credential or the workspace the gateway uses: the mapping from application to provider account belongs to the gateway's configuration, under change review.
Prompt logging: audit against data minimisationLog decisions and metadata by default; content on purpose
Prompt logging pulls two ways. Investigators want the full text: what did the agent send when it drafted that recommendation? Data owners want the opposite: submission details and retrieved passages copied into one more store, readable by gateway operators, is exactly the copy problem private environments try to avoid.
A workable split: the gateway always records the decision and the metadata (calling application, model, token counts, budget state, run_id and trace_id for correlation, the policy version) and records content only for applications whose classification allows it, with the same access controls and retention as the source data. The audit record never contains the provider credential or the caller's token (UW-A24 checks the example's log for exactly that). Diagnostic traces of full prompts belong to observability and need their own redaction (Audit, and Observability Engineering for traces as a discipline).
Cloud model permissions are coarser than they lookPredefined roles, first-use enablement, key paths
Each cloud has its own surprises, all as documented on 2026-11-18. On Google Cloud the predefined roles/aiplatform.user grants use of all resources in Agent Platform; the documentation's least-privilege example is a custom role with aiplatform.endpoints.predict. On AWS, model access in commercial Regions is enabled on first use for most models, so a deny policy on bedrock:InvokeModel (by model ARN where needed) is the control, not the absence of a subscription; Anthropic models earlier than Claude Opus 4.7 still need a one-time use-case form. Bedrock also offers API keys: short-term keys inherit the permissions of the IAM principal that generated them, while long-term keys create an IAM user and are "Recommended only for exploration". On Azure, the classic RBAC page separates the role that can call inference with Entra ID from the role that can view and regenerate keys, and keys bypass RBAC.
The common lesson: a cloud identity removes the stored secret, and moves the risk into role design. Review model permissions like any other production permission, and look for the key path next to the identity path.
Workload Identity Federation to a provider APINo stored key even without a cloud-hosted model
A gateway that calls a provider's own API does not have to hold a static key either. Product (Claude API, as documented on 2026-11-18): Workload Identity Federation exchanges "your identity provider's identity token" for a short-lived bearer token through POST /v1/oauth/token, intended for cloud workloads, CI/CD and Kubernetes "where you want to eliminate static secrets"; such tokens select a workspace at exchange time, and "API keys and Workload Identity Federation grant the same access to Claude API endpoints." The pattern is the same as CI-to-cloud OIDC on Coding agents and federation on Cloud IAM: the provider trusts your identity provider's signed statement about the workload, under conditions you configure, and the residual risk moves to those conditions.
In one breath#
A model call authenticates an application or account to a provider, and that choice decides who pays, who sees the prompts and who can stop the spending. A personal key puts all three on one developer; a service credential puts them on each application; an LLM gateway with per-application identity, ideally calling a cloud-managed model with its own cloud identity, puts them where the organization can enforce and audit them. The person's delegated authority never travels to the model, and no key ever appears in a prompt, a tool configuration or source code.
The model path is settled: the agent reaches models only through the gateway, as itself. The agent's other path, to the insurer's tools, is where the underwriter's delegated authority does travel, through a shared tool server that many agents and hosts can discover. Whether discovering a tool means being allowed to call it is the question of MCP.