Fireworks AI ZDR: how to set it up without missing the exceptions

Fireworks has one of the cleaner answers to the usual enterprise question: "Do you keep our prompts?"

For normal inference on open models, it says no. Prompts and generations live in volatile memory while the request runs, then are not written to persistent Fireworks storage. Fireworks still records operational metadata such as token counts. That is the baseline.

That baseline applies to a normal inference request. Fireworks also has a Responses API that stores conversations by default, prompt caching that affects tenant isolation, and a proprietary function-calling model with a 30-day logging exception. Data residency is an Enterprise setting with a US-only restriction today, and switching it on can leave an old deployment running, billable, and unable to serve requests.

For the cross-provider comparison, see Zero Data Retention (ZDR) for LLM Providers.

What Fireworks means by ZDR

Fireworks' published policy applies to prompts and generations sent to open models. It says they are not logged or stored unless the customer explicitly opts in. During an ordinary request, the content exists in memory because inference cannot happen without it. Prompt caching may keep the prompt and its KV cache in volatile memory for longer. Fireworks says neither case writes the content to persistent storage.

The company still keeps metadata needed to deliver the service, with request token count as its example. That is a different category of data. It is also why a team should avoid saying "nothing is retained" when it really means "prompt and output content are not persistently stored on this inference path." Fireworks' ZDR policy spells out that boundary.

The policy is about content retention. It does not mean the model runs in your own network. Serverless inference is shared infrastructure. Dedicated deployments are private to your account and logically isolated from other customers, but Fireworks still operates them. Choose dedicated capacity for performance, isolation, model control, or deployment requirements. Do not choose it because you think it silently changes the ZDR policy. The model and deployment docs make that distinction.

Start with the API path, not the provider name

For a stateless chat or completion request, Fireworks' ordinary open-model inference path is the straightforward one. For a chat product using the Responses API, read the next section before shipping anything.

The Responses API keeps conversation state for you. With store=True, which is the default, Fireworks retains the full user prompt, model response, and any tools the model called. The retention period is 30 days.

That is a useful product feature. It lets a later request reference previous_response_id instead of resending the entire conversation. It is not a strict ZDR path.

For a sensitive request, set the choice in code rather than relying on somebody remembering it:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.environ["FIREWORKS_API_KEY"],
)

response = client.responses.create(
    model="accounts/fireworks/models/<approved-open-model>",
    input="<application input>",
    store=False,
)

store=False means Fireworks cannot retrieve the earlier conversation through previous_response_id. Your application has to hold that history if it needs it. For regulated workloads, that is often the better place for the decision: you choose the database, encryption, access rules, and deletion schedule instead of accepting a provider-side 30-day store.

The Responses API documentation also says stored responses can be deleted by response ID. I would treat deletion as recovery for an accidental store=True, not as the design for a ZDR workload.

The setup I would use for sensitive inference

The configuration belongs in code and account settings, where a normal review can see it.

1. Keep the approved path narrow

Use an allowlist of open model IDs for sensitive traffic. Keep FireFunction out of it. Fireworks says that its proprietary FireFunction model logs inputs and outputs for 30 days for bulk analytics. That exception applies to FireFunction, not to function calling in general, but it is enough reason not to let a model alias drift into production without review.

Do the same for FireOptimizer and any other advanced feature that asks to collect examples. Fireworks says customers can opt into prompt and generation logging for advanced features such as FireOptimizer. That may be exactly what a tuning project needs. It belongs in a separate account, project, or at least a separate approval path from production inference. Fireworks lists both exceptions in its model overview.

2. Use a service account, not a personal key

Create a service account for the deployed application and give it the least capable role that still works. Fireworks documents an inference-user role that can view resources and run inference but cannot create or modify them. This makes it harder for a production key to create datasets or fine-tuning jobs by accident, and it leaves a clearer identity in audit logs.

firectl user create --user-id production-inference --service-account \
  --role=inference-user

firectl api-key create --service-account production-inference

The account administrator creates the service-account key. Keep the key in the application's normal secret store. Do not put it in a frontend, notebook shared outside the production team, or a support ticket. Fireworks' service-account guide has the role details.

3. Make the Responses API default impossible to miss

If the application calls client.responses.create, wrap it. Do not let every feature team hand-write the call.

def create_sensitive_response(client: OpenAI, model: str, input: str):
    return client.responses.create(
        model=model,
        input=input,
        store=False,
    )

Then add one integration test that inspects the request your application sends, or a small adapter test that fails if store is missing or true. The code is boring. That is what makes it durable.

Prompt caching needs a tenant boundary

Fireworks enables prompt caching for every model and deployment. It reuses an exact matching prefix, so a static system prompt, tool definitions, or shared context can be processed once and reused. Cached prompts usually stay in memory for at least several minutes. Depending on the model, load, and deployment, they may remain for hours. The generated response is not cached for later users.

That usually fits a ZDR policy because the cache is volatile rather than persistent. It still creates a design choice for a multi-tenant dedicated deployment.

Serverless caches are separated by Fireworks account. Dedicated deployments share a cache across requests by default. Fireworks says output privacy is preserved, but a caller may infer that a prompt prefix is cached from response time. For full separation, pass x-prompt-cache-isolation-key or prompt_cache_isolation_key.

response = client.chat.completions.create(
    model="accounts/fireworks/models/<approved-open-model>",
    messages=messages,
    extra_headers={
        "x-prompt-cache-isolation-key": tenant_cache_key,
    },
)

Use an opaque, stable tenant value for tenant_cache_key. Do not put a patient name, email address, or raw customer ID there. The header may be harmless to Fireworks' content-retention policy and still be captured by a gateway or tracing system you operate.

Cache isolation and sticky routing solve different problems. With several replicas, Fireworks lets you send user or x-session-affinity so requests from one user or session land on the same replica and get better cache hits. That value is a routing hint, not a privacy boundary. Use the isolation key when requests must not share cache state. The prompt-caching guide explains both controls and the timing side channel.

There is a small performance cost to isolation because you intentionally give up some reuse. That is the right trade when tenant boundaries matter.

Data residency is not a region flag

Fireworks has two related concepts that are easy to mix up.

Deployments can run in GLOBAL, US, EUROPE, or APAC multi-regions, and there are individual regions for some hardware. That is a placement choice. The regions guide says multi-region deployments are the default.

Data residency is an Enterprise account policy that enforces a region for every API key on the account. Today, the documented restriction is US. It requires the us.api.fireworks.ai endpoint and US-served models for serverless inference. Dedicated deployments must be in US or in one of its single US regions.

Before an account admin enables it, do this in order:

1. Inventory every deployed client and model ID. Change serverless clients to the regional endpoint and a regional model first.
2. List dedicated deployments with firectl deployment list. Recreate anything outside the target region before saving the policy.
3. Set the policy in the console under Settings → Governances → Data Residency, or run:

firectl policy residency set US
firectl policy residency get

4. Send a real production-shaped request through each client after the setting is on. The policy rejects requests that do not match the selected region.

There is an uncomfortable operational detail here: Fireworks does not move, stop, or check an existing dedicated deployment when you enable residency. An old out-of-region deployment can keep running and costing money while the policy rejects requests to it. Replace it first, then remove the old deployment after the cutover.

Regional restrictions also block training, FireRouter, and BYOC while they are set. FireRouter can send a request to a third-party provider, and BYOC runs in the customer's cloud account, so Fireworks cannot enforce a region for either. A team cannot treat residency as an inference-only setting. Fireworks' data-residency guide has the current restriction, endpoint, migration behavior, and limitations.

Security claims and the contract are different things

Fireworks documents TLS 1.2+ in transit, AES-256 at rest for stored workflow data, logical isolation for dedicated workloads, SOC 2 Type II, ISO 27001, ISO 27701, ISO 42001, and HIPAA support. Those claims are useful inputs to vendor review. They do not establish that a particular application is HIPAA compliant.

For PHI or another regulated data class, get the actual agreement, confirm whether a BAA is available for the account and products you plan to use, and review the deployment with the people responsible for compliance. The public documentation does not make those decisions for you. It also says that Fireworks does not currently offer client-side encryption or customer-managed keys for data stored at rest. Its data-security documentation and encryption FAQ are useful places to start.

Training, datasets, and files are another review

ZDR for inference does not turn a training workflow into a no-storage workflow.

If you upload a dataset, create a fine-tuning job, keep checkpoints, or run evaluation data through the platform, you are using stored resources. Fireworks documents encrypted storage for models, datasets, LoRA adapters, and other stored resources. It also documents customer-controlled bucket integrations for datasets and models in some workflows. Those can be useful controls. They are not the same as ordinary transient inference.

Do not put production PHI, financial records, or sensitive legal material into a training dataset because the normal inference endpoint has ZDR. Read the retention rules for the exact training surface, decide whether customer-controlled storage is required, and give that workflow its own review. Fireworks' data-security documentation is the right starting point.

The same caution applies to any evaluation or observability product that wants example prompts. It may be useful. It has its own data flow.

Workflows that are outside a strict ZDR path

This is the line I would draw in a production design: the normal inference endpoint can handle a reviewed sensitive request. Anything that creates a named resource gets a separate review.

The Batch API is a good example. It is useful for large, non-urgent workloads, but it is not a way to run sensitive prompts through a no-storage path. The JSONL input, output, error file, and any continuation lineage all need ownership and deletion rules.

If training is required, Fireworks documents a separate secure-training path. Bring-your-own-bucket workflows keep training data in a customer GCS or S3 bucket; the Training API receives tokenized batches rather than a raw dataset file. Neither statement makes training equivalent to stateless inference. Checkpoints, traces, metadata, and the rest of the job still need their own review. Fireworks' secure-training documentation explains the different surfaces.

Audit evidence is useful, but it is not prompt evidence

Enterprise accounts can use Fireworks audit logs to review storage read, write, and delete activity. The logs include account context, and the CLI can filter by resource, user, event type, and API key. That makes them useful for answering questions such as: who created a dataset, who changed a deployment, and which service account used a key?

firectl audit-logs list \
  --start 2026-09-01 \
  --end 2026-10-01 \
  --filter 'resource:"production"' \
  --no-paginate \
  -o json

The end date resolves to the start of that UTC day, so use the day after the last day you need. That is a small detail, but it matters when an audit request asks for a complete range. The audit-log guide also notes that an empty page is not necessarily the end of a paginated result set.

Do not claim that audit logs prove a prompt was never retained. The public documentation describes account and storage activity, not a prompt-content proof. Use audit logs as one piece of evidence alongside the documented endpoint behavior, code review, account settings, and your own logs.

The copies outside Fireworks still matter

A prompt can be kept before it reaches Fireworks or after the response returns:

• application and reverse-proxy logs
• retry queues and dead-letter messages
• RAG source documents, chunks, and embeddings
• prompt tracing, analytics, and evaluation tools
• chat-history tables
• external tools called by the model

Draw one request from the user to the model and back. Write a retention rule next to every system that sees the content. That exercise often finds more data than the provider review does.

The sentence I would put in the architecture decision

This service uses Fireworks open-model inference. Sensitive Responses API calls set store=False. FireFunction and content-logging features are not approved for this data class. Prompt caching is accepted as volatile memory; dedicated multi-tenant deployments use an isolation key. Training and evaluation data are reviewed separately. Conversation history, logs, queues, tracing, and tool calls have their own retention rules.

That is the practical version of "Fireworks has ZDR." It names the endpoint, the defaults we changed, and the places where the answer would change again.