Prompt caching

Models reuse the work done on a prompt prefix they have recently seen, so a request that starts like an earlier one (the same system prompt, the same document, the same conversation so far) gets its first token sooner. Cached input tokens are reported in the response’s usage (prompt_tokens_details.cached_tokens). Caching is automatic and best-effort: a prefix stays cached only while there is room for it.

Your cache is your own

Every account has its own cache. A prefix one account sent is never reused for another account’s request, and no other account can observe what yours has cached. All of your API keys share your account’s cache, including keys you create after rotating an old one.

Splitting your cache further: `cache_salt`

If you serve several end customers through one account, you can keep their prefixes apart too. Send a cache_salt string with the request: requests reuse each other’s prefixes only when they carry the same cache_salt. A salt is always scoped to your account, so two accounts that pick the same value still never share a cache.

Any non-empty string; an empty string or no cache_salt means your account’s default cache.
Use a stable value per partition, such as your own customer id. Changing it starts that partition from an empty cache.
Supported on /v1/chat/completions, /v1/completions, /v1/responses and /v1/messages.

from openai import OpenAI

client = OpenAI(base_url="https://api.primalabs.ai/v1", api_key="YOUR_API_KEY")

client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"cache_salt": "customer-1234"},
)

from anthropic import Anthropic

client = Anthropic(base_url="https://api.primalabs.ai", api_key="YOUR_API_KEY")

client.messages.create(
    model="MODEL_ID",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"cache_salt": "customer-1234"},
)

curl https://api.primalabs.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role": "user", "content": "Hello"}],
    "cache_salt": "customer-1234"
  }'

A cache_salt that is not a string is rejected with a 400.

Get started

Guides

Your cache is your own

Splitting your cache further: `cache_salt`

​Your cache is your own

​Splitting your cache further: cache_salt

Your cache is your own

Splitting your cache further: `cache_salt`