Prompt Caching

Overview

Prompt caching lets you avoid reprocessing the same prompt content on every request. When you send a long system prompt, tool definitions, or conversation history repeatedly, the LLM provider can cache that content and reuse it on subsequent requests — reducing both latency and cost.

The LLM Gateway supports prompt caching across all major providers, with each provider using its own caching mechanism:

Claude models

Claude models require you to explicitly mark which content blocks to cache using the cache_control field. Add cache_control with type set to "ephemeral" on any message you want cached.

1
import os
2
import requests
3
4
headers = {
5
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
6
}
7
8
system_prompt = (
9
    "You are a customer support agent for Acme Corp. "
10
    "You have access to our full product catalog, pricing, "
11
    "and policy documentation. Always be helpful and concise."
12
)
13
14
response = requests.post(
15
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
16
    headers=headers,
17
    json={
18
        "model": "claude-sonnet-4-6",
19
        "messages": [
20
            {
21
                "role": "system",
22
                "content": system_prompt,
23
                "cache_control": {"type": "ephemeral"}
24
            },
25
            {
26
                "role": "user",
27
                "content": "What is your return policy?"
28
            }
29
        ],
30
        "max_tokens": 1000
31
    }
32
)
33
34
result = response.json()
35
print(result["choices"][0]["message"]["content"])
36
37
# Check cache usage in the response
38
usage = result["usage"]
39
cache_details = usage.get("prompt_tokens_details", {})
40
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")

You can also set cache_control on tool result messages to cache tool interaction history in multi-turn agentic conversations.

Minimum cacheable prompt length

Claude only caches prompts that meet a minimum token threshold, and the threshold depends on the model. If the cacheable portion of your prompt falls below this threshold, the request is processed without caching and no error is returned.

Cache control with TTL

The LLM Gateway extends Anthropic’s native cache_control with an optional ttl field for specifying cache duration. This is a Gateway-specific parameter — Anthropic’s native API does not support it.

1
{
2
  "cache_control": {
3
    "type": "ephemeral",
4
    "ttl": "5m"
5
  }
6
}

Where to place cache_control

The cache_control field can be placed on:

• System messages — Cache long system prompts that don’t change between requests
• User and assistant messages — Cache conversation history in multi-turn flows
• Tool result messages — Cache tool call outputs in agentic workflows

OpenAI models

OpenAI models cache prompts automatically. No configuration is needed — the gateway passes your requests through and caching happens on OpenAI’s infrastructure.

1
import os
2
import requests
3
4
headers = {
5
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
6
}
7
8
# OpenAI models cache automatically — no cache_control needed
9
response = requests.post(
10
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
11
    headers=headers,
12
    json={
13
        "model": "gpt-4.1",
14
        "messages": [
15
            {
16
                "role": "system",
17
                "content": "You are a customer support agent..."
18
            },
19
            {
20
                "role": "user",
21
                "content": "What is your return policy?"
22
            }
23
        ],
24
        "max_tokens": 1000
25
    }
26
)
27
28
result = response.json()
29
30
# Cached tokens still appear in usage
31
cache_details = result["usage"].get("prompt_tokens_details", {})
32
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")

You can optionally configure cache behavior with two additional request-level fields:

Gemini models

Gemini models also cache automatically — no configuration needed.

1
import os
2
import requests
3
4
headers = {
5
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
6
}
7
8
# Gemini models cache automatically
9
response = requests.post(
10
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
11
    headers=headers,
12
    json={
13
        "model": "gemini-2.5-flash",
14
        "messages": [
15
            {
16
                "role": "system",
17
                "content": "You are a customer support agent..."
18
            },
19
            {
20
                "role": "user",
21
                "content": "What is your return policy?"
22
            }
23
        ],
24
        "max_tokens": 1000
25
    }
26
)
27
28
result = response.json()
29
30
# Cached tokens appear in usage
31
cache_details = result["usage"].get("prompt_tokens_details", {})
32
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")

Kimi models

Kimi models also cache automatically — no configuration needed.

1
import os
2
import requests
3
4
headers = {
5
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
6
}
7
8
# Kimi models cache automatically
9
response = requests.post(
10
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
11
    headers=headers,
12
    json={
13
        "model": "kimi-k2.5",
14
        "messages": [
15
            {
16
                "role": "system",
17
                "content": "You are a customer support agent..."
18
            },
19
            {
20
                "role": "user",
21
                "content": "What is your return policy?"
22
            }
23
        ],
24
        "max_tokens": 1000
25
    }
26
)
27
28
result = response.json()
29
30
# Cached tokens still appear in usage
31
cache_details = result["usage"].get("prompt_tokens_details", {})
32
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")

You can optionally configure cache behavior with the same request-level fields supported for OpenAI models:

Reading cache metrics from the response

All providers return cache usage data in the usage.prompt_tokens_details field of the response:

1
{
2
  "usage": {
3
    "input_tokens": 500,
4
    "output_tokens": 150,
5
    "total_tokens": 650,
6
    "prompt_tokens_details": {
7
      "cached_tokens": 450,
8
      "cache_creation": {
9
        "ephemeral_5m_input_tokens": 0,
10
        "ephemeral_1h_input_tokens": 0
11
      }
12
    }
13
  }
14
}

When cached_tokens is greater than zero, those tokens were served from cache and billed at the discounted cached input rate rather than the standard input rate.

Best practices

• Cache your system prompt — System prompts are the best candidates for caching since they stay the same across requests. Place cache_control on the system message for Claude, or rely on automatic caching for OpenAI and Gemini.
• Cache tool definitions — If you use the same tools across multiple requests, the tool definitions in your prompt are automatically eligible for caching.
• Order messages for maximum cache hits — Put static content (system prompt, tool definitions) at the beginning of the message array. Content before the cache breakpoint is more likely to match across requests.
• Monitor cache metrics — Check prompt_tokens_details.cached_tokens in responses to verify caching is working and estimate your cost savings.

API reference

Request

Top-level request parameters

These fields are set at the top level of the request body:

Message-level cache_control

The cache_control field can also be set on individual messages. Message-level cache_control lets you mark specific cache breakpoints — the provider caches all content up to and including the marked message. This is the recommended approach for Claude models.

Cache control object

The cache_control object has the same structure whether used at the request level or message level:

Response

Usage fields for cache metrics