API 504 Gateway Timeout: diagnose the upstream delay

What does HTTP 504 prove?

RFC 9110 defines 504 Gateway Timeout narrowly: a server acting as a gateway or proxy did not receive a timely response from an upstream server. The status code identifies the layer that gave up waiting. It does not identify the slow component.

Run the smallest useful test

Check authentication separately before repeating a slow generation. The model-list route is cheap and does not ask an inference model to generate output.

: "${MONKEYAI_API_KEY:?Set MONKEYAI_API_KEY first}"
curl -i https://monkeytheme.com/v1/models \
  -H "Authorization: Bearer $MONKEYAI_API_KEY"

A 200 JSON model list confirms that this key can reach the model-list route. It does not guarantee that every model or inference route is healthy. A 401 is an authentication failure, not an upstream timeout.

Next, retry the failed workload once with a shorter prompt and a small output limit. Keep the same model and endpoint so you only change one variable. If the small request succeeds, save both timings; that is useful evidence, not proof that request size caused the original 504.

Capture status, headers and timing in one command

This command keeps the response headers and body in separate temporary files. It also prints connection time, time to first byte and total time, which makes a support report much more useful than a screenshot of "504."

: "${MONKEYAI_API_KEY:?Set MONKEYAI_API_KEY first}"
workdir=$(mktemp -d)
curl -sS \
  -D "$workdir/headers.txt" \
  -o "$workdir/body.txt" \
  -w 'status=%{http_code}\nconnect=%{time_connect}s\nfirst_byte=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  https://monkeytheme.com/v1/models \
  -H "Authorization: Bearer $MONKEYAI_API_KEY"
printf 'Saved to %s\n' "$workdir"

If body.txt contains JSON, preserve its error type and message after removing private data. If it contains an HTML gateway page, record that fact too; an HTML 504 and a structured JSON API error can come from different layers. Do not publish headers.txt without checking it for cookies, trace data or other sensitive values.

Capture evidence before retrying again

Why streaming can still time out

Streaming lets the client process output while generation continues, but it does not remove proxy timeouts. Nginx documents proxy_read_timeout as the allowed gap between successive reads from upstream, not a limit on the complete response. Its documented default is 60 seconds.

If the upstream sends nothing before that gap expires, the proxy can close the connection. Once a stream has already sent data to the client, a proxy cannot safely replay the same response as though nothing happened.

Production-anchored case study (August 7–8, 2026)

The following signals come from a real journalctl -u sub2api --since '24 hours ago' sweep on the gateway host. Numbers are not staged or fabricated; they reflect one stress-test model (gpt-5.6-sol) hitting the same scenario repeatedly. The lesson is not "the gateway is broken" — the lesson is the exact log shape that distinguishes a 502-from-upstream from a 503-from-no-available-accounts.

Three error fingerprints from the same window, all anonymized to internal model gpt-5.6-sol and account id 16:

# (1) Upstream returned 401 → gateway switched accounts (502 surfaced after max_switches exhausted)
... openai_chat_completions.upstream_failover_switching {
  "model": "gpt-5.6-sol", "stream": true,
  "account_id": 16, "upstream_status": 401,
  "switch_count": 1, "max_switches": 10
}

# (2) Final 502 to the client after upstream exhausted switches
... http request completed {
  "path": "/v1/chat/completions", "method": "POST",
  "status_code": 502, "latency_ms": 570,
  "account_id": 16, "platform": "openai", "model": "gpt-5.6-sol"
}

# (3) Faster path: gateway had no available accounts at all → 503 in 13–25 ms
... openai_chat_completions.account_select_failed {
  "model": "gpt-5.6-sol", "stream": true,
  "error": "no available accounts", "excluded_account_count": 0
}
... http request completed {
  "status_code": 503, "latency_ms": 15
}

Retry without multiplying work

1. If the response includes Retry-After, honor it.
2. Otherwise wait briefly, then use bounded exponential backoff with jitter.
3. Retry a failed generation only when duplicate work is acceptable or you have an idempotency strategy.
4. Stop after a small number of attempts and send the evidence above to support.

OpenAI’s public error guide documents retry advice for temporary 503 conditions and its rate-limit guide covers Retry-After and backoff. It does not guarantee that a 504 response includes Retry-After.

Operator checks

If you operate the relay, correlate the client timestamp with reverse-proxy and application logs. Check the upstream address, connect time, time to first byte, response status, and whether any bytes reached the client.

Raising every timeout globally is a blunt fix. Confirm which timer fired and which route needs more time before changing production limits.

Sources

• RFC 9110 §15.6.5: 504 Gateway Timeout
• OpenAI API error codes
• OpenAI rate limits and retry guidance
• OpenAI streaming responses
• OpenAI background mode
• OpenAI Python library retry behavior
• Nginx HTTP proxy module
• Sub2API edge-security and long-lived streaming baseline

FAQ

Related guides