API 502 Bad Gateway: separate upstream failures from timeouts

What HTTP 502 actually means

RFC 9110 defines 502 Bad Gateway: a server acting as a gateway or proxy received an invalid response from an upstream server. The status code identifies the proxy layer that surfaced the failure, not the upstream that produced it. An upstream here can be the model provider, a CDN in front of it, an internal service, or the OpenAI-compatible relay in front of the public route.

The article API 504 Gateway Timeout covers the no-reply-in-time case. This page covers the bad-reply case. Both come from the same family of proxy-failure status codes, but they need different diagnostics.

502 vs 504 vs 503: read the latency

You can usually tell them apart with the client-side latency_ms and the response body. The numbers below are drawn from a real journalctl -u sub2api --since '24 hours ago' sweep on the gateway host, anonymized to model gpt-5.6-sol and account id 16. They are not staged.

If latency is under 30 ms with a 503, the relay never reached upstream — it is an operator-side account-pool exhaustion, not a network problem. If latency is 100-1500 ms with a 502, the relay did reach upstream and the upstream replied badly. Mixing the two produces the wrong fix.

Run the smallest useful test

Check authentication separately before assuming the upstream is the problem. The model-list route is cheap and does not call an inference model.

: "${MONKEYAI_API_KEY:?Set MONKEYAI_API_KEY first}"
curl -i https://monkeytheme.com/v1/models \
  -H "Authorization: Bearer ***"

A 200 JSON model list confirms that the key reaches the model-list route. A 401 is an authentication failure, not a 502. A 404 means the route is wrong, not the upstream.

If /v1/models returns 200 but a specific chat call still returns 502, the failure is on the inference path, not on the relay or your credentials. Reduce the test to a short prompt with a small output limit and reuse the same model id so you only change one variable at a time.

Capture status, headers and timing in one command

Keep the response headers and body in separate temporary files. Print connection time, time to first byte and total time. A 502 with body text is much more useful than a 502 with only a status code.

: "${MONKEYAI_API_KEY:?Set MONKEYAI_API_KEY first}"
workdir=$(mktemp -d)
curl -sS \
  -D "$workdir/headers.txt" \
  -o "$workdir/body.txt" \
  -w 'status=%{http_code}\nconnect=%{time_connect}s\nfirst_byte=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  https://monkeytheme.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'
printf 'Saved to %s\n' "$workdir"

If body.txt contains JSON, preserve its error type and message after removing private data. If it contains an HTML gateway page, record that fact too; an HTML 502 from a CDN edge and a structured JSON 502 from the relay describe different layers. Do not publish headers.txt without checking it for cookies, trace data or other sensitive values.

What production logs look like for 502

The following log shape comes from the same 24-hour window as the latency table. Every 502 in that window was preceded by an upstream_failover_switching warning with upstream_status of 401 and a single account id (16) in the field. That pattern is what tells you this is an upstream-replies-badly scenario, not a no-reply-in-time scenario.

# (1) Upstream returned 401 to the relay, relay switched accounts
... openai_chat_completions.upstream_failover_switching {
  "model": "gpt-5.6-sol", "stream": true,
  "account_id": 16, "upstream_status": 401,
  "switch_count": 1, "max_switches": 10
}

# (2) Relay exhausted its account pool, no fallback survived
... openai_chat_completions.account_select_failed {
  "model": "gpt-5.6-sol", "stream": true,
  "error": "no available accounts", "excluded_account_count": 1
}

# (3) Final 502 returned to the client after ~448 ms
... http request completed {
  "path": "/v1/chat/completions", "method": "POST",
  "status_code": 502, "latency_ms": 448,
  "account_id": 16, "platform": "openai", "model": "gpt-5.6-sol"
}

Three signals distinguish a 502-from-upstream-bad-reply from a 503-from-no-accounts at a glance:

Retry without multiplying work

A bounded retry on a 502 is reasonable when the bad upstream is transient and the relay can hop to a healthy account. It is not reasonable when every retry hits the same account or the same upstream path.

1. Confirm the response includes no Retry-After header before you ignore it; if it does, honor it.
2. Use bounded exponential backoff with jitter; one retry is usually enough to distinguish a real outage from a one-off blip.
3. If the response body is identical across two retries, stop — a third retry will reproduce the same failure.
4. Send the evidence captured above to support. A redacted request_id and the model id are usually enough for the relay operator to find the corresponding upstream log line.

Operator checks

If you operate the relay, correlate the client timestamp with reverse-proxy and application logs. Check the upstream address, connect time, time to first byte, response status, the TLS handshake, and whether any bytes reached the client.

Raising every timeout globally is a blunt fix. Confirm which layer the 502 came from before changing production limits; a CDN-level 502 will not improve by raising the application read timeout.

Sources

• RFC 9110 §15.6.3: 502 Bad Gateway
• RFC 9110 §15.6.5: 504 Gateway Timeout
• OpenAI API error codes
• OpenAI rate limits and retry guidance
• OpenAI Python library retry behavior
• Nginx proxy_next_upstream

FAQ

Related guides