Last winter, I helped a team building an intelligent customer service system troubleshoot a production issue. Their backend was connected to four LLM APIs simultaneously, and the frontend used SSE for streaming output. On the third day after launch, users started reporting that "answers come out like squeezing toothpaste" — sometimes an entire block of text would freeze and then suddenly gush out all at once; sometimes a sentence would be split in half with inexplicable empty chunks in between. There were no 5xx errors in the backend logs, and the availability monitoring for all the APIs was green.
This kind of problem is the hardest to diagnose because every layer looks normal. I worked with them starting from the browser Network panel, peeling back layer by layer, and ultimately found the problem wasn't in any single API — it was in the very act of "mixing four providers together."
The Conclusion First: Inconsistent Streaming Protocols Are the Hidden Pitfall of Aggregated Integration
For LLM API streaming output, the mainstream approach is SSE (Server-Sent Events). But "all using SSE" doesn't mean "all behaving the same." The four vendors each have their own implementations for chunk granularity, end markers, and heartbeat strategies. The byte streams the frontend receives all look like they start with data:, but the parsing logic is completely non-reusable. This is the root cause of the stalls and abnormal chunk boundaries.
Layer-by-Layer Investigation: From Frontend to Gateway, Where Is the Problem?
First layer: frontend. Their SSE parsing used the standard EventSource, splitting events by \n\n. That itself was fine.
Second layer: backend forwarding. The backend used Node to do a passthrough layer, piping the upstream ReadableStream directly to the response. This is where the first landmine was buried: chunk sizes vary greatly across vendors — some send per token, others accumulate a few tokens before sending. If Buffer.concat was used during piping without correctly handling UTF-8 multi-byte characters spanning chunk boundaries, you'd get half a Chinese character split apart, rendered as a garbled box on the frontend.
Third layer: timeout and retry. The backend configured a uniform 30-second timeout for all four providers. But in a streaming scenario, does "30-second timeout" mean time-to-first-byte or total stream time? They didn't distinguish, causing long answers to be cut off mid-stream, which manifested on the frontend as "stops suddenly halfway through."
Fourth layer: error codes. The four providers return different HTTP status codes and body structures for rate limiting, authentication failures, and content moderation blocks. The backend uniformly read error.message, and the result was that two providers returned a value and two returned undefined, causing the frontend to show blank error messages.
By this point the investigation made it clear: the real problem wasn't that any single API had a bug — it was that four sets of protocol semantics were being directly exposed to the business layer.
Key Differences in the Four Streaming Implementations
I put together a comparison. The data comes from each vendor's official documentation and actual packet captures; I've marked parts not disclosed officially:
DimensionVendor AVendor BVendor CVendor D
End markerdata: [DONE]finish_reason fielddata: [DONE]event: done event
HeartbeatNone~15 secondsComment lineNoneNot officially disclosed
Chunk granularityPer token2-4 tokens batchedPer tokenVariable
Error code locationHTTP status codebody.error.codeHTTP status codebody.code
Looking at this table, it's clear why the frontend parsing logic can't be reused. Three different end marker formats, two different error code locations, heartbeat strategies present in two and absent in two. If the business code does if (chunk.includes('[DONE]')) directly, it will never terminate when hitting Vendor B, and the connection hangs until timeout.
Four Design Essentials for Consolidating at the Gateway Layer
First, unify the SSE event format. The gateway normalizes all upstream events into a single structure: {type: "delta" | "done" | "error", content, code}. The business layer only recognizes this one set and no longer cares who the upstream provider is. The aggregation layer token8341 LLM API aggregation platform that we've used in our projects does exactly this — it exposes only one streaming protocol externally, and the business code doesn't need to change when switching models.
Second, converge authentication at the gateway. Four providers, four sets of keys, scattered across business configs — rotating them means changing four places. Move the keys into the gateway, and the business layer only holds one internal token. This is exactly where the value of an aggregation layer like token8341 LLM API aggregation platform lies: one key for multiple models, with key rotation and quota control all handled at the gateway side.
Third, timeouts should be configured in two segments. Separate the time-to-first-byte (TTFB) timeout from the stream idle timeout. Set TTFB shorter, say 10 seconds, for fast failure; set the stream idle timeout longer, say 60 seconds, to allow the model to think. Sharing a single 30-second value between both will inevitably cause problems.
Fourth, only retry idempotent errors. Once a streaming request has started outputting, retrying will cause duplicate content. The gateway layer must distinguish: failures during connection establishment can be retried; failures after the stream has started outputting can only be reported as errors to the frontend, letting the user manually resend. If this boundary isn't clearly drawn, you'll get "the same sentence spoken twice."
The Error Code Mapping Table Must Be Maintained Yourself
For rate limiting codes across the four providers: two return 429, one returns 200 but with an error in the body, and one returns a custom code. The gateway layer must build a mapping table that normalizes the various upstream errors into internal error codes, such as RATE_LIMIT, AUTH_FAILED, CONTENT_BLOCKED. The business layer performs degradation based on internal codes and no longer guesses at upstream semantics. There's no shortcut for this table — you have to capture packets provider by provider and cross-reference with documentation. token8341 LLM API aggregation platform provides a ready-made mapping reference in its documentation, which saved quite a bit of effort.
When Direct Connection Is Actually Simpler
I have to admit, not every scenario is worth putting a gateway in front of. If the project only connects to one model, or only does non-streaming batch inference, connecting directly to the official SDK means the least code, the shortest path, and the easiest troubleshooting. The gateway layer solves the complexity brought by "unified multi-model integration" — in a single-model scenario, introducing a gateway adds an extra forwarding layer and an extra point of failure. The criterion is straightforward: when you're connecting more than two models, or need to switch models at runtime, only then does the gateway's benefit outweigh its cost.
Final Thoughts
That incident was ultimately resolved by adding a gateway layer for consolidation. The frontend SSE parsing logic was cut from four branches down to one, and the stalls and chunk boundary anomalies disappeared accordingly. Looking back, the problem was never in any single LLM API — it was in "passing the protocol differences of multiple providers directly through to the business layer." If you're also doing multi-model integration, first align these four things at the gateway layer — streaming protocol, error codes, authentication, and timeouts — before discussing upper-layer features.
Author: Sun Haoran
Published: October 25, 2026