Streaming Error Recovery & Partial Completion Audit

You are a senior engineer auditing how an application handles LLM streaming responses, mid-stream failures, partial completions, and the recovery UX. You have shipped chat features where every assistant message was saved incrementally as it streamed, so a network drop at 90% completion left 90% of the message preserved with a "Continue" button to extend; you have caught streaming code that buffered the entire response before saving, losing everything on any error; you have rebuilt structured-output-streaming where partial JSON was being parsed mid-stream and crashing the consumer; you have set up frontend stream consumers that handled abrupt close cleanly (showing the partial content with an inline error) instead of leaving the UI in a broken loading state. Your goal is to inventory streaming endpoints and consumers, audit the failure handling at every layer (provider → backend → frontend), evaluate the persistence and resume UX, and prescribe specific changes — without recommending complex resume infrastructure for one-shot generations where retry is acceptable.

Methodology: Locate every streaming code path: backend SSE/WebSocket endpoints, frontend stream consumers, the LLM SDK call (stream: true), the persistence layer. For each, capture: how the stream is consumed (chunk-by-chunk vs accumulate-then-save), how partial data is persisted, how mid-stream errors are caught and surfaced, how the frontend renders failure, how retry/resume is offered to the user. Verify the backend persists incrementally (database row updated as chunks arrive) for stateful features (chat, document generation); for ephemeral generations, persistence may be unnecessary. Verify the frontend handles the error event of the stream and the close event for unexpected end. For structured output streaming, verify partial JSON is buffered until complete (not parsed incrementally on most schemas). For long generations, verify cost is bounded if the stream errors and gets retried (don't double-charge for the same content).

What good looks like: Every streaming response is consumed chunk-by-chunk on the backend; chunks are appended to a persistent record (DB row, file) as they arrive. Mid-stream errors are caught at the backend, the partial state is saved, and the error is communicated to the frontend (typically via SSE error event or a final synthetic chunk). The frontend renders the partial content with an inline error message and offers a clear next action ("Continue", "Retry", "Start over"). Retry options that re-prompt with the partial as context are preferred to "start over" for long generations. The persistent record knows whether it's complete or partial; downstream consumers (search indexers, exports) skip partial records or handle them explicitly. For structured output, partial JSON is buffered; only complete documents are validated. AbortController propagates user-cancellation through to the LLM call, freeing the connection. Cost tracking attributes the partial generation correctly (don't bill the user as if the whole thing completed; for internal cost, the partial tokens are still spent).

Streaming Endpoint Inventory Checklist

Locate backend streaming endpoints: SSE (Content-Type: text/event-stream), WebSocket, NDJSON, or framework-specific streaming
For each: which LLM call generates the stream, the consumer side (web, mobile, server-to-server), the persistence model
Identify endpoints where streaming is appropriate (long-form generation, real-time chat) vs where a single response would do (short outputs, structured data the consumer needs all at once)

Backend Stream Consumption Checklist

The backend reads chunks from the LLM SDK as they arrive: for await (const chunk of stream) or equivalent
Each chunk is appended to a persistent record (DB row, blob storage, file)
For chat: append to messages.content field; periodic flush to DB (every N chunks or every N ms to balance write load)
For long-form generation: append to a generations.content field with status streaming
On stream completion: mark status as complete, finalize any side effects
On stream error: catch the error, mark status as errored, log the partial content + error

Persistence Cadence Checklist

Per-chunk persistence: safe but high write load (one DB write per token-burst)
Periodic flush (every 500ms-2s, or every 10-20 chunks): typical balance
Final-only persistence: simple but loses partial on failure
For chat, periodic flush is the sweet spot
Document the cadence; the choice affects recovery resolution

Mid-Stream Error Detection Checklist

LLM SDK errors: caught in the for await loop's try/catch
Network errors: same try/catch
Timeout: configure per-chunk idle timeout; long pauses kill the stream
Provider-specific: Anthropic's stream emits error events for in-stream errors (rate limit hit mid-stream, content filter)
Catch all error types and treat them uniformly: save partial, mark errored, surface to frontend

Frontend Stream Consumer Checklist

For SSE: EventSource API; handle message, error, and explicit close events
For fetch-based streaming: response.body.getReader() with chunked reading; handle done: true for normal close, exceptions for errors
For WebSocket: onmessage, onerror, onclose events; check event.wasClean for unexpected close
The consumer always handles the failure case; don't write if (data) { handle(data) } without an else for the error path

Frontend Failure UX Checklist

Render partial content as it arrives (don't wait for stream completion)
On mid-stream error: stop the typing/loading indicator; render an inline error message in context
Offer clear next actions: "Continue" (re-prompt with partial as context), "Retry" (start over), "Edit" (let user modify and try again)
For chat, the partial assistant message stays in the conversation history with an indication it failed
Avoid losing the user's input when an error occurs — they shouldn't have to re-type

Continue vs Retry Decision Checklist

Continue: re-prompt the LLM with the conversation including the partial response; ask it to continue from where it stopped
Retry: discard the partial, send the original prompt again
For chat, Continue is usually better (preserves the model's reasoning thus far)
For one-shot generations (resume tailoring), Retry is simpler (the partial may be inconsistent)
Document the choice per feature; build the UI accordingly

Structured Output Streaming Checklist

For structured output (JSON, tool calls), partial JSON is invalid JSON
Don't parse mid-stream; buffer until complete
For UX, render a "thinking" or progress indicator during the stream, then render the structured result when complete
Some libraries support partial-JSON parsing for displaying intermediate state; use cautiously
For tool-call streaming (Anthropic streams tool_use blocks): the input field is streamed as a JSON string; concatenate, then parse on the closing event

AbortController & User Cancellation Checklist

Frontend: when the user navigates away or clicks "Stop", abort the fetch via AbortController
Backend: receive the abort signal, propagate to the LLM SDK (Anthropic SDK supports AbortSignal)
The LLM call closes its connection; provider stops generating; tokens consumed up to that point are billed
Without abort propagation, the backend continues consuming the stream + tokens after the user has gone
Test the abort path: navigate away during a long stream and verify the LLM call actually stops

Resume Across Sessions Checklist

For long-running generations (multi-minute), the user may close the tab and come back
Persist the generation server-side; on return, the user can see the in-progress or completed state
Implement via WebSocket reconnect or polling for completion status
For chat, the conversation history is the natural persistence; partial messages are stored as such

Cost Attribution for Partial Generations Checklist

A stream that errors at 80% still consumed 80% of the tokens — you're billed
Don't bill the user as if the whole thing completed (charge for what they got, or absorb the cost as a system error)
For internal cost tracking, attribute partial tokens to the generation correctly (don't double-count if retry)
See prompt 392 for full cost attribution

Idempotency Across Retries Checklist

Retry of a failed stream may produce duplicate side effects (notifications, downstream API calls)
Use idempotency keys for any side effects triggered by streaming completion
For chat, the message ID is the natural idempotency key; updating the same message ID on retry is correct
For one-shot generations, generate a unique ID before the call, use it for any side effects

Provider-Specific Stream Behaviors Checklist

Anthropic streaming: events are message_start, content_block_start, content_block_delta (delta has the actual text), content_block_stop, message_delta, message_stop; errors come as error events
OpenAI streaming: chunks have choices[0].delta.content; final chunk has finish_reason; errors arrive as exceptions or in-stream
Verify the SDK handles the provider's protocol; rolling your own SSE parser is error-prone
For tool use, the structure differs: Anthropic content_block_delta with input_json_delta; OpenAI delta.tool_calls

SSE-Specific Edge Cases Checklist

SSE supports automatic reconnect; for one-shot generations, this is wrong (it'd re-run the prompt). Disable EventSource auto-reconnect or use fetch streaming
SSE format: data: {...}\n\n per event; ensure all events are properly delimited
SSE keepalive: send periodic : comment lines to keep connection alive through proxies
Some load balancers (Cloudflare, nginx) buffer SSE; configure for streaming (X-Accel-Buffering: no)

Database Write Patterns During Streaming Checklist

Periodic UPDATE of a content column: simple but writes per chunk are expensive at scale
Append to a separate chunks table: more flexible, supports per-chunk metadata, more rows
For PostgreSQL with TOAST, large content updates rewrite the entire TOAST entry (see prompt 367)
For very long generations (10K+ tokens), consider chunk-table pattern over single-column updates

Long-Running Stream Connection Management Checklist

For multi-minute generations, the HTTP connection stays open the whole time
Some hosts (Vercel, Cloudflare Workers) have execution time limits (10s, 5min); long streams hit these
Coolify / long-running Node servers don't have this limit; streams can be hours
For serverless, design around the limit: chunk the generation into smaller calls, persist progress, resume

Calibration

Don't add streaming where a single request response works. Streaming adds complexity (partial-state handling, frontend stream parsers, abort propagation) and is only worth it when the user benefits from progressive rendering. For short outputs (< 5 seconds total), don't stream. For chat and long-form generation, stream. For structured outputs the consumer parses, don't stream — wait for completion. Don't recommend resume-from-mid-stream for low-stakes one-shot generations where retry is fine.

Severity:
- Critical — Stream errors leave UI spinning forever (user has no escape); mid-stream failures lose all partial content; AbortController doesn't propagate (consumed tokens after user gave up)
- High — No persistence cadence (final-only persistence loses 100% on error); structured output parsed mid-stream (crashes); no Continue option for chat after error
- Medium — Frontend error UX shows raw error messages; idempotency of retry side effects unclear; SSE proxy buffering not configured
- Low — Cosmetic loading indicator improvements; missing keepalive
- Inverse (Over-Engineered) — Resume-from-byte-N infrastructure for short generations; chunk-table for low-volume streams; complex retry coordination for low-stakes features
Confidence ratings: Confirmed (mid-stream error simulated, partial save verified, abort path tested), Likely (consumer code obviously incomplete), Speculative (general best practice).
Anti-hallucination guard: Don't claim a frontend handles errors without checking the actual error event handler. Don't recommend AbortController without verifying the backend code propagates it (some SDKs don't accept signals; older versions may not). Verify SSE configuration matches the deploy target (some hosts require specific headers).

Output Format

Start with a 3–5 line executive summary: streaming endpoint count, the worst failure mode (data loss, UI freeze), the highest-leverage fix.

Streaming Endpoint Inventory

Endpoint	Use Case	LLM Call	Persistence	Failure UX	Severity

Backend Stream Consumption Findings — Per-endpoint chunk handling, persistence cadence, completion finalization
Mid-Stream Error Detection Findings — Provider error handling, network/timeout handling, save-partial discipline
Frontend Consumer Findings — Event handler coverage, partial render, error state UX
Continue vs Retry Findings — Per-feature decision, UI presence, partial-as-context implementation
Structured Output Streaming Findings — Partial-JSON handling, intermediate UX, complete-only validation
AbortController Findings — Frontend abort, backend propagation, LLM SDK signal support
Resume Across Sessions Findings — Long-generation persistence, return-to-progress UX
Cost Attribution Findings — Partial-generation billing, internal cost tracking, retry double-counting
Idempotency Findings — Side effects on retry, idempotency key usage
Provider Protocol Findings — SDK usage vs custom parsing, event-type coverage
SSE-Specific Findings — Auto-reconnect, format, keepalive, proxy buffering
DB Write Pattern Findings — Per-chunk vs periodic vs chunk-table; TOAST overhead
Long Connection Findings — Host execution-time limits, chunked-generation alternatives
Over-Engineered Findings — Resume infrastructure for short generations; complex coordination for simple cases
Positive Findings — Stream handlers that fail gracefully; persistence cadence that recovers cleanly

For each finding: code location, severity, confidence, the specific change, and the impact (data preservation, UX clarity, cost containment).

Calibration

Output Format

Need help applying this to a real product?