Why You Need a Smoke Test Before Switching LLM Providers
Sending one prompt isn't enough before swapping LLM providers. A 10-minute smoke test covering timeouts, streaming, rate limits, and tool-call retry safety.
The author describes moving away from a trivially simple LLM provider migration test — swap the base URL, send one plain prompt, confirm a response — after realizing it says nothing about production behavior. A happy-path request never reveals timeout handling, retry safety, streaming completion signals, rate-limit response shapes, usage metadata, or tool-call error classification.
Instead, before trusting a new provider in anything production-like, they now run a five-part, roughly 10-minute smoke test: log the full response envelope (ID, finish reason, usage) on a normal request; force an aggressive timeout to see whether the failure is cleanly classifiable as retryable or not; check whether a streamed response ends with a proper finish reason rather than just stopping; classify rate-limit versus token-limit failures separately, since shrinkable context and non-shrinkable tool-result payloads need different handling; and finally test the retry policy path for tool-call-shaped requests where a side effect (like sending an email or charging a card) may already have fired.
The goal isn't ranking providers as better or worse — it's confirming that the application can actually understand what happened after a failure. This matters especially for agentic or workflow-driven systems, where mistaking a partially streamed response for a complete one, or silently truncating oversized context, can produce a confidently wrong answer instead of a clean retry. The takeaway for engineers is to test failure classification and retry logic, not just the happy path, before swapping LLM providers.