Same model, different provider, different structured output
A technical note discusses issues with structured extraction in an LLM workflow. The same model was routed to different providers, leading to inconsistent extraction results. Key lessons include the importance of logging provider identity and considering provider choice in structured-output workloads.
- ▪The extraction layer failed to capture budget and date flexibility from user inputs.
- ▪Initial theories about prompt specificity and schema shape did not resolve the extraction issues.
- ▪Removing assistant history improved extraction success, indicating that the shape of the assistant message affected outcomes.
Opening excerpt (first ~120 words) tap to expand
Model provider variance in structured extraction May 2026 · Technical note TL;DR We spent several hours debugging what looked like a prompt-engineering issue in a structured extraction pipeline. This turned out not to be a prompt issue. The same OpenRouter model (google/gemini-3-flash-preview) was being routed to different upstream providers, and one provider consistently failed to extract certain fields when assistant history contained list-shaped content. Practical lesson: log the routed provider treat provider identity as part of the request fingerprint be careful with latency-based routing on structured-output workloads For structured-output workloads, provider choice can be part of correctness, not just latency or cost.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Guilherme Costa.
Discussion
0 commentsMore from Guilherme Costa
