Jun 23, 2026 路 1 min read
Shipping Inference server to production
Why the boring 20 percent (deploys, timeouts, backpressure) is the part that ships.
The demo worked on the first try. Production took the next three weeks.
What the demo skipped
- Timeouts and retries under real network jitter.
- Backpressure when the queue outruns the model.
- A deploy story that does not drop in-flight requests.
The fast path is easy. The failure paths are the product.
