Presets fill resident sequence, reasoning mode, visible output and the three SLO targets with realistic values for the workload. Everything stays editable afterwards.
Thinking tokens generated before the visible answer. They add decode time and, if the toggle below is on, extend the KV footprint too.Light ≈ 2,000 tok · Heavy ≈ 8,000 tok
On for standard serving (thinking stays in context). Off if the engine discards reasoning traces from the cache after each call.
Running batch: only admitted sequences hold KV and the rest queue (the modern default). All sessions: every concurrent call keeps its cache resident (lowest latency variance, far more memory).
Concurrency estimator · Little's law
→ 0 concurrent calls
Bandwidth sets decode speed, dense FP16 Tensor-Core TFLOPS sets prefill speed, VRAM sets what fits. 2026 parts (VR200, MI455X, Rubin CPX) carry pre-launch estimates.
Not sure what to pick? This sets Tensor parallel to the smallest slice count that fits one copy of the selected model, then adds workers until the peak concurrent calls are admitted at the current batch. Adjust freely afterwards.
Performance is sized on the N load-bearing workers; redundancy adds procurement, not throughput. N+1 survives one node failure. N+N duplicates the system in-site. DR mirrors it to a standby site (active/passive). Active/Active runs two live sites behind global load balancing, each able to carry the full load; add N+1 per site to also absorb node failures after a site loss. N+N + DR runs N+N in each of two active/active sites (4N), the heaviest pattern.