GPUscale

gpuscale.net

v4.7

Local Open-Source GPU VRAM calculator for self-hosted LLM fleets.

Private by default: projects save to this browser only

Project

Inputs · work top to bottom, results update live

Use cases

Each card is one workload this project serves, with its own model, shape and concurrency. The stations below edit the selected card. Use cases that pick the same model and precision are served by one shared pooled deployment.project structure

Workload

Use-case preset

Presets fill resident sequence, reasoning mode, visible output and the three SLO targets with realistic values for the workload. Everything stays editable afterwards.

Set this only if your serving stack has automatic prefix caching on. It cuts first-token time and KV memory for the shared part; it does not speed up decode.

Reasoning mode

Thinking tokens generated before the visible answer. They add decode time and, if the toggle below is on, extend the KV footprint too.Light ≈ 2,000 tok · Heavy ≈ 8,000 tok

Reasoning tokens extend the KV cache

On for standard serving (thinking stays in context). Off if the engine discards reasoning traces from the cache after each call.

Active users at peak

Headcount actively using this workload in the busiest hour, not simultaneous requests. Little’s law turns it into the concurrent LLM calls the sizing runs on (shown underneath, and editable below). The estimator drawer refines the traffic shape: interactions per hour, calls per turn, burst margin.demand

KV residency policy

Running batch: only admitted sequences hold KV and the rest queue (the modern default). All sessions: every concurrent call keeps its cache resident (lowest latency variance, far more memory).

SLO targets

Pass/fail thresholds for the compliance panel. A target of 0 disables that check. Presets refill these with workload-appropriate values.

TTFT ≤ms

TPS ≥tok/s

P95 ≤s

Conversation length · from call duration

Average callmin

Accumulatestok/min

System + toolstok

Size for

End of call sizes every session at its longest, which is the conservative choice and the one that keeps a first-token promise on the last turn. Average in flight sizes at half that: correct for steady-state memory when call ages are spread evenly, and roughly half the KV bill.

→ 0 tokens

A conversation holds its whole transcript, so the longer the call the more KV cache each session occupies. It matters most when the KV policy is all active sessions stay in KV, because then every caller in flight holds their cache for the whole call.

Concurrency estimator · Little's law

Peak sessions

Turns / session / hr

% turns hitting LLM

LLM calls / turn

Burst factor

Call duration s

→ 0 concurrent calls

Model

Architecture geometry drives every estimate: total params size the weights, active params drive compute and decode reads, and the effective KV heads × head dim encode each model's real cache cost (MLA, sparse and SSM hybrids use sheet-compatible effective values).

Custom model · enter geometry manually

Precision

Weights

KV cache

Bytes per weight. FP8 is the 2026 serving default (half of BF16, minimal loss). NVFP4/MXFP4 are Blackwell-native 4-bit floats; Q-tiers are GGUF/llama.cpp formats.Typical production: FP8 · edge/workstation: Q4_K_M or MXFP4

Bytes per KV element. FP8 KV halves cache cost with negligible quality impact and is the vLLM production default.Typical: FP8 · long-context on tight VRAM: INT4

Hardware & resilience

GPU

Bandwidth sets decode speed, dense FP16 Tensor-Core TFLOPS sets prefill speed, VRAM sets what fits. 2026 parts (VR200, MI455X) carry pre-launch estimates.

Failure protection

What the fleet must keep serving through. Advanced mode offers the full set of patterns (N+2, mirrors, active/active splits) with their economics.resilience

Resilience model

Pick what the deployment must survive. The sizing rule: every figure in the results is what you can guarantee even during the covered failure, so throughput and concurrency never grow with resilience. The extra hardware either idles until a failure (spares, mirrors, DR standby) or serves day-to-day as burst headroom (Active/Active). The economics strip under the topology shows guaranteed vs normal-day capacity, idle hardware and the cost multiplier for the chosen model.

Auto-size memory target%

How full Auto-size may pack the serving GPUs. 80% leaves industry-standard growth headroom; 90 to 95% packs tighter at the cost of burst and growth margin (the thin-headroom warning starts at 92%). Manual sliders are never limited by this.

Uses everything above. Derives concurrency from active users, picks the smallest tensor parallel that fits one copy (widened for the TTFT target), the fewest GPU cards and the batch that admit the peak calls, then adds replicas until the speed and P95 targets pass, packs those cards onto whole nodes, and places supporting models and spares. The decision and its reasoning appear in the Summary panel. No SLO targets means offline sizing: minimal hardware, largest fitting batch, queueing allowed.solver

Fine-tune · Auto-size sets these, edit freely

Advanced tuning

Efficiency assumptions · defaults are sensible

Results · computed live from your inputs

—

Summary

Memory ledger

Time to first token

0ms

Per-user speed

0tok/s

Aggregate throughput

0tok/s

Mean user latency

SLO compliance

Deployment topology & fleet map · every model on every GPU

Resilience buys survival, not speed: the guaranteed figures are what remains during the failure you chose to cover. Idle spares add nothing day-to-day; active/active sites add burst headroom you should not plan sustained load on.

Speed per user as demand growsmore simultaneous calls = slower streams

per-user tok/s aggregate tok/s memory limit (red dashes) speed target (amber dashes) below target meets target operating point

Speed per user vs conversation lengthlonger context = slower streams

per-user tok/s speed target (amber dashes) below target meets target current context

Request latency anatomy

Reading of this configuration

Recommendations