The GPUscale manual

Everything the studio does, why it does it, and the arithmetic underneath. Read section 1 to get a fleet on screen in five minutes; read section 5 if you need to defend a number to somebody who signs the purchase order.

Quick start

GPUscale answers one question: what hardware does this LLM workload need, and will it keep the promises you made about latency? It is a single static page. There is no account, no backend and no upload; every number is computed in your browser from closed-form formulas you can audit in assets/app.js.

1. Describe a use case, not a machine. Pick a workload preset (or type the numbers), pick a model, say how many people use it at peak. Leave the hardware alone for now.
2. Pick one GPU for the project in Hardware & resilience, and how many of them sit in a node (8 for an HGX box, 72 for an NVL72 rack).
3. Press Auto-size. The solver chooses tensor-parallel width, replica count and batch size, then packs the cards into nodes and applies your resilience pattern.
4. Read the verdict bar, then the fleet map. If something is red or amber, the Recommendations at the bottom say what to change and, where possible, offer a button that changes it and re-solves.
5. Add more use cases. Two cards on the same model and precision are served by one pooled deployment sized for their combined load, which is how real platforms are built.

https://gpuscale.net/#p=t:name=Northwind+Health+AI+platform;gpu=H200+141GB+NVL;perw=8;
  resil=n1;uc=Clinical+knowledge+assistant;model=Llama+3.3+70B;quant=FP8;kv=FP8;
  preset=Clinical+knowledge;users=1500;uc=Document+intake;model=Qwen3+32B;quant=FP8;
  kv=FP8;preset=Document+Q&A;users=400;uc=Coding+agent;model=Qwen3+32B;quant=FP8;
  kv=FP8;preset=Code+agent;users=250;uc=Voice+triage;model=Qwen3+8B;quant=FP8;kv=FP8;
  preset=Voice+agent;users=300

How the studio thinks

Three ideas explain almost every number on the page. If you understand these, the rest of the manual is reference material.

Three currencies, spent by two different phases

A GPU sells you three things, and an LLM request spends them in two distinct phases. Sizing goes wrong when people track only the third one.

Diagram 1. The two phases have opposite bottlenecks, which is why one number never describes a deployment.

One model call, never one task

Every workload field in the studio describes a single call to the model. This is the single most common way to get a sizing wrong by an order of magnitude.

Resident sequence means the tokens the server is holding for that request right now: the system prompt, the retrieved passages, the conversation so far, the tool traces. Not the model's maximum context, which is a capability, not a workload.

Use cases become pools, pools own cards, nodes are bought whole

A project is a list of use cases. Two use cases that run the same model at the same weight and KV precision with the same KV policy are served by one deployment sized for their combined load, exactly as they would be in production. That merged deployment is called a pool.

Diagram 2. The order of operations that turns demand into a bill of materials.

Three consequences worth remembering:

• Pooling is not an optimisation you opt into. It is what your serving stack would do anyway. If you need two separate deployments of the same model, tick Isolate on the use case.
• The pool is sized for the combined concurrency of its members, and each member is then evaluated against its own SLO targets at the pool's shared batch size.
• Only the fleet is rounded to whole nodes, once, at the end, after supporting models and co-residency have been placed.

The interface, control by control

Inputs run down the left column in the order you should work: Use cases, then Workload, Model, Precision, then Hardware. Results on the right recompute on every keystroke. Nothing is ever "applied" or "submitted".

The project bar

Use cases

Each card is one workload with its own model, tokens, concurrency and SLO targets. The card shows its model, precision, derived concurrency, and chips for any supporting models attached to it.

• + Add clones the current card as a starting point.
• The copy and × icons duplicate and delete.
• Click the name to rename it. The name is what appears in the fleet map, the charts and the report.
• Support chips (Embeddings, Reranker, Guard, ASR, TTS, OCR) attach automatically from the preset. Click one to inspect or change the model behind it; click × to detach it.
• A pooled badge means this card shares a deployment with another card. Hover it to see which.
• Isolate (in the card menu) forces a card into its own pool even when it could share.

Workload

Start from a preset: 24 of them, each with SLO targets traced to published production guidance (section 11). A preset only prefills the fields; edit anything afterwards.

Each preset also suggests a model, marked suggested for this preset next to the model dropdown. It is a starting point of the right size class and modality for that workload, not a ranking, and every one is checked at build time against the preset's own first-token and speed targets so selecting a preset can never leave you looking at a configuration that is red on arrival. The moment you pick a model yourself the suggesting stops for that card, and later preset changes leave your choice alone.

Conversation length: how long is the call?

A conversation holds its own transcript, so the resident sequence grows with the length of the call. A two-minute IVR interaction and a twenty-minute contact-centre call are not the same workload, and sizing both at one fixed number is how a voice fleet arrives short.

It matters most when the KV policy is all active sessions stay in KV, which is the correct policy for a live voice path: every caller in flight holds their whole cache for the whole call, so call length multiplies straight into the memory bill.

The drawer opens by itself for a pinned-session workload and stays shut otherwise.

Nothing here changes your sizing until you press Apply to resident sequence. Presets that describe a session (the two voice presets and contact-centre assist) prefill the drawer with the shape their own resident figure was built from, so 4,096 for the voice agent stops being an assertion and becomes a 3,100-token system prompt plus five minutes of conversation. Change the five and press Apply.

Model

The dropdown carries total and active parameters, hidden size, layers, KV heads, head dimension and maximum context for each entry, plus the architecture class and who published it.

Active parameters is what decides speed and first-token time. For a dense model it equals the total; for a mixture of experts it is much smaller, which is why a 671B MoE can decode faster than a 70B dense model while needing ten times the memory.

Custom model unlocks seven fields (params, active, hidden, layers, KV heads, head dim, context) for anything not in the library. Everything else in the studio works identically; exports carry the geometry so the file stays readable later. While it is on, the dropdown is replaced by a line naming the geometry actually in force, rather than greyed out still showing a library model that is not being used.

Precision

Weights and KV cache are quantized separately, because they trade against different things. Weight precision sets the fixed cost per replica and the bytes re-read per decode step. KV precision sets the per-token cache cost, so it scales with concurrency and context.

19 weight formats from FP32 down to 2-bit GGUF K-quants, and five KV formats. FP8 KV halves the cache against BF16 with negligible quality impact and has become the production default; the studio says so in the recommendations when you leave a KV-dominated pool at BF16.

Hardware and resilience

GPU is a project-wide choice: 37 accelerators with dense (not sparsity) TFLOPS, real HBM bandwidth, VRAM and TDP. Mixing GPU types in one project is deliberately not modelled, because in practice a serving pool is homogeneous.

GPUs per node is your chassis: 8 for an HGX or MI300X box, 4 for many OEM servers, 72 for an NVL72 rack. It matters for two reasons: nodes are bought whole, and a tensor-parallel group wider than one node pays a cross-node interconnect penalty.

Resilience model chooses among 12 patterns, from bare N to N+N with disaster recovery (section 9).

Auto-size memory target is the fraction of VRAM the solver is allowed to plan into, 80% by default. The remainder absorbs fragmentation, CUDA graphs and the burst above your stated peak.

Below the button, Workers and Tensor parallel width show what the solver chose. You can override them by hand; the next Auto-size will overwrite your values, and so will opening a share link, because a shared link carries the workload and re-solves the hardware.

Diagram 3. Why the solver reaches for TP before replicas when a speed target is missed.

Advanced tuning

Normal and Advanced mode

Normal numbers the stations, asks for active users rather than concurrency, and hides the tuning controls and the manual topology fields. It is the right mode for a first pass and for handing the tool to somebody who is not a serving engineer. Advanced exposes everything. The switch is non-destructive: nothing is lost or reset, and the mode is remembered per browser.

Reading the results

The right column answers, in order: does it fit, what does it cost, where does the memory go, how fast is it, does it keep the promises, what does the rack look like, how does it degrade, and what should you change.

The verdict bar

Project fleet

The pool chips carry the deployment shape: precision, tensor-parallel width, GPUs per replica, replica count and pooled concurrency. The per-use-case cards carry what that workload experiences: admitted versus offered calls, first token against its target, per-user speed against its target, P95 against its target, KV cost per call, and which supporting models are attached. Numbers that miss a target are red; numbers with no target are grey.

Summary and the sizing decision

Three blocks:

• Use cases · demand. One line per card: active users, the concurrency that implies and what share of your users that is in flight, the model, and whether its SLOs are met.
• Deployments. One line per pool: how many pooled calls it serves on how many GPUs, the memory used, and the smallest tensor-parallel slice that fits one copy inside the memory target.
• Sizing decision. Written by the solver each time it runs, this is the audit trail: for each pool, the width it started from, whether a first-token target forced it wider, whether the group crosses a node boundary and what that costs, and any target it could not reach at any width.

Memory ledger

Each bar is one pool, drawn to the same scale so pools are comparable. The four segments are weights, KV cache, activations and overhead; the pale remainder is headroom. Hover any segment for exact gigabytes and its share. The line underneath each bar breaks the same numbers out in text, including the sequence count the KV figure is for, so a screenshot of this panel is self-explanatory.

Supporting models are accounted separately, on their own shared GPUs, and shown on their own line.

Key numbers

SLO compliance

Each badge lists that use case's enabled targets and whether each passes. Disabled targets (0) are shown as off rather than as a pass, so an empty promise never reads as a kept one.

Deployment topology and fleet map

Each node is a box; each GPU inside it is a cell. Cells are coloured by the pool that owns them, with adjacent replicas of the same pool in alternating shades so you can count them. Inside a cell, the stack is drawn bottom-up: model weights, then KV cache, then working set, with the percentage of the card in use. Empty space above the stack is memory the speed target cannot spend.

Underneath the map, a text form of the same information is available for screen readers and for pasting into a ticket, listing every node and what sits on each GPU.

The two charts

Both charts plot the decode equation from section 5.8 with one variable swept and everything else held at your configuration. One line per pool.

• Shaded bands mark the region above your slowest target (comfortable) and below it (missing). The boundary is the target itself.
• The amber dashed line is the binding per-user speed target, labelled with its value. Where a curve crosses it is the load or context length at which that pool stops keeping its promise.
• A hatched line in the legend marks a target that is currently not met at the operating point, so a failing pool is identifiable without reading the colours.
• The vertical marker is where you are now.

The first chart is the more important one: it is the shape of the batching trade. Adding requests to a replica raises total throughput and lowers every individual user's speed, because the KV that must be re-read each step grows with the batch.

Request latency anatomy

Four segments per use case: first token (prefill), framework overhead, hidden reasoning, and visible output. The reasoning segment is usually the surprise: a reasoning model with an 8,000-token budget spends most of the request producing tokens nobody reads, and that time counts against your P95 exactly like visible output does. The marker shows the P95 for that use case against its target.

Recommendations

Recommendations are graded: red is a defect in the configuration, amber is a risk or an over-provision, and plain is an explanation or an opportunity. Many carry buttons.

The full catalogue of recommendation types is in section 10.

The mathematics

The engine is about fifty lines of arithmetic between /*ENGINE-START*/ and /*ENGINE-END*/ in assets/app.js. There is no simulation, no fitted curve and no lookup table: every output is a closed form of the inputs, which is why the Excel export can reproduce it cell for cell and why you can check any figure by hand. This section is that arithmetic, in the order the engine evaluates it.

Notation

Weights

Worked: 70B at FP8 is 70 × 1 = 70.0 GB per replica. At BF16 it is 140 GB, at NV FP4 it is 35 GB.

KV cache per token, and effective KV

Every token a request holds occupies a key and a value vector in every layer. The uniform-attention form is:

Worked: Llama 3.3 70B has 80 layers, 8 KV heads and a head dimension of 128. At FP8, 2 × 80 × 8 × 128 × 1 = 163,840 bytes = 163.84 KB per token. A 8,192-token request therefore holds 1.34 GB of cache, and fifty of them hold 67 GB.

Architectures that do not fit the uniform form

Three families would be badly mis-sized by the formula above, so the library encodes them with an effective KV geometry chosen to make that same formula land on the true cost:

Prefix caching

Engine v27. If part of the sequence is byte-identical on every call, a server with automatic prefix caching computes it once and stores it once per replica rather than once per request.

Activations, overhead, and the fit inequality

Diagram 4. Why a fleet sized for latency looks empty, and why that is correct.

Prefill: time to first token

Worked: 8,192 tokens of a 70B dense model at TP4 on an H200 (989 dense FP16 TFLOPS) at MFU 0.5:

TTFT = 2 × 8192 × 70 / (989 × 4 × 0.5)
     = 1,146,880 / 1,978
     = 580 ms

At a 50% shared prefix only 4,096 tokens are prefilled and the same expression gives 290 ms.

Decode: tokens per second per user

Generating one token requires reading every active weight and every cached key and value for every request in the batch. Decode is therefore bandwidth bound, and the studio's central equation is a division of available bytes per second by bytes needed per token.

Worked: BW_eff = 4.8 × 4 × 0.85 × 0.65 × 1000 = 10,608 GB/s, weights term 70 × 1 = 70 GB, KV term 50 × 8192 × 0.00016384 = 67.1 GB:

tok/s = 10,608 / (70 + 67.1) = 10,608 / 137.1 = 77.4 tok/s per user
agg   = 77.4 × 50           = 3,868 tok/s for the pool

Admission and queueing

End-to-end latency and P95

tok/s implied by a P95 target  =  gen / (P95 / 1.3 − (TTFT + t_ovh)/1000)

Concurrency from a headcount

Little's law: the number of requests in flight is the arrival rate times the time each one takes.

Worked: 1,500 active users, 4 interactions/hour, 100% reaching the model, 1.5 calls each, 8 seconds per call, 2.5× burst gives 1500 × 4 × 1 × 1.5 × 8 / 3600 × 2.5 = 50 concurrent calls, 3.3% of the user base in flight.

A complete worked example

Every number below comes from the engine; you can reproduce all of them with a calculator.

Against the Clinical knowledge preset's targets of 800 ms, 40 tok/s and 50 s, all three pass with room to spare. (The example sets reasoning to zero rather than the preset's 1,000-token budget, purely to keep the arithmetic followable. Switch it on and gen becomes 1,350 tokens, and because those tokens also extend the KV cache the per-user speed falls to 73.0 tok/s, so the mean latency rises to 19.1 s and the P95 to 24.8 s: still inside the 50 s promise, but four times the number above.) Now set the shared cached prefix to 50%: the prefill halves to 4,096 tokens, TTFT falls to 290 ms, KV total falls to 34.2 GB because the shared half is held once instead of fifty times, and the per-user speed is unchanged at 77.4 tok/s, exactly as section 5.4 promises.

The auto-size solver

Auto-size answers: what is the cheapest topology that fits this pool and keeps every member's promises? It runs per pool, and it is deterministic: the same inputs always produce the same plan.

Diagram 5. The solver in full. Step 3c is the important one: the replica count that hits a speed target is algebra, not a loop.

Invariants the solver is held to

These are enforced by a fuzz harness over thousands of random configurations, and each of them was added after a real defect:

Pooling, co-residency and MIG slicing

Three different mechanisms let one GPU carry more than one thing. They apply in this order.

Pooling · one deployment, several use cases

Two use cases with the same model, the same weight precision, the same KV precision and the same KV policy are one deployment. Their concurrency adds, the pool is solved once against the combined load, and each member is then checked against its own targets at the pool's shared batch size. Tick Isolate to prevent it.

Whole-GPU co-residency

Separate pools can share one physical card the way vLLM memory fractions or Triton multi-model do. The studio merges them only when every one of these holds, and it records which one refused:

MIG slicing

A pool at TP1 whose copy fits a hardware partition can run one replica per slice and share physical GPUs with other models and with supporting services, the way Triton and vLLM deployments do on partitioned clusters. Only real slice geometries are offered, per GPU, and the per-slice speed is computed from that slice's own bandwidth and compute rather than the whole card's.

Supporting models

A RAG copilot is not one model. It is a generator plus an embedder plus a reranker, probably a guard model, and for a voice path an ASR and a TTS. Those services are small, they are always on, and leaving them out of a capacity plan is how a fleet arrives 15% short.

Presets attach the right kinds automatically; the chips on each use-case card show what is attached and let you change the model or detach it. 54 supporting models are in the library across six kinds, from tiny CPU-viable encoders to flagship tiers, including Arabic-specialised ASR, TTS, OCR, embedding, reranking and guard models.

Not in the library? Every chip's model dropdown ends with Custom (enter geometry), which asks for the only two numbers the sizing needs: gigabytes per running instance, and how many concurrent streams or queries one instance serves before another is needed. The fields are seeded from whatever was selected, so a house ASR starts from the nearest published model rather than from zero. A custom chip is drawn with a dashed border, and the geometry travels in the JSON export and in share links.

Supports are aggregated across the project by kind and model, so ten use cases that all want the same embedder do not buy ten embedders. They are then packed onto MIG slices, AMD partitions or fractional GPUs, and appear in the fleet map as shared cards and in the memory ledger on their own line. Their GPUs are counted once in the fleet total.

Resilience patterns

Capacity and survival are different purchases. The resilience model decides how much hardware you buy beyond the serving fleet and what that hardware guarantees during a failure.

Spare nodes are shared across pools where the hardware is uniform, because any spare can take over from any failed node. That is why a project with three pools and N+2 buys two spares, not six.

The SLO optimiser

Latency promises buy hardware. The optimiser's job is to show you the exchange rate and let you take the trade in one click, with an Undo.

What it proposes

The rules it plays by

• Never silently change a number you typed. Every change is a button you press, and every button quotes its effect first.
• Price by re-solving, not by estimating. The saving on the button comes from running the whole solver on the modified project.
• Quote the unit you buy. Cards are what a change moves; nodes are what you order. When a real card saving does not reach the procurement line, the recommendation says so in those words rather than implying a smaller order.
• Never propose something a workload class forbids. Floors come from the use case's own fields, not from the preset dropdown, so switching a voice agent to a custom workload does not change what it is.
• Round the paired promise the safe way. When a suggestion pairs a new speed with a new P95, the P95 is rounded up, because writing a promise the recommended speed cannot meet would be a lie.
• One Undo restores everything, across every use case the change touched.

Workload presets and the evidence behind them

The 24 presets are not opinions. Each one's SLO targets, context length and token budgets trace to published production guidance or a benchmark standard, and the trace is recorded in docs/PRACTICES.md along with a July 2026 review of every one of them.

Two rules govern every preset, and tools/check_presets.py enforces the first mechanically on every release:

1. Self-consistency. p95Target ≥ 1.3 × (ttftTarget/1000 + (reasonTok + visibleOut) / tpsTarget). A preset may never demand a P95 its own first-token and speed targets make arithmetically impossible.
2. Per request, not per task. Every field describes one model call.

Exports, share links and AI assistants

JSON

Schema gpuscale.net/5. Carries every use case, the project-wide hardware, and a full geometry snapshot of every model and GPU used, so the file still opens correctly after a library entry is renamed or removed: the studio rebuilds a missing model as a custom one and tells you it did. Older v3 and v4 single-scenario files import as a one-use-case project.

Excel workbook

Five sheets, and the important thing about it is that the Results sheet contains formulas, not values. Change a cell on the Inputs sheet and the whole model recalculates in Excel, mirroring the engine cell for cell.

PDF report

It adds two things the screen does not show: a printed inputs table covering every use case, and a Reading of this configuration narrative that states the assumptions in prose for a reader who was not in the room.

Share links

Three payload forms, all carrying the project inside the URL fragment. Nothing is uploaded and no short-link service is involved unless you self-host the optional worker.

Opening a link imports it as a new local project, never overwriting existing work, saves it, and clears the address bar so a refresh keeps your edits rather than the sender's snapshot.

Getting a link from ChatGPT, Gemini or Claude

The readable form exists because no chat model can deflate or reliably base64 a kilobyte of JSON, so without it an assistant could only give you arithmetic you cannot check. With it, an assistant can answer a sizing question with a link that opens a fully solved fleet:

https://gpuscale.net/#p=t:gpu=H200+141GB+NVL;perw=8;resil=n1;uc=Support+chat;
model=Llama+3.3+70B;quant=FP8;preset=Simple+RAG;users=2000

Pairs separated by ;, every uc= starts a use case, spaces written +. It carries the workload only; the studio solves the hardware. The complete key table, worked examples, encoding recipes and the rules that keep a generated link honest are in docs/URL-FORMAT.md. Point your assistant at that file, or at llms.txt, and it can build links unaided.

In the page, GPUscale.textShare() prints the readable link for whatever is on screen, which is the fastest way to learn the format. For agents with a shell, the downloadable gpuscale-link.skill in the footer validates every name, clamps every field and round-trips the link before printing it.

Libraries, and extending them

Everything the studio knows lives in five one-entry-per-line files under data/. There is no build step: edit a line, reload the page.

Adding a model is one line:

{"name":"MyModel 34B","params":34.0,"active":34.0,"hidden":7168,"layers":60,
 "kvHeads":8,"headDim":128,"ctx":131072, ...}

Use the effective-KV convention from section 5.3 for MoE, MLA and hybrid attention. docs/DATA.md has the complete schemas and worked examples for each family. The generated artifacts (the portable single-file build and both skills) are rebuilt by three scripts under tools/ so they cannot drift from the live engine.

Accuracy, limits and validation

Deliberately not modelled

How the engine itself is verified

Every release runs the same battery before it ships, and the numbers are recorded in the changelog:

• Parity against the previous release for any input that should be unaffected by a change. Engine v27 was byte-identical to v26 across 9,000 comparisons with prefix caching off.
• An independent reimplementation in Python, written from the documented formulas rather than from the code, checked field by field over thousands of random states. Latest run: 2,500 states, 75,000 field checks, zero mismatches.
• Solver optimality fuzz: no sampled plan is beatable by a cheaper feasible one.
• Reference projects re-rendered and diffed line by line against the previous release, so an unintended change to any number is caught before release.

Privacy, offline use and troubleshooting

• Nothing is uploaded. No accounts, no analytics, no cookies, no telemetry. Projects autosave to your browser's local storage; clearing site data clears them.
• Works offline. The page is installable, and the portable build is the whole studio in one HTML file you can email into an air-gapped network.
• A share link is a one-shot import into a new local project. It can never clobber existing work.
• Numbers look wrong? Check Resident sequence first: it is the tokens held per request, not the context window, and it is the field that is wrong nine times out of ten.
• Fleet looks too big? Read the Why the fleet is this size recommendation. It itemises the order by cause.
• Reporting a bug: include the version and engine numbers from the footer and a share link. The link reproduces your project exactly, and it contains nothing but your configuration.

Glossary

Where to go next