Compute is expensive and scarce, but paid GPUs are not automatically useful GPUs. This series looks at the capacity and scheduling systems that close that gap.

A simple model that splits a compute cluster into four layers: reservation, provisioning, scheduling, and workload. The model turns 'my pod is Pending' into a diagnosis instead of a three-team argument.

The same 'Pending' can come from any of the four layers. The scheduler often prints the same line for all of them, so the message alone won't tell you which one is broken.

The same chip delivers a ~10× range of throughput depending on precision. The spec-sheet number is a ceiling, and most workloads never touch it.

The second thing inside the accelerator: its memory. Two numbers — capacity (does it fit) and bandwidth (can you feed the compute) — and which one limits you depends on the workload, not the chip. This is why the FLOPs ceiling from 1.1 is rarely your speed.

The third component: the links between the GPUs inside one box. On a p5.48xlarge, 8 GPUs aren't 8 independent cards — NVLink wires them into one unit. The catch is that your software has to actually use that fast lane, or it silently falls back to a path ~14× slower.

1.1–1.3 were the accelerator itself; this is the machine around it — the CPU, system RAM, and local NVMe that feed the GPUs across a thin PCIe straw. We walk one training step through the host and see where each stage can stall.

The last piece: once a job outgrows a single box it crosses the datacenter network — a leaf-spine tree, not a flat wire. Two GPUs one leaf apart are a few switch hops closer than two GPUs across the spine, and a collective runs at the speed of its worst hop. So placement is a throughput decision.

A whole accelerator handed to a job that uses a sliver of it is the most common waste; the fork between splitting in space and sharing in time.

Carve one physical H100 into isolated instances so two workloads run at once on a card a single job would hog — a step-by-step MIG demo, then the rigidity cost.

Pack several sub-saturating jobs onto one GPU — MPS runs them truly concurrently, time-slicing rotates turns — with a step-by-step MPS-vs-time-slicing demo, then the shared-fault-domain cost.

Sharing only pays off if one job can't steal another's memory or tail latency; where each sharing mode's isolation breaks.

The four-phase loop that places every GPU job — and where gang scheduling, fragmentation, and topology-aware placement all live inside it.

A distributed training job needs every piece placed simultaneously — partial placement deadlocks and wastes every GPU it grabbed.

Free GPUs you can't use because they're scattered: how placement policy decides whether big jobs ever fit.

Where a job's ranks land on the fabric decides whether collectives ride NVLink or crawl across spine switches — topology-aware scheduling claims the fast path.

Guaranteeing each team a floor without letting the cluster sit idle behind that guarantee — lend the spare, take it back on demand.

Not enough to go around — who gets evicted, how, and why evicting too eagerly costs more than it saves.

Serving is provisioned for peak and idle most of the day; backfill it with batch without hurting latency.

Demand has a daily rhythm; ride it instead of provisioning for peak and paying for the trough.

Jobs that expand into idle GPUs and contract when demand returns — absorbing the spare instead of leaving it idle.

Saving progress cheaply so an interruption costs minutes, not days — and the tradeoff between checkpoint overhead and lost work.

Giving capacity back without losing work — spot reclaim, preemption, and the orderly cordon → drain → evict dance.

At scale, hardware fails and slows constantly; detecting the fail-slow node before it burns a whole run's cycles.

Three numbers people all call 'utilization' that disagree — and which one tells you you're wasting money.

A dollar figure on a single idle percentage point at fleet scale — the number that justifies the whole series.

Why a paid commitment must run hot to beat on-demand, and when spot wins.

How much to reserve is a prediction problem; sizing committed capacity against forecast demand instead of guessing.

The whole series in one argument — supply and scheduling aren't two problems, and the seam between them is the leverage.