Infrastructure · 20 Nov 2026

How long does the 95th agent wait?

A hundred agents need 1 GiB each over one 400 Gb/s link. Agent 95 waits at least 2.04 seconds before GPU work resumes. Mistral and vLLM’s measurements show how host memory and sparse retrieval change the economics of that wait.

One hundred agents wake up at once. Each needs 1 GiB of state from storage before it can generate another token. Give them a single 400 Gb/s link and the absolute ceiling is 46 complete transfers in the first second. Agent 95 waits at least 2.04 seconds before any GPU work, even with no protocol overhead. Idle accelerators cannot increase the link’s bandwidth.

Mistral and vLLM demonstrated why this is a design choice. Adding 440 GiB of host memory to a four-GB300 serving configuration increased available cached context from 1.97 million tokens to as much as 11.25 million in steady state. Sparse attention reduces how much must return to the GPU. Buyers should ask vendors to wake a realistic cohort together, then report the 95th wait and the cost of finishing all the jobs.

GB300 hardware imagery · Ingrasys

← Back to the archive