The First Thing I Validate Is the Load Generator

NVIDIA published Benchmarking LLM Inference at Scale with AIPerf on September 18. AIPerf is the successor to GenAI-Perf, with multiprocess workers, ZeroMQ between components, and a walkthrough using Qwen3-0.6B behind vLLM.

While reading through the aiperf profile flags, I kept coming back to one question: what did the benchmark actually measure?

That question matters more than the tool name or the final percentile.

Why this post?

I have spent enough time around Java backends to see load tests fail in a predictable way: the injector reaches 100% CPU while the service under test still has headroom. The resulting p99 may be perfectly accurate, but it describes the client rather than the server.

LLM endpoints make this easier to miss. A hand-written asyncio load generator looks mostly like I/O, but tokenization, SSE parsing, JSON processing, and histogram updates all consume CPU. If they run in the same process responsible for scheduling requests and measuring time, the load generator can become part of the latency it is trying to observe.

NVIDIA describes GenAI-Perf as having this limitation because it ran on top of Perf Analyzer as a single process. AIPerf changes that architecture by separating load generation from metric processing.

The design became easier for me to reason about once I reduced it to four things: three execution planes, a credit-based request path, separate latency metrics for prefill and decode, and a requirement that the benchmark itself be validated before its output becomes a release gate.

Three planes and a credit

The architecture documentation divides the benchmark harness into three planes:

Requests move through the system using credits.

The Dataset Manager loads prompts into memory-mapped files. Those prompts may be synthetic, come from ShareGPT, or come from traces such as Mooncake, Baseten, or WEKA. Workers read from those files rather than waiting for prompt strings to move through a Python queue.

An optional warmup phase prepares JIT paths, caches, and connection pools. Warmup samples are discarded so startup behavior does not contaminate steady-state measurements.

The Timing Manager issues credits, where one credit authorizes one request. The timing mode determines how those credits are produced: a fixed trace schedule, request-rate mode with constant, Poisson, or gamma arrivals, or a user-centric delay between turns.

Credits are distributed over a ZMQ ROUTER/DEALER socket. New sessions are assigned to the least-loaded worker, while multi-turn sessions remain attached to the same worker. Completed work returns through a separate PUSH/PULL path, keeping high-volume completion traffic separate from request dispatch.

A worker sends the HTTP request, records the raw timing information, and forwards the result to a record processor. Metric calculation happens in another process.

That separation is the important architectural change. Load generation and analysis no longer compete inside the same Python process.

The credit model also makes the arrival semantics explicit. A worker only sends when it receives permission. If credits are generated at a requested rate without a concurrency limit, the benchmark behaves like an open arrival system. AIPerf also allows --concurrency to be used as a ceiling. Once that ceiling is applied, a Poisson arrival process is no longer purely open because backpressure can prevent some scheduled work from being issued immediately.

That distinction becomes important when interpreting the resulting request rate.

When the injector measures itself

CPython’s global interpreter lock allows one thread at a time to execute Python bytecode within a process. PEP 703 is the effort to make the GIL optional, but the conventional CPython runtime used by most tooling still has this constraint.

A load generator may appear mostly unaffected because socket waits release the GIL. The CPU work surrounding those waits does not. Examples include:

• decoding SSE chunks
• tokenizer round trips when token counting happens on the client
• updating histogram state
• JSON parsing and object creation

At low concurrency, this overhead may be negligible. At the concurrency levels used to size GPU serving systems, the event loop can begin queueing its own callbacks. TTFT rises, achieved throughput flattens, and GPU utilization reported through DCGM may remain below saturation. At that point, the benchmark is limited by the injector.

I have seen the same class of failure with an undersized Gatling injector and with jmeter running from a laptop. The symptom is consistent: the service still has capacity, but the machine generating load does not.

Threads do not remove this problem in a GIL-bound Python process because they share the same interpreter lock. Processes provide a different boundary: each process has its own interpreter and its own GIL.

AIPerf follows that model. Workers are processes, record processors are processes, and coordination uses ZMQ. Dataset bytes are kept in memory-mapped files rather than repeatedly serialized through Python queues. Telemetry export runs in its own child process with a bounded queue, including drop-oldest behavior when the collector falls behind, so a slow OTLP exporter does not have to block the timing path.

Before trusting measurements from a load generator, I would still validate it independently. One simple test is to point the harness at a dummy endpoint that emits a fixed stream after a known delay. If measured TTFT tracks that delay while GPU telemetry remains idle, the client is at least demonstrating that it can keep time under the chosen load. If TTFT grows as concurrency increases even though the dummy’s behavior is unchanged, the injector is contributing to the result.

The migration guide is therefore more than a translation of command-line flags. Moving from GenAI-Perf to AIPerf should also be treated as a re-baseline. A high-concurrency GenAI-Perf p99 should not automatically be compared with an AIPerf p99 unless the old load generator has been shown not to be the bottleneck.

TTFT for prefill, ITL for decode

A generative request has two main execution phases, and they place different demands on the GPU.

Prefill reads the full prompt, builds the KV cache, and produces the first token. It is compute-heavy, and its cost generally increases with input length. The user-visible metric is time to first token (TTFT). NVIDIA defines TTFT as including queueing, prefill, and network time, so a high TTFT does not identify a single cause. It may come from a long prompt, or from requests waiting for a batch slot before prefill begins.

Decode generates the remaining tokens one at a time. Each step reads model weights and accesses the growing cache, which makes this phase much more sensitive to memory bandwidth. The corresponding user-visible metric is inter-token latency (ITL), also called time per output token. AIPerf uses the same formula described in the NIM benchmarking guide:

ITL = (request_latency - TTFT) / (output_tokens - 1)

The first token is excluded because TTFT already accounts for the work required to produce it. ITL measures the decode portion of the request, while total request latency includes both TTFT and the remaining generation time.

If those phases are collapsed into a single end-to-end average, a slow prefill and a slow decode can look identical even though they require different fixes. A queueing problem can easily be misdiagnosed as a model-performance problem.

That distinction is one reason DistServe (OSDI 2024) and Splitwise (ISCA 2024) place prefill and decode on different GPU pools. DistServe models uniform prefill as an M/D/1 queue, which gives it a closed-form treatment of TTFT. Orca takes a different approach by deliberately mixing phases through iteration-level scheduling. vLLM makes the KV cache pageable so mixed batches can use memory more efficiently. A single latency average hides most of those distinctions.

There are two additional measurement details worth making explicit:

• Streaming determines whether TTFT and ITL can be observed. Without --streaming, the client receives the completed response rather than a sequence of token events. There is no first-token timestamp or inter-token sequence from which to calculate ITL. The metric depends on how the response is delivered, not only on what the server eventually returns.
• An empty first chunk is not a token. The metrics reference ignores an initial SSE message that contains no content. Reasoning models add another distinction: AIPerf’s TTFT is the first token of any kind, including a reasoning or thinking token. Time to first output token (TTFO) measures the first non-reasoning token. That is what GenAI-Perf previously called TTFT, so old and new dashboards are not directly comparable unless the definition change is accounted for.

A healthy mean TTFT can still coexist with a poor interactive experience when the tail is large. For an interactive system, p99 often carries more operational meaning than the mean because it captures the users who wait much longer than the typical request.

Closed concurrency and open arrivals

Fixed --concurrency N creates a closed system. When one response finishes, another request is sent, keeping the number of outstanding requests near N.

Little’s law still applies:

N = λW

The important detail is that λ, the achieved request rate, is now an output of the system rather than an independently controlled input. If the server becomes slower, the load generator also sends new requests more slowly because it waits for earlier requests to finish. The offered load therefore cannot continue increasing independently, and a queue in front of the model will not grow the way it can under an open arrival process.

This is still a useful workload. It answers a specific question: given N clients that wait for a response before issuing another request, what throughput and latency does the system achieve? A small interactive chat population can resemble that pattern. A public API receiving independent requests usually does not.

Using --request-rate with --arrival-pattern poisson creates an open system instead. Inter-arrival times are exponentially distributed while the mean request rate is configured explicitly. Requests arrive unevenly, including occasional gaps and short bursts. That variation is expected behavior for a Poisson process.

NVIDIA’s baseline for establishing the measurement loop on Qwen3-0.6B uses fixed 128-token input and output lengths:

aiperf profile \
    --model Qwen/Qwen3-0.6B \
    --endpoint-type chat \
    --streaming \
    --url localhost:8000 \
    --synthetic-input-tokens-mean 128 \
    --synthetic-input-tokens-stddev 0 \
    --output-tokens-mean 128 \
    --output-tokens-stddev 0 \
    --extra-inputs min_tokens:128 \
    --extra-inputs ignore_eos:true

Their open-arrival example changes both the arrival process and the request-length distribution:

aiperf profile \
    --model Qwen/Qwen3-0.6B \
    --endpoint-type chat \
    --streaming \
    --url localhost:8000 \
    --request-rate 10 \
    --arrival-pattern poisson \
    --synthetic-input-tokens-mean 512 \
    --synthetic-input-tokens-stddev 128 \
    --output-tokens-mean 128 \
    --output-tokens-stddev 32 \
    --random-seed 42 \
    --request-count 200

The cumulative dispatch plot fluctuates around the target rate of 10 requests per second. A constant-rate test would produce a much straighter dispatch line. The variation here is simply what a Poisson process looks like.

The second experiment also introduces variable request lengths. Input length has a mean of 512 tokens and a standard deviation of 128, with the observed ISL ranging from 154 to 818. Output length has a mean of 128 and a standard deviation of 32. The test also removes min_tokens and ignore_eos, allowing the model to stop naturally at EOS.

That changes the scheduling problem substantially. Prefill cost varies from request to request, as does decode occupancy. Continuous batching may have to admit a long prompt while several shorter decodes are active. KV-cache pages are allocated and released as requests of different lengths move through the batch. This is the type of memory-management problem that PagedAttention was designed to address, and it is mostly absent from a static 128/128 workload.

NVIDIA’s TTFT histogram for the Poisson run is wider than the concurrency-1 baseline, which is expected. More requests share the GPU, prefills overlap with active decodes, and some prompts are substantially longer than others.

I would still keep the single-user test because it establishes a useful floor. I would not use it as the only release criterion for a production service.

For a performance gate intended to resemble production, I would keep three workloads:

1. A fixed-length concurrency sweep. Keep ISL and OSL static, enable streaming, pin output length, and sweep concurrency. This gives a reproducible floor and a latency-throughput curve.
2. An open-arrival workload with realistic length variation. Use Poisson arrivals or, preferably, a captured production trace. This exposes queueing, batching, and interactions between requests of different lengths.
3. The same open workload with a concurrency ceiling when the product has one. If production explicitly caps or sheds load, reproduce that limit so the benchmark measures the overload policy as well as the model server.

Flags that are part of the measurement

Some CLI options are not merely tool configuration. They change the experiment itself.

--streaming. If TTFT or ITL is part of the SLO, streaming determines whether those metrics are observable. With streaming disabled, the client primarily measures completion latency rather than first-token and inter-token timing. Treating that number as TTFT would change the definition of the metric.

--random-seed. NVIDIA’s blog notes that rerunning the Poisson example produces the same sequence of generated requests. The reproducibility documentation defines the boundary more carefully. The seed makes the generated workload deterministic: prompt generation, sampling order, Poisson interval draws, and session IDs can be reproduced. It does not make execution timing deterministic.

Worker assignment, ZMQ routing, asynchronous I/O, and server scheduling can still vary between runs, so TTFT and ITL will not be bit-for-bit identical. I therefore treat the seed as a way to pin the workload when comparing server builds, not as a reason to expect an identical p99 from every run. A CI gate based on exact percentile equality would be unnecessarily fragile.

Output length. --output-tokens-mean 128 does not by itself guarantee a 128-token response. Without min_tokens and ignore_eos:true, the model may encounter EOS earlier and stop generating. That affects throughput, the number of tokens over which ITL is averaged, and the comparability of one run with another.

For a controlled baseline, I pin output length by using a zero standard deviation, setting min_tokens to the target length, and ignoring EOS. For a production-shaped workload, I allow output length to vary and report the output-length distribution alongside ITL. Otherwise, a 40-token response and a 128-token response can be compared as though they represented the same amount of decode work.

Warmup, tokenizer configuration, and the exact vLLM image tag belong in the same category. Each can change the workload or server behavior. Keeping them fixed is what allows a comparison to isolate the server change being tested.

Correlating latency with GPU behavior

Even after validating the client and defining the workload, a change in a latency percentile is not enough to identify the source of a regression.

AIPerf can collect GPU power, utilization, and memory metrics during the same run when DCGM or pynvml is available. Server-side Prometheus metrics such as KV-cache utilization, batch size, and queue depth can be collected alongside them. The GPU telemetry tutorial covers the practical setup, and Dynamo exposes DCGM metrics on port 9401.

I want those signals on the same timeline as p95 and p99 because different combinations point toward different parts of the system:

Splitwise showed that decode and prefill do not necessarily consume the same power budget. That distinction matters when diagnosing a regression.

If p99 increases while GPU power falls, I would first look at batch formation and scheduling rather than assume the checkpoint became slower. If power is pinned at the device limit while ITL rises, the scheduler may be admitting enough prefill work to interfere with decode.

AIPerf also derives tokens-per-joule and related efficiency metrics from the same measurement window. I would not make those the first release gate, but I would keep them next to the latency metrics. A change that improves p99 by consuming substantially more power is still worth understanding even if latency alone looks better.

Why Java records?

I use Java often enough that records are a convenient way to make the validation rule explicit in code. Records let me represent the pieces of the benchmark as small immutable values rather than relying on comments or conventions.

The invariant is straightforward: a performance result should become a release gate only after the load generator, workload shape, and acceptance thresholds have each been validated independently. I want that validation represented explicitly rather than inferred from the presence of a particular command-line flag.

public enum Arrival {
    CLOSED_CONCURRENCY,
    CONSTANT_RATE,
    POISSON,
    GAMMA,
    TRACE
}

public record Validated<T>(T value, boolean independentlyValidated) {}

public record LoadGenerator(
        boolean multiprocessWorkers,
        boolean metricsOffHotPath
) {}

public record WorkloadShape(
        Arrival arrival,
        int islMean,
        int islStddev,
        int oslMean,
        int oslStddev,
        boolean streaming,
        boolean outputLengthPinned,
        Long seed
) {}

public record AcceptanceThresholds(
        Duration ttftP99,
        Duration itlP99,
        double gpuUtilAtGate,
        double kvCacheUtilAtGate
) {}

public record BenchmarkClaim(
        Validated<LoadGenerator> generator,
        Validated<WorkloadShape> shape,
        Validated<AcceptanceThresholds> thresholds
) {
    public boolean trustworthy() {
        return generator.independentlyValidated()
                && shape.independentlyValidated()
                && thresholds.independentlyValidated();
    }
}

I would validate the load generator first by pointing it at a dummy server with a known response delay. The measured latency should track that delay while the GPU remains idle. AIPerf’s multiprocess workers and off-hot-path metric processing are architectural reasons to expect the injector to scale; the dummy test is what verifies that assumption in the environment where the benchmark will actually run.

The workload shape needs a separate validation against production. That includes prefix reuse, ISL and OSL distributions, arrival behavior, and whether clients consume streamed responses. A fixed 128/128 closed-loop workload is useful as a baseline, but using it as the only release test assumes production behaves the same way.

Acceptance thresholds need the same treatment. They should be agreed on as product requirements and paired with enough GPU context to make the result interpretable. A requirement such as p99 TTFT < 400 ms, for example, is incomplete if the same run is allowed to push the KV cache to the edge of exhaustion.

If any of those three components is still based on an assumption rather than validation, trustworthy() remains false regardless of how polished the benchmark report looks.

What I would use in practice

1. Validate the injector against a dummy server before running against a GPU. If the client cannot reproduce a known delay accurately, every later latency measurement is suspect.
2. Keep both a controlled baseline and a production-shaped workload. The baseline should use fixed lengths, pinned output, streaming, a reproducible seed, and a concurrency sweep. The production workload should include realistic arrival behavior, length variation, and whatever concurrency ceiling the product actually enforces.
3. Gate TTFT p99 and ITL p99 independently. End-to-end latency is still useful for dashboards, but it should not replace separate prefill and decode signals when deciding whether a build regressed.
4. Pair latency regressions with GPU telemetry. Utilization, memory, power, and KV-cache state provide context that the latency percentile alone cannot. If those signals remain stable, I would investigate the client, network, and workload before blaming the model server.
5. Treat the random seed as a workload pin rather than a metric pin. The same generated requests can be reproduced, but scheduling and timing still vary. Compare distributions and tolerances rather than expecting an identical p99 from every execution.

What I am still unsure about

NVIDIA’s post says the same tool can handle multi-node Kubernetes environments. The architecture page I read is more cautious: Kubernetes-related paths exist in the repository, but the documentation does not describe them as a fully supported run type. I would want that support status clarified before designing a fleet-wide load-generation setup around it.

I also want better visibility into credit backpressure when --concurrency is used as a ceiling on a Poisson workload. If workers remain occupied long enough that credits cannot be issued at the configured rate, the achieved dispatch rate can fall below the requested rate. A report could then resemble a successful 10-request-per-second test even though the generator did not actually offer 10 requests per second for much of the run.

I would like to see offered rate and achieved rate together, similar to the dispatch-versus-roofline view already used in the examples.

Prefix caching is another limitation of purely synthetic prompts. Production datasets such as ShareGPT or Mooncake traces may contain meaningful prefix reuse, while randomly generated 512-token prompts usually do not. A server that benefits substantially from a repeated system prompt can therefore look worse under synthetic input than it does in production. The opposite distortion is also possible if production traffic has less reuse than the benchmark assumes.

AIPerf can replay traces, but the default walkthrough does not exercise that dimension.

Until the load generator, workload shape, and acceptance thresholds have each been validated independently, I treat the output as measurements from a script rather than a benchmark suitable for a release decision.

References

More reading on the components discussed above:

• Benchmarking LLM Inference at Scale with AIPerf. NVIDIA’s walkthrough of AIPerf and the examples used here.
• Architecture of AIPerf. Control, data, and analytic planes; credits; worker processes; and ZMQ communication.
• AIPerf on GitHub and the migration guide from GenAI-Perf.
• Metrics reference and NIM LLM metrics. TTFT, ITL, TTFO, and the ITL calculation.
• Random number generation and reproducibility. What --random-seed does and does not make deterministic.
• GPU telemetry tutorial. Collecting DCGM and pynvml metrics.
• LLM inference benchmarking: fundamental concepts. Concurrency, request rate, ignore_eos, and streaming.
• The Python GIL and PEP 703. The limits of CPU-bound threading in a conventional CPython process.
• ZeroMQ and the ROUTER/DEALER guide. The communication model used between AIPerf components.
• Orca (OSDI 2022). Iteration-level scheduling and the foundations of continuous batching.
• vLLM / PagedAttention (SOSP 2023). Paged KV-cache management and the memory implications of variable request lengths.
• DistServe (OSDI 2024) and Splitwise (ISCA 2024). Treating prefill and decode as workloads with different resource characteristics.
• Little’s law, Poisson arrivals, queueing theory, and M/D/1. Queueing concepts behind closed and open workload models.
• NVIDIA DCGM. GPU telemetry for utilization, power, memory, and related measurements.
• How NVIDIA Dynamo 1.0 powers multi-node inference. Background on the serving stack used by later AIPerf examples.
• Java records (JEP 395). The language feature used for LoadGenerator, WorkloadShape, AcceptanceThresholds, and BenchmarkClaim.

• ← Previous Log