Benchmarking LLM Inference at Scale with AIPerf

Benchmarking LLM Inference at Scale with AIPerf

A demo can serve a handful of requests per second and feel instantaneous. The same service, under hundreds of concurrent users, can spend seconds queueing before it emits a single token. Nothing changed in the model. What changed is that the first measurement was taken against an idle system, and the second was taken against a loaded one.

That gap is the entire reason a dedicated benchmarking tool for large language model inference exists. NVIDIA's AI Blog post Benchmarking LLM Inference at Scale with AIPerf (https://developer.nvidia.com/blog/benchmarking-llm-inference-at-scale-with-aiperf) covers AIPerf, a tool built for exactly this problem. This article treats that post as its factual anchor and then explains the methodology around it: how to frame an inference benchmark, which numbers deserve trust, and where measurement quietly goes wrong.

A note on sourcing, stated plainly. The verified fact underlying this article is that NVIDIA published the post above about AIPerf. Specifics of the tool — its interface, supported endpoints, workload types, and feature set — belong in that post and in its documentation, and they change over time. Everything below about methodology is guidance and interpretation, not a claim about AIPerf's internals. Where this article cannot verify something, it says so rather than guessing.

Why LLM Inference Breaks Conventional Load Testing

Traditional web load testing assumes requests are short, cheap, and interchangeable. LLM inference violates all three assumptions at once.

A request does not produce one response; it produces a stream of tokens over time. Latency is therefore not a scalar but a curve with at least two distinct regions: the wait before the first token and the cadence between subsequent tokens. A user's perception of speed depends mostly on the first region. A user's perception of fluency depends on the second. A single "average response time" collapses both into a number that describes neither.

Cost is also variable in a way that web requests are not. A short prompt that produces twenty tokens and a long prompt that produces two thousand tokens are the same HTTP request from the network's perspective and completely different workloads from the server's. Benchmarks that treat them as equivalent produce throughput figures that cannot be compared to anything.

Finally, inference servers are stateful in ways that matter. Batching decisions, KV cache occupancy, and memory pressure all depend on what else is in flight. A request that arrives into an empty cache behaves differently from one that arrives behind a thousand similar prompts. Any benchmark that does not control for that state is measuring history as much as capacity.

What AIPerf Is and Where It Fits

General-purpose load generators can produce HTTP traffic. Producing a realistic token-streaming workload, recording per-token timing, sustaining enough concurrency to saturate a modern accelerator, and aggregating the results into something decision-grade is a different engineering problem — one that justifies a purpose-built tool.

That is the category AIPerf occupies, per NVIDIA's post. The practical implication for a team evaluating inference infrastructure is straightforward: the choice of tool matters less than the discipline applied to it. A well-designed benchmark run with a modest tool beats a careless run with a sophisticated one every time.

The remainder of this article is about that discipline.

The Metrics That Actually Matter

Five measurements do most of the work.

Time to first token (TTFT) captures prefill, queueing, and scheduling — everything that happens before the user sees anything. It is the dominant driver of perceived responsiveness in interactive applications.

Inter-token latency (ITL), sometimes expressed as time per output token, captures the decode loop. It determines whether output feels fluid or stuttering. It is usually reported as a median because the first token is excluded, and because decode cadence is comparatively stable until the server saturates.

End-to-end latency is the sum of both plus the full generation. It matters most for batch and agentic workloads where no human is watching the stream.

Throughput must be specified precisely, because two different numbers share the name. Request throughput counts completed requests per second. Token throughput counts generated tokens per second. A system can improve one while degrading the other, and vendors rarely state which they mean.

Goodput is the metric that resolves this. It counts only the requests that met a defined service-level objective. A configuration that produces 6,000 tokens per second while missing its latency target on a third of requests has a goodput problem, not a throughput triumph.

Report percentiles, not averages. A p50 that looks healthy next to a p99 that is forty times larger describes a system that is fine most of the time and unusable some of the time — which, for an interactive product, is a failing system.

Always attach the workload to the number. "5,000 tokens per second" means nothing without the prompt and output length distributions, the concurrency level, the hardware, and the software versions. Without those, it is a marketing figure, not a measurement.

Defining the Workload Before You Measure Anything

Most bad benchmarks are bad before the first request is sent, because the workload was never specified.

Start with input length. Real prompts are not uniform; a chat assistant with retrieval produces a distribution with a long tail. Synthetic tests that use a single fixed prompt length are useful for isolating variables and misleading when presented as representative.

Then output length. If the server stops at a maximum token count, requests that hit the cap reveal something about the workload, not the server. Decide in advance whether capped responses count as successes.

Then arrival pattern. This is where the most consequential design decision lives. In a closed-loop test, a fixed number of virtual clients each send their next request only after the previous one completes. This model is easy to reason about and systematically understates overload: when the server stalls, the clients politely wait, and the stall is never recorded as latency. In an open-loop test, requests arrive on an independent schedule regardless of whether earlier requests have finished. Queueing delays then appear where they actually occur — in the user's experience.

This failure mode is known as coordinated omission, and it is the single most common way an inference benchmark flatters a system. If your test only ever asks "how fast was this request?" and never "how long would a user have waited for their request?", you have measured service time, not latency.

Finally, decide the concurrency sweep in advance. A single concurrency point tells you almost nothing; the shape of the curve is the finding.

A Practical Example: Building a Benchmark Plan

The scenario below is illustrative. The numbers are design choices, not measured results.

Suppose you are qualifying a chat assistant that prepends a long system prompt and retrieved context, then generates short answers. Expected input is roughly 2,000 tokens, expected output roughly 150.

Step one: state the objective as a constraint, not a wish. For example: p95 TTFT under 800 ms and p95 inter-token latency under 50 ms, sustained at 40 requests per second. Writing this down first prevents the common outcome of running a benchmark and then deciding which number looked impressive.

Step two: build the sweep. Run at concurrency levels of 1, 8, 32, 128, and 512. Hold each level for a fixed window long enough to reach steady state. Keep the prompt-length distribution identical across levels so that changes in the curve reflect load, not content.

Step three: pin everything. Model revision, serving configuration, parallelism settings, batch limits, client hardware, and network path. An unpinned benchmark produces an anecdote, not a result.

Step four: warm up, then discard. The first requests against a cold cache are not representative of steady-state operation. Run a warmup phase and exclude it from reporting.

Step five: repeat and report spread. Three runs at each concurrency level, with the variance shown. A single run that happens to look good is not evidence.

Step six: find the knee, not the peak. Throughput rises roughly linearly with concurrency until a queue forms. Past that point, throughput flattens or declines while latency climbs steeply. The useful output of the exercise is not the highest number the system ever produced; it is the highest concurrency at which the service-level objective still holds. That is your operating point.

Where Inference Benchmarks Go Wrong

The client becomes the bottleneck. Tokenization, TLS handshakes, and per-token sampling are CPU work. At high concurrency, the load generator can saturate before the server does, and you end up measuring your benchmarking machine. Confirm client headroom before trusting any result at the top of the curve.

Cache effects inflate results. Repeating a small set of identical prompts produces unrealistically high cache reuse. Use prompt diversity that matches production, or state explicitly that you are measuring a cache-warm best case.

Tokenizers are mismatched. Counting tokens with a different tokenizer than the server uses produces throughput figures that are wrong by a few percent and are never reconciled.

Configuration drifts between runs. Comparing an eight-way parallel run against a four-way one, or a run with different sampling parameters, produces a difference that has nothing to do with the change under test.

The tail is averaged away. Reporting mean latency hides exactly the behavior that generates user complaints and incident tickets.

Throughput is optimized at the expense of responsiveness. A configuration with excellent token throughput and poor TTFT may be the right choice for offline batch processing and the wrong choice for a chat interface. The metric set should follow the product.

Overload behavior is never tested. Push past the knee deliberately. Does the system degrade gracefully, shedding load and holding latency, or does it fall off a cliff? That behavior is more valuable to know than the peak number.

Scaling the Benchmark Itself

Above a few hundred concurrent streams, the benchmark becomes a distributed system and inherits distributed system problems.

Percentiles do not average. Merging the p99 of three load generators by taking their mean is arithmetically meaningless. Correct aggregation requires merged latency distributions or histograms collected from each worker — a detail that silently corrupts results in otherwise careful evaluations.

Clock alignment matters for the same reason. If workers disagree about the current time, per-request timing gets noisy at exactly the scale where precision matters most.

Sustained high-concurrency runs also need connection reuse and careful memory management on the client side. A generator that allocates per token will spend its time in garbage collection rather than in measurement.

Finally, multi-turn and long-context workloads add a dimension that single-request tests miss entirely. Session state, growing context, and repeated prefixes change cache behavior in ways that only appear when a benchmark models a conversation rather than a request.

Interpreting and Reporting Results

A benchmark report that cannot be reproduced is not a report. At minimum, it should contain the workload definition, the hardware and software versions, the concurrency levels tested, the metric values with percentiles, and an explicit verdict against the service-level objective.

A compact table works well, and the one below is a template with placeholder values, not measured data:

The deliverable of this exercise is not a number. It is a documented operating envelope: the range of load inside which the service behaves as promised, and a description of what happens outside it.

Open Limits

Several things this article cannot settle. The performance of a specific model on specific hardware under a specific serving stack depends on all three, and no general guidance substitutes for running your own workload. Tool features evolve, so the authoritative description of AIPerf's capabilities is the source post itself, not a secondary summary. And a benchmark, however carefully run, characterizes a configuration at a moment in time; it does not predict behavior after a driver update, a batching change, or a shift in traffic mix.

Treat published results, including favorable ones, as hypotheses about your deployment rather than conclusions about it.

Conclusion

LLM inference at scale does not behave like web traffic, and tools designed for web traffic measure the wrong things. The work of benchmarking it comes down to a few disciplined choices: define the workload before running it, sweep concurrency rather than testing a single point, prefer open-loop arrival patterns so queueing delays are visible, report percentiles instead of averages, and treat goodput — not peak throughput — as the number that determines whether the system is actually usable.

AIPerf exists to make that measurement practical at scale. The tooling shortens the distance between a question and an answer; it does not decide which question to ask. That decision — the objective, the workload, and the operating point you are willing to ship — remains yours.