ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

Hybrid language models — architectures that interleave attention layers with state space or linear-recurrent layers — promised the best of two worlds: the exact recall of attention and the constant-memory streaming behavior of recurrence. In practice, many teams discover that the promise holds during prefill and then quietly erodes during decode. Throughput per output token drops, memory accounting becomes hard to reason about, and features that are routine for pure transformers — beam search, speculative decoding, many samples per prompt — become awkward.

NVIDIA's blog post "ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding" (dated October 12, 2026) examines that tension and proposes treating state storage as a first-class design concern rather than an implementation detail. This article explains the problem space, walks through a practical setup for experimenting with hybrid decoding on your own hardware, and outlines what to measure. Conceptual explanation here is interpretation; specifics of the proposed method should be taken from the original post.

Why Hybrid Models Are Attractive — and Awkward to Decode

A transformer layer keeps a key/value cache that grows linearly with sequence length. An SSM layer keeps a fixed-size recurrent state that is rewritten by every token. Stack them together and you get a model that can attend precisely where it matters and stream cheaply everywhere else.

The catch is that the two memory systems have almost nothing in common:

• KV cache is append-only, per-token, and content-addressed by position. You can slice it, page it, share prefixes, and drop a suffix without touching the rest.
• SSM state is a running summary. It has no position index. Token t produces a new state that overwrites the old one, and that state is a compressed function of everything before it.

That asymmetry is the root of the decoding problem. A KV cache is a data structure you can manipulate. A recurrent state is a value you must either preserve or recompute.

Why Decoding, Not Prefill, Is the Hard Part

Prefill processes a whole prompt in one pass. Every token is available at once, the GPU is saturated with matmul work, and the cost of writing state is amortized across a long sequence. Decode is the opposite: one token per step, tiny matmuls, and a memory-bound kernel whose runtime is dominated by reading and updating state.

For a pure attention model, decode performance is largely a question of KV cache size and how efficiently you can read it. Quantization, paging, and prefix sharing all attack that. For a hybrid model, you inherit all of those concerns plus a second, structurally different family of state that cannot be paged or prefix-shared in the same way.

The State-Storage Bottleneck in Branching Workloads

The problem becomes acute whenever one prefix spawns several continuations:

• Beam search keeps k hypotheses alive, each with its own state.
• Speculative decoding proposes several draft tokens, verifies them in parallel, and must roll back the rejected ones.
• Best-of-n sampling generates n independent completions from a shared prompt.
• Tree-structured reasoning explores branches that split and rejoin.

For attention layers, all of these are tractable. Rolling back a rejected draft is a matter of truncating the KV cache. Sharing a prompt prefix means sharing its cached keys and values. For SSM layers, neither operation is free. A branch point means the state must exist independently for each branch, and a rollback means the state must be restored to an earlier value that no longer exists unless you saved it.

There are only three ways out, and each has a cost:

1. Snapshot aggressively. Keep a copy of the recurrent state at every point you might need to return to. Memory scales with branching factor times checkpoint frequency.
2. Recompute from a checkpoint. Store state rarely, and re-run the layers from the last checkpoint when you need an intermediate state. Memory is low; latency is high.
3. Restructure the storage itself. Change what is stored, when, and in what form, so that reconstruction is cheap enough to be the default.

What a Replay-First Design Implies

The name of the approach points at the third option. "Replay" suggests that reconstruction is no longer an emergency fallback but the normal path, and that the storage layer is designed around making replay cheap.

Read that way, the design question shifts from "how much state do we keep?" to "what is the minimum we must keep so that any required state can be rebuilt quickly?" That framing has several consequences worth thinking through:

• Storage becomes a tunable axis rather than a fixed cost. You can trade memory for recompute along a continuous scale, and choose a point per workload.
• The decode loop gains a new primitive: restore state to position p. Whether that is implemented by loading a snapshot or replaying a short suffix changes latency, not semantics.
• Serving systems can allocate state budgets the way they already allocate KV cache budgets — as a per-request resource with eviction policies, rather than an unavoidable per-sequence allocation.
• Correctness becomes easier to argue about, because a replayed state should be numerically identical (or close) to the stored one. That is a testable property, unlike the fuzzy quality trade-offs of quantization.

Whether the published method matches this description in detail is something the source answers directly; the reasoning above is a framework for reading it, not a substitute for it.

Requirements

Before experimenting, confirm that your environment can support hybrid decoding at all. The realistic prerequisites are:

• A CUDA-capable GPU with enough device memory for the weights plus the largest state footprint you intend to allow. Start with a model that fits comfortably, then shrink the budget.
• A current Python 3.x interpreter supported by your chosen framework version. Mixing a bleeding-edge Python with pinned CUDA wheels is the most common source of setup failure.
• PyTorch built for your CUDA driver. A version mismatch between the driver and the wheel produces confusing runtime errors rather than clean failures.
• A framework that understands hybrid architectures. Recent transformers releases support hybrid cache objects; serving engines such as vLLM and TensorRT-LLM handle hybrid models with varying degrees of maturity. Verify support for your specific model before benchmarking.
• Enough host RAM and disk if you plan to checkpoint or offload state. A state tensor that is small on GPU can become large when multiplied across sequences and checkpoints.

Exact version requirements for the ReplaySSM implementation, if one is released, belong to the original post and its accompanying artifacts.

Step-by-step installation

The commands below build a generic environment for hybrid-model inference. They do not install a specific ReplaySSM package — check the source for the official path.

First, verify that the GPU is visible and that the driver is healthy.

nvidia-smi

Create an isolated environment so framework versions do not collide with anything else on the machine.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel

Install PyTorch. Select the index URL that matches your CUDA runtime; the default PyPI wheel may target a different CUDA version than your driver.

pip install torch --index-url https://download.pytorch.org/whl/cu124

Confirm that PyTorch sees the GPU before installing anything else, because every later failure is easier to diagnose from a known-good baseline.

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Install the modeling and serving libraries you intend to compare.

pip install transformers accelerate
pip install vllm

If your hybrid model uses Mamba-style layers, kernel libraries that accelerate linear-recurrent layers are commonly used alongside the main framework. Install them only if the model you are running documents them as a dependency.

pip install flash-linear-attention causal-conv1d

Finally, record the exact versions in a lockfile so that a benchmark result can be reproduced later.

pip freeze > requirements-lock.txt

Configuration

Configuration for hybrid decoding falls into two categories: knobs your framework already exposes, and any switches the ReplaySSM implementation adds. The first category is usable today.

Set the allocator to reduce fragmentation when state tensors of varying sizes are allocated and freed across requests.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Pin the run to a single GPU so that measurements are not confused by placement decisions.

export CUDA_VISIBLE_DEVICES=0

When serving, enable prefix caching and chunked prefill if your engine supports them. Both change how state is shared and scheduled, and both interact with any state-storage strategy.

vllm serve YOUR_HYBRID_MODEL_ID \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

For the second category, consult the source. Any configuration block for ReplaySSM should be read from the official documentation rather than inferred, because field names and defaults determine whether you are measuring the feature at all.

Usage examples

Example 1: Baseline decode throughput

Establish a baseline before changing anything. This script decodes a fixed number of tokens and reports tokens per second, which is the metric most sensitive to state-handling overhead.

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_HYBRID_MODEL_ID"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="cuda"
).eval()

prompt = "Explain why recurrent state is expensive to branch."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

steps = 128
with torch.no_grad():
    out = model(**inputs, use_cache=True)
    next_id = out.logits[:, -1].argmax(-1, keepdim=True)

    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(steps):
        out = model(
            input_ids=next_id,
            past_key_values=out.past_key_values,
            use_cache=True,
        )
        next_id = out.logits[:, -1].argmax(-1, keepdim=True)
    torch.cuda.synchronize()
    elapsed = time.perf_counter() - start

print(f"{steps / elapsed:.1f} tokens/sec")

Example 2: Measuring the state footprint

Peak memory tells you how much headroom remains for larger batches or more branches. Reset the statistics before the measured region so you capture only what the decode loop allocates.

import torch

torch.cuda.reset_peak_memory_stats()
# ... run the decode loop from Example 1 ...
peak_gib = torch.cuda.max_memory_allocated() / 2**30
print(f"peak allocated: {peak_gib:.2f} GiB")

Subtract the weights from this number to estimate the state footprint, then divide by the number of concurrent sequences to get a per-sequence figure. That figure is the number that determines your maximum batch size under a state-storage strategy.

Example 3: Speculative decoding and rollback

This is the workload where storage policy matters most. The structure below is schematic; adapt the cache calls to your framework's API.

# 1. Save a reference to the state after the accepted prefix.
state_checkpoint = cache.snapshot()

# 2. Propose k draft tokens and run them through the model in one pass.
draft_ids = drafter.generate(prefix_ids, k=4)
candidate_logits = target_model.forward_with_cache(draft_ids, cache)

# 3. Accept the longest prefix that matches the target distribution.
accepted = longest_matching_prefix(candidate_logits, draft_ids)

# 4. If nothing was accepted, restore rather than re-run the whole prefix.
if accepted == 0:
    cache.restore(state_checkpoint)
else:
    cache.truncate_to(accepted)

The snapshot and restore calls are where the two strategies diverge. A snapshot-based implementation pays memory at step 1. A replay-based implementation pays recompute at step 4. The right choice depends on your acceptance rate: with a high acceptance rate, rollbacks are rare and cheap replay wins; with a low acceptance rate, snapshots may pay for themselves.

What to Measure Before and After

A state-storage change is only worth adopting if it moves a metric you care about. Track these consistently:

• Time per output token, not just total request latency. Prefill noise hides decode regressions.
• Peak device memory at a fixed batch size, which determines whether you can raise the batch.
• Maximum sustainable concurrency, measured by stepping batch size up until you hit an out-of-memory error or a latency target.
• Acceptance rate, if speculative decoding is in play. It directly controls how often replay is triggered.
• Rollback cost, measured as the latency added by a rejected draft. This is the number that replay-based designs are trying to reduce.

Run each configuration at least three times with a warmup pass discarded. Hybrid decoding measurements are noisy, and single-run comparisons routinely produce misleading conclusions.

Limits and Open Questions

A few honest caveats. The benefit of any state-storage optimization depends on the ratio of attention layers to recurrent layers in your model; a lightly hybridized model may see little change, while a heavily recurrent one may see a lot. Replay introduces recompute, so it only pays when memory or rollback latency is the binding constraint rather than raw FLOPs. And the trade-off curve is workload-specific: the optimal checkpoint interval for best-of-n sampling is unlikely to be optimal for beam search.

Where the published method lands on that curve, and which workloads it targets, are questions the original post addresses directly. Treat the framing here as scaffolding for reading it, not as a substitute.

Conclusion

Hybrid models are not harder to run than transformers; they are harder to schedule. The reason is that they carry two incompatible kinds of memory: an addressable KV cache and an un-addressable recurrent state. Everything that makes modern serving efficient — paging, prefix sharing, speculative rollback — was designed around the first kind.

ReplaySSM's contribution, as framed by its title, is to stop treating the second kind as an unavoidable cost and start treating it as a design variable. If state can be reconstructed cheaply enough, storage becomes a knob rather than a wall, and the branching workloads that hybrid models currently handle poorly become ordinary again.

The practical path is straightforward: build a clean environment, measure decode throughput and peak memory before changing anything, isolate the rollback cost in a speculative decoding loop, and only then decide where on the memory-versus-recompute curve your workload wants to sit.