Batching

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

The idea#

Why run many sequences together?#

In each step, the model reads all its weights from device memory. For one sequence, a decode step computes very little: one new token (or a few). So the device spends most of the step on memory reads, and its math units wait.

A batch puts the tokens of many sequences into the same step. The weights come from memory once and serve every token in the batch. Up to a limit, 64 tokens take about the same time as 1 token. The page GPUs, kernels, and speed gives the numbers.

An engine must answer three questions in every step:

1. Which sequences run? This is the job of the scheduler.
2. How are their tokens stored? With padding, or packed end to end.
3. How many tokens fit? A token budget limits the size of one step. A long prompt must then be split.

Static and continuous batching#

With static batching, the engine takes a group of requests and runs them until the last one ends. A sequence that ends early keeps its place in the batch, but it does no work. New requests wait for the next group.

With continuous batching, the engine makes a new batch in every step. A finished sequence leaves at once. A waiting request can join at the next step. This is also called iteration-level batching.

A worked example. The batch has 3 slots. Five requests want 3, 7, 4, 5 and 2 output tokens (A to E). One step makes one token per sequence.

The useful work is the same: 3 + 7 + 4 + 5 + 2 = 21 tokens. Continuous batching finishes in 8 steps instead of 12, and D and E get their tokens much earlier.

An everyday comparison#

Think of a tour bus and a city tram. The tour bus leaves with a full group and comes back only when the last person is ready. Empty seats stay empty for the whole tour. The tram stops at every station. People get off, and new people get on at once. Continuous batching is the tram.

The comparison stops here: a tram seat costs the same for every person, but a batch slot does not. A new sequence must first read its whole prompt (prefill). That step costs much more than one decode token.

Padded and packed batches#

The sequences of a batch have different lengths. A simple tensor has rectangular rows, so the engine has two choices:

• Padded. Make a tensor [batch, max_len]. Fill the short rows with pad tokens. The device computes the pad tokens too, and then drops the results.
• Packed (varlen). Put all tokens end to end in one tensor [total_tokens]. A second tensor, cu_seqlens, marks where each sequence starts. This is a packed batch.

A worked example. Three prompts have 5, 2 and 3 tokens.

padded:  [3, 5] = 15 slots, 5 of them padding (33 % waste)
packed:  [10]   = 10 slots, no padding
         cu_seqlens = [0, 5, 7, 10]
         sequence 0 = tokens 0..4, sequence 1 = tokens 5..6, sequence 2 = tokens 7..9

Matrix multiplies do not care about sequence boundaries: each token row is independent. Only attention needs the boundaries, because a token must look only at tokens of its own sequence. Varlen attention kernels read cu_seqlens to find them.

Chunked prefill#

One step has a token budget. A prompt can be longer than the budget. With chunked prefill, the engine splits the prompt into chunks and runs one chunk per step. The KV cache keeps the chunks that are done, so the next chunk can attend to them.

Example: the budget is 4,096 tokens and a prompt has 10,000 tokens. The chunks are 4,096, 4,096 and 1,808 tokens, in three prefill steps.

Chunking also keeps each step short. Without chunks, one very long prompt makes one very long step, and all decoding sequences wait for it.

Why decode is memory-bound#

A prefill step has many tokens per sequence. A decode step has only one token per sequence (or a few, with speculative decoding). So in decode, the batch size is the number of running sequences.

Take one weight matrix of 4,096 × 4,096 BF16 values: 32 MiB. The table uses illustrative device numbers: 8 TB/s memory bandwidth and 2,250 TFLOP/s of BF16 math.

The step takes about as long as the larger of the two times. With 1 or 64 tokens, the memory read sets the time: the multiply is memory-bound. So 64 tokens cost almost the same as 1 token. With 512 tokens, the math sets the time. This is why decode needs large batches. It is also why the KV cache size matters so much: the cache limits how many sequences can run at the same time.

Mixed steps or separate steps#

Many engines mix prefill chunks and decode tokens in one step. Others keep the two kinds apart: a step is a prefill step or a decode step. With separate kinds, every decode step has the same simple form, one or a few tokens per sequence. The cost is that a waiting prompt must wait for a prefill step, and the decoding sequences pause during that step.

DeepInfer keeps the two kinds apart. It also has engine types that do only one kind: a PROMPT engine does only prefill, and a COMPLETION engine does only decode (csrc/engine/scheduler.hpp:37-43). The deployment notes describe separate prefill and decode machines in production (doc/inference_deployment_qa.md:31).

The picture#

Read panel 1 from left to right: each column is one step. A dashed cell is a slot with no work. Panel 2 shows the same three prompts in the two layouts. Panel 3 shows what one DeepInfer step holds; the budget of 4,096 tokens is only an example.

In DeepInfer#

This is a basics page, so this part only gives pointers. The pages The scheduler and Input metadata (varlen batching) explain the full code.

1. The C++ scheduler makes one batch per step, and the batch has one kind. In a NORMAL engine, _schedule() returns a prefill batch or a decode batch, never a mix (csrc/engine/scheduler.hpp:288-341). All data-parallel ranks vote, so all of them do the same kind of step. See Data parallelism and step votes.
2. Prefill fills a token budget, decode takes all running sequences. A prefill step takes waiting prompts in order until max_num_batched_tokens is full. It cuts the last prompt to the rest of the budget (csrc/engine/scheduler.hpp:467-475). A decode step takes every running sequence, up to max_num_seqs (csrc/engine/scheduler.hpp:622-643).
3. The batch is packed. The model gets the tokens end to end, with cu_seqlens_q and cu_seqlens_k of shape [num_seqs + 1]:

    cu_seqlens_q: torch.Tensor | None = None
    """cumsum(seqlen_q). Shape: [num_seqs + 1]."""
    cu_seqlens_k: torch.Tensor | None = None
    """cumsum(seqlen_k). Shape: [num_seqs + 1]."""

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

seqlen_q     = [5, 2, 3]            total_q = 10
seqlen_k     = [5, 102, 3]
cu_seqlens_q = [0, 5, 7, 10]
cu_seqlens_k = [0, 5, 107, 110]
positions    = [0 1 2 3 4 | 100 101 | 0 1 2]

def _get_max_num_batched_tokens(
    max_model_len: int,
    max_num_batched_tokens: int | None,
    max_num_seqs: int,
) -> int:
    max_num_decoding_tokens = max_num_seqs * (1 + envs.DEEPINFER_USE_NEXTN_DECODE)
    if max_num_batched_tokens is not None:
        assert max_num_batched_tokens >= max_num_decoding_tokens
        return max_num_batched_tokens
    return max(max_model_len, 4096, max_num_decoding_tokens)

            bool is_any_point = (decode_step != 0) && (decode_step % prefill_any_attempt_interval == 0);
            bool is_all_point = (decode_step % prefill_attempt_interval == 0);
            if (!is_any_point && !is_all_point) {
                return _schedule_decode(acquire);
            }
            bool want = (get_gpu_cache_usage() <= GPU_CACHE_PREFILL_WATERMARK) && _has_prefill_waiting();
            bool do_prefill;
            if (is_any_point) {
                do_prefill = (_dp_allreduce(want ? 0 : 1, c10d::ReduceOp::MIN) == 0); // ANY：任一 rank 想做即触发
            } else {
                do_prefill = (_dp_allreduce(want ? 1 : 0, c10d::ReduceOp::MIN) == 1); // ALL：全员想做才触发
            }
            if (!do_prefill) {
                return _schedule_decode(acquire);
            }

What can go wrong#

• A prompt that is too long. Without chunked prefill, a prompt longer than max_num_batched_tokens is ignored, with the warning "Input prompt ... is too long" (csrc/engine/scheduler.hpp:377-384). Turn on chunked_prefill, or increase the budget.
• A budget that is too small. If you set max_num_batched_tokens below max_num_seqs × (1 + DEEPINFER_USE_NEXTN_DECODE), start-up fails on an assert (deepinfer/config.py:224-226). It must also be at least one block (128 tokens) (csrc/engine/scheduler.hpp:116).
• Slow first token while many sequences decode. A new prompt can wait up to 70 decode steps for a prefill point (csrc/engine/scheduler.hpp:308-312). With more than one DP rank, the ALL vote can fail; then the prompt waits for the next ANY point, up to 100 decode steps. Lower DEEPINFER_PREFILL_ATTEMPT_INTERVAL for a lower first-token latency, at some cost in decode speed.
• A full cache blocks new prompts. Above 90 % cache usage, a rank does not want a prefill (csrc/engine/scheduler.hpp:313). The running sequences must finish first.

Why it matters. Batch size is the main control of the speed of an inference engine. Large decode batches make the device do useful math instead of waiting for memory. Packed batches avoid wasted work. The scheduler decides which tokens go into each step, so most speed settings are batching settings.