Prefill and decode

Arithmetic intensity is the math per byte: FLOPs divided by bytes moved. Use the rule of thumb from What an LLM does: about 2 FLOPs per weight per token. All numbers below are illustrative, for a dense model with 10 billion weights in BF16 (2 bytes each).

Take an illustrative device with 2,000 TFLOP/s and 8 TB/s. It can do 2,000 / 8 = 250 FLOPs in the time it reads one byte (its ridge point).

1. Prefill: 2,000 FLOPs per byte is above 250, so the math sets the time: 4 × 10¹³ / 2 × 10¹⁵ = 20 ms.
2. Decode with 64 sequences: 64 is below 250, so the memory sets the time. 2 × 10¹⁰ B / 8 × 10¹² B/s = 2.5 ms per step, for 64 tokens.
3. Decode with 1 sequence: also 2.5 ms, but for 1 token. The device does almost no math.

        switch (engine_type) {
        case EngineType::PROMPT:
            // PROMPT 引擎只做 prefill，即使没调度到请求也返回（空的）prefill 结果
            return _schedule_prefill(acquire);
        case EngineType::COMPLETION:
            // COMPLETION 引擎只做 decode
            return _schedule_decode(acquire);
        case EngineType::NORMAL: {
            // NORMAL 引擎：决定这一步全组做 prompt 还是 decode。dp 内必须同型，靠投票统一。
            //
            // 两类尝试点：
            //   ALL 点（每 prefill_attempt_interval 个 decode step）：全员都有 prefill 活才做，
            //     防止一个 rank 干活、其余 rank 陪跑空 prompt step。
            //   ANY 点（每 prefill_any_attempt_interval 个 decode step）：一个 rank 有活就一起做，
            //     给孤立 rank 兜底——它的 prefill 最多等这么多个 decode step。
            //   两类点重合时按 ANY 算。
            //
            // prompt step 会把 decode_step 清零（见 schedule()），所以刚做完 prefill 的下一步
            // decode_step==0，又落在 ALL 点上：想连着 prefill 同样要全员有活。
            // decode_step 各 rank 一致，投票的次数和语义全组对得上，不会卡在 allreduce。
            bool is_any_point = (decode_step != 0) && (decode_step % prefill_any_attempt_interval == 0);
            bool is_all_point = (decode_step % prefill_attempt_interval == 0);
            if (!is_any_point && !is_all_point) {
                return _schedule_decode(acquire);
            }
            bool want = (get_gpu_cache_usage() <= GPU_CACHE_PREFILL_WATERMARK) && _has_prefill_waiting();
            bool do_prefill;
            if (is_any_point) {
                do_prefill = (_dp_allreduce(want ? 0 : 1, c10d::ReduceOp::MIN) == 0); // ANY：任一 rank 想做即触发
            } else {
                do_prefill = (_dp_allreduce(want ? 1 : 0, c10d::ReduceOp::MIN) == 1); // ALL：全员想做才触发
            }
            if (!do_prefill) {
                return _schedule_decode(acquire);
            }
            std::unique_ptr<SchedulerOutputs> outputs;
            if (want) {
                outputs = _schedule_prefill(acquire);
            }

            // 第二轮投票：确认这轮真有 rank 排出了 prefill。want 只看了 waiting 非空，
            // 但 _schedule_prefill 仍可能空手（slot 满、KV 块不够）。
            // 全组都空手时必须转 decode：硬做空 prompt step 会把 decode_step 清零，
            // 下一步接着投票、接着空手——running 满 + waiting 非空就死循环了。
            bool any_scheduled = (_dp_allreduce(outputs ? 0 : 1, c10d::ReduceOp::MIN) == 0);
            if (!any_scheduled) {
                return _schedule_decode(acquire);
            }
            if (outputs) {
                return outputs;
            }
            // 组内确有 rank 在做 prompt：本 rank 陪跑空 prompt step 保持步型一致
            return _make_outputs({}, true, 0, {}, {}, {}, 0);
        }

1. Lines 288-294: an engine of type PROMPT does only prefill; COMPLETION does only decode. These two types are for separate prefill and decode machines.
2. Lines 308-312: a NORMAL engine counts decode steps in decode_step. Between attempt points, it always does decode.
3. Line 313: this rank wants a prefill if cache usage is at most 90 % and the first waiting request is schedulable.
4. Lines 315-319: the data-parallel ranks vote with an all-reduce (MIN). At an ANY point one "yes" is enough; at an ALL point every rank must say yes.
5. Lines 324-338: if the vote says yes, the rank schedules a prefill. A second vote makes sure that at least one rank found work.

Worked example (one rank, defaults 70 and 100). A long decode is running, and the counter decode_step is 5 when a new request arrives.

1. Counter values 5 to 69: no attempt point, so each step is a decode step. The request waits.
2. Counter value 70: an ALL point. The rank wants a prefill, and the vote of one rank says yes. This step is a prefill step for the new request.
3. The counter goes back to 0. The next step is again an ALL point. If more requests wait, it is also a prefill step.

So a new request can wait up to 69 decode steps for its prefill. The page Data parallelism and step votes explains the votes with several ranks.

One step may hold at most max_num_batched_tokens tokens. The default is the largest of max_model_len, 4,096 and max_num_seqs × (1 + drafts) (deepinfer/config.py:219-228). With the default max_num_seqs = 256 (deepinfer/engine/arg_utils.py:36) and 5 drafts, the last term is 256 × 6 = 1,536.

A long prompt can be larger than the budget that is left in the step. With chunked prefill, the scheduler cuts the prompt and runs the rest in later prefill steps (csrc/engine/scheduler.hpp:470-475). Chunked prefill is off by default (deepinfer/engine/arg_utils.py:42). Without it, a prompt that does not fit in the rest of the budget waits for a later step (csrc/engine/scheduler.hpp:504-507). Also, the prompt limit is the smaller of max_model_len and max_num_batched_tokens (csrc/engine/scheduler.hpp:112), and the config refuses a budget below max_model_len (deepinfer/config.py:208-212).

Worked example (chunked prefill on, budget 8,192, no context parallelism). A prompt of 20,000 tokens runs in three prefill steps, as chunks of 8,192, 8,192 and 3,616 tokens. Every chunk writes its keys and values to the cache. Only the last chunk produces the first output token.

The deployment notes describe the production prefill budget: 32,768 tokens per step, with the chunk size solved from a time budget (doc/inference_deployment_qa.md:115). That load balancer is not in this repository.

With speculative decoding, a decode step does not send one token per sequence. It sends the last accepted token plus the draft tokens that the draft module proposed in the step before. The main model checks all of them in one step. This turns part of decode into a small prefill: several tokens per sequence share one read of the weights.

• Number of tokens. Without the confidence scheduler, a sequence sends 1 + len(draft_tokens) tokens (csrc/engine/mtp_varlen.hpp:153-157). That scheduler, DEEPINFER_ENABLE_MTP_VARLEN, is off by default (deepinfer/environ.py:131). The sampler checks that this is at most nextn + 1 (csrc/engine/prepare_sample.cpp:212).
• After prefill. The draft list starts with placeholder ids, vocab_size − 1, one per draft (deepinfer/model_executor/layers/mtp_info_pool.py:9-11).
• Sampling. In a decode step, the sampler gets every row of the sequence, not only the last one (csrc/engine/prepare_sample.cpp:202-206). It accepts or rejects each draft.

The page Speculative decoding explains the accept rule. The production numbers: the draft module proposes 5 tokens, and dynamic verification checks 3.2 to 4.4 of them on average (doc/inference_deployment_qa.md:243, doc/inference_deployment_qa.md:247).

Because the two phases stress the hardware differently, large services often run them on different machines. The DeepSeek deployment notes describe:

• ordinary 8-GPU machines for prefill, and 64- to 128-GPU supernodes for decode (doc/inference_deployment_qa.md:11);
• prefill with expert parallelism 2 to 8, decode with expert parallelism 64 to 128 (doc/inference_deployment_qa.md:47-49).

The engine types PROMPT and COMPLETION exist for this split (csrc/engine/scheduler.hpp:288-294). But this release does not include the parts that connect them: the README lists prefill/decode disaggregation as "not included in this release" (README.md:38-41). An offline LLM runs a NORMAL engine that does both.