FAQ

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

Each answer is short and links to the page with the full explanation. Every statement about the code has a citation at commit 9bd9cf2. "QA doc" means doc/inference_deployment_qa.md.

• Is DeepInfer AFD? · What is not in this release? · Why does my first request wait about 70 steps?
• Why does step() return the previous step's output? · Why must I clone RequestOutput?
• Which model sizes are in the repo? · Which hardware does it run on?
• Where do docs and code disagree? · Known open questions · How do I get exactly this code version?
• Questions from the deployment Q&A

Is DeepInfer AFD (Attention-FFN Disaggregation)?#

No. In AFD, attention layers and FFN layers run on different groups of devices. In DeepInfer, every device runs attention and its share of the MoE experts. Hidden states never go to a separate FFN group: the MoE layer sends tokens to the devices of their experts with expert-parallel dispatch and combine (deepinfer/model_executor/models/v4_modules/mlp.py:461-511). One deployment also uses one hardware type: each process picks one platform, and its device groups use NCCL on NVIDIA or HCCL on Ascend (deepinfer/platform.py:222-227, deepinfer/model_executor/parallel_utils/parallel_state.py:82). The only split that the docs describe is between vision encoder, prefill and decode instances (E/P/D), and that split is not in this release (README.md:38-42). Read Is DeepInfer AFD? for the four reasons.

What is not in this release?#

The README names three parts of a full service that the release does not include (README.md:38-42):

• a KV cache system (the QA doc describes a 3FS store with prefix matching, doc/inference_deployment_qa.md:137-157);
• VE / prefill / decode disaggregation ("most required engine functionality is already implemented");
• load balancing across data-parallel replicas for prefill and decode.

The code agrees, and shows more gaps:

Why does my first request wait about 70 steps?#

Three rules combine. This site derived the result from the code and did not run it.

1. A NORMAL engine tries a prefill only at attempt points: every 70 decode steps (ALL vote) or every 100 (ANY vote) (csrc/engine/scheduler.hpp:308-319, deepinfer/environ.py:174-178).
2. The only model class has a vision part, so the engine has a multimodal processor, and LLM gives every request multimodal data, also a text-only request (deepinfer/entrypoints/llm.py:158-159). A sequence with multimodal data starts with schedulable = False (csrc/engine/pybind_sequence.cpp:35). Only the multimodal processor's fetch() makes it schedulable.
3. In each step, schedule() runs before fetch() (deepinfer/engine/llm_engine.py:885, deepinfer/engine/llm_engine.py:939-941). So step 1 (an ALL point) finds no schedulable request.

So step() calls 1 to 70 are empty decode steps, and call 71 is the earliest prefill. With more than one DP rank, the ALL vote can fail, so the wait can be up to 100 steps (the ANY point). An empty step runs the model on one placeholder sequence (deepinfer/worker/worker.py:121-126). To change the intervals, set DEEPINFER_PREFILL_ATTEMPT_INTERVAL and DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL before the first import of deepinfer.config, because they are read at import time (deepinfer/config.py:193-194). Read Tour: Life of a request, stop 5, deep dive, and Data parallelism and step votes.

Why does step() return the previous step's output?#

Because of zero-overhead scheduling (ZOS), which is on by default for the NORMAL engine (deepinfer/engine/arg_utils.py:51-55). The worker starts forward pass N on the device, then post-processes step N−1 on the CPU while the device works (deepinfer/worker/worker.py:143-144). A ZOS decode step is kept for the next call (deepinfer/worker/worker.py:182-187). So call N returns the tokens of decode step N−1. Prefill steps return their token at once. After the last step, LLM calls flush_inflight_reqs() to get the outputs that the worker still holds (deepinfer/entrypoints/llm.py:177-178). Set DEEPINFER_ZERO_OVERHEAD_SCHEDULING=0 to turn ZOS off. Read Zero-overhead scheduling (ZOS).

Why must I clone RequestOutput?#

step() returns a NumPy array of uint64 raw pointers into a C++ RequestOutputPool with 10000 slots (csrc/engine/outputs.hpp:45-81). The engine gives the objects back to the pool in the next pre_sampler_sync, so they become invalid during the next step() (deepinfer/engine/llm_engine.py:830-831). RequestOutput.clone(ptr) makes an independent copy (csrc/engine/pybind_output.cpp:17-18). LLM._run_engine does it for every pointer (deepinfer/entrypoints/llm.py:179-182). Code that calls llm.llm_engine.step() directly must do the same. If the pool is full, the engine raises "RequestOutputPool exhausted (max_size=10000)" (csrc/engine/outputs.hpp:65).

Which model sizes are in the repo?#

None. All sizes come from the checkpoint config.json, which the parser reads at start-up (deepinfer/hugging_face/config.py:237-311). The repo names the model, DeepSeek-V4.1-Flash (README.md:5-7), and the default checkpoint id deepseek-ai/DeepSeek-V4.1-Flash-1017 (examples/_config.py:32). These values are not in the repository; pages on this site that show them mark them as examples:

• number of layers, hidden size, attention heads, q_lora_rank, o_lora_rank, o_groups, head_dim;
• vocabulary size (129280 is only inferred from the image-token ids, deepinfer/hugging_face/tokenizer.py:48-56);
• sliding window, compress ratios per layer, indexer top-k, indexer heads and head size;
• routed and shared expert counts, experts per token, routed_scaling_factor;
• hc_mult, number of DFlash layers (num_nextn_predict_layers), the KV-sharing source lists.

The code fixes only a few values: the block size of 128 tokens (deepinfer/model_executor/model_metadata_utils.py:133-135), one KV head, SwiGLU and BF16 (deepinfer/hugging_face/config.py:51-59), FP8 weights with FP4 experts in 32 × 32 blocks (lines 254-262), and a RoPE part of 64 values in the layers with an indexer (deepinfer/model_executor/models/v4_modules/mla_attn.py:121). The QA doc gives a few numbers, for example "the first 20" stored layers (doc/inference_deployment_qa.md:163).

Which hardware does it run on?#

The README lists NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs (README.md:33-36). The code enforces this at the very end of start-up: get_bf16_tflops_limit() returns a peak only for Ascend and for NVIDIA devices with compute capability 10.x, and raises RuntimeError("unknown GPU type") for all others (deep_cluster/hardware/device.py:151-156). The engine always calls it when it makes the statistics logger (deepinfer/engine/llm_engine.py:298-305). So on an older GPU, start-up fails after it loads the weights and captures the graphs. Build with DEEPINFER_ACCELERATOR=nvidia or ascend, and keep it exported (README.md:59-62). Read Hardware platforms: NVIDIA and Ascend and Tour: Start-up, stop 14.

Where do docs and code disagree?#

This site always describes the code. The table lists each known difference with both citations.

Known open questions#

These are questions that this site could not close by reading the code. Each one gives the evidence.

Do aborted chunked prompts leak KV blocks?#

Possibly. This is derived from the code and not run. It needs chunked_prefill=True and an abort between two chunks.

1. After a chunk that is not the last one, the prompt goes back to the front of waiting and keeps its blocks and its state slot (csrc/engine/scheduler.hpp:1015-1019).
2. abort_sequence marks a waiting sequence FINISHED_ABORTED and does not free anything (csrc/engine/scheduler.hpp:144-162).
3. The next prefill schedule moves it to ignored_seqs (csrc/engine/scheduler.hpp:362-367); a decode schedule does the same (csrc/engine/scheduler.hpp:569-577). Neither path frees blocks.
4. An ignored sequence gets one output with its finish reason (csrc/engine/outputs.hpp:297-304). Python frees only finished_seqs, through finalize_seq (deepinfer/engine/llm_engine.py:844-846). abort_request frees only multimodal pool handles (deepinfer/engine/llm_engine.py:651-656).

In this release the offline LLM never calls abort_request. The engine calls it only for multimodal failures (deepinfer/engine/llm_engine.py:939-945), and those requests own no blocks. So the leak, if real, needs a caller that aborts a request in the middle of a chunked prefill.

What does DeepSelect's sorted_index=True guarantee?#

The sparse indexer calls deep_select.interface.topk with sorted=False and sorted_index=True, and then converts the indices to physical cache slots (deepinfer/model_executor/models/v4_modules/index_attn.py:494-513). DeepSelect is a git submodule that is not checked out here, so the exact meaning of sorted_index, and whether the later kernels depend on the order, cannot be confirmed from this repository. On Ascend the same call goes to a wrapper that passes through to the library when return_value=False (deepinfer/npu_ops/deep_select.py:20-26).

Does the combine transfer overlap the shared expert?#

The code launches the shared expert before the combine. The combine gets an event captured before the shared expert, and limit_num_sms keeps some SMs free for communication (deepinfer/model_executor/models/v4_modules/mlp.py:600-609). Whether the device really runs the combine transfer at the same time as the shared expert is decided inside DeepEP, which is not checked out. So this site does not claim either answer. Read Expert parallelism: dispatch, grouped GEMM, combine.

How do I get exactly this code version?#

Clone the repository, check out commit 9bd9cf243fc53ef9afbf8f04d391d989117d6ab9, and get the submodules at their pinned commits. The home page has the commands and the submodule table: The code version this site explains.

Questions from the deployment Q&A#

The QA doc describes DeepSeek's production service. Some of it is outside this repository. Each answer says what the doc says and what this release has.

Which parallel sizes does production use?#

Prefill: EP 2–8 with DP 2–8, the smallest that memory allows. Decode: EP 64–128 ("Wide-EP"), the largest that the GPUs allow. The vision encoder uses one GPU per instance (doc/inference_deployment_qa.md:45-53). Attention uses DP with #DP = #EP (doc/inference_deployment_qa.md:33). In this release, one NORMAL engine does prefill and decode, and the example script sets dp = ep // cp (examples/offline_inference.py:27). Read Parallelism basics and Expert parallelism: dispatch, grouped GEMM, combine.

What is the smallest deployment?#

The doc says 1P + 1D works, and the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs (doc/inference_deployment_qa.md:65-67). This release has no P/D split, so the offline example runs on one host with 8 devices by default (examples/offline_inference.py:13).

How many redundant experts does production use?#

Prefill starts from 4 per GPU with a 100-step window and a rearrangement every 1,000 steps; decode starts from 1 per GPU with a 1,000-step window and a rearrangement every 10,000 steps (doc/inference_deployment_qa.md:95-103). Both round up so that the expert count divides by ep_size, as the code does (deepinfer/model_executor/eplb/__init__.py:21-28). In this release the count is a total, not per GPU: DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, default 0 (deepinfer/environ.py:114), and rearrangement is off by default (DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL = 0, deepinfer/environ.py:116). Read Expert load balancing (EPLB).

Where does the sliding-window cache live, and how large is one slot?#

Not in the paged blocks: each sequence has one fixed state slot (doc/inference_deployment_qa.md:163). The stored part is "20 layers × 134 tokens × 528 B = 1,415,040 B per sequence" (doc/inference_deployment_qa.md:169). In the code, the ring has window + DEEPINFER_MAX_BACKTRACKING_TOKENS rows (deepinfer/worker/cache_engine.py:88-93) and one entry is 512 B of FP8 data plus 16 B of scale (deepinfer/worker/cache_engine.py:216-218). Read Sliding-window attention.

Where is FP4 used?#

The doc lists three places: the main KV cache (per-16 blocks, e4m3 scales), the indexer keys (MXFP4) and the routed-expert weights (MXFP4) (doc/inference_deployment_qa.md:215-221). In the code, the expert format must be FP4 (deepinfer/hugging_face/config.py:254-262), and the main KV cache is FP4 unless config.json sets use_fp4_sparse_kv to false (deepinfer/hugging_face/config.py:303). Read Number formats: BF16, FP8, FP4.

How much does DSpark gain, and how do I turn it on?#

The doc reports about 30 % fewer verified drafts per step (5.00 → 3.2–4.4) and +4–7 % throughput at about 60 sequences per rank, from EP64/DP64 benchmark data (doc/inference_deployment_qa.md:245-256). These are benchmark figures. Turn it on with DEEPINFER_ENABLE_MTP_VARLEN=1 (README.md:24). It needs CUDA graphs: start-up measures the decode time per graph size only when graphs are on (deepinfer/worker/base.py:276-281), and with no measured curve the scheduler verifies all drafts (csrc/engine/mtp_varlen.hpp:153-157). Read DSpark: DFlash drafts and confidence-scheduled verification.

How does asynchronous scheduling work?#

While the device runs forward and sample, the CPU schedules the next step, launches it, and post-processes the step before (doc/inference_deployment_qa.md:268-274). It is on by default (README.md:26). See the answer above for the step numbering, and Zero-overhead scheduling (ZOS).

How do I see which kernels run?#

The doc says: profile with examples/benchmark.py and read the trace (doc/inference_deployment_qa.md:225-227). With PROFILE=1, the benchmark writes one trace with 2 prefill steps and 3 decode steps (examples/README.md:47). For a running engine, DEEPINFER_PROFILE_HTTP_ENABLE=1 starts an HTTP endpoint that writes a profiler trace for a number of steps (deepinfer/engine/profile_server.py:1-11). Read engine/stats_logger.py: step statistics, MFU and moe_lb.

When is Mega-MoE used?#

The doc calls Mega-MoE the general solution inside one scale-up domain (doc/inference_deployment_qa.md:229-231). The code uses it when the MMA type is supported and the EP group fits in one NVLink domain; on Ascend the domain check is always true (deepinfer/model_executor/models/v4_modules/mlp.py:184-188, deepinfer/model_executor/parallel_utils/parallel_state.py:130-134). With the released checkpoint format, the MMA type is always fp8xfp4. Read Mega-MoE: the whole MoE layer in one call.

Is the 3FS KV cache store in this repository?#

No. The doc describes a dedicated 3FS cluster and its time-to-live of 3 to 7 days (doc/inference_deployment_qa.md:129-153). The KV cache management code is "not yet fully open-sourced" (doc/inference_deployment_qa.md:21). In this release the cache lives only in device memory, and the engine frees the blocks of a sequence when it finishes (deepinfer/engine/llm_engine.py:804-806, deepinfer/engine/llm_engine.py:844-846). Read Paged KV cache and block manager.