DeepInfer Explained

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

DeepInfer is DeepSeek's inference framework: the program that runs a trained model and makes its answers. It "powers DeepSeek's large-scale production services" (README.md:3). This open-source release runs one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and on Huawei Ascend 950 NPUs (README.md:5-7, README.md:33-36). It does offline inference: your Python script gives it a list of prompts and gets the answers back (README.md:3). DeepInfer "was originally built primarily on vLLM" (README.md:95), and many files still say "Derived from vLLM; modified by DeepSeek."

This site explains the code at commit 9bd9cf2. To get exactly this version of the code, read The code version this site explains. You need to know Python. You do not need to know machine learning: Part 1 teaches the basics first.

The system map#

The map shows the main parts of DeepInfer and the order in which a request goes through them. Each box is a link. Click a box to open its page. Grey links are pages that are not written yet.

How to read the map. Follow the left column from 1 to 7. Boxes 1 to 4 and 7 run on the CPU. Boxes 5 and 6 run on the device. The right column shows parts that every step uses. The map leaves out many details, for example the image pre-processing and the statistics.

Start here#

Choose the path that matches what you know now.

• A. New to machine learning#

  Read Part 1 in order. Each page is short and has no DeepInfer code.

  1. What an LLM does
  2. Tokens and tokenizers
  3. Inside a transformer
  4. Attention, step by step
  5. Prefill and decode
  6. The KV cache
  7. Sampling: temperature, top-k, top-p
  8. Batching
  9. GPUs, kernels, and speed
  10. Parallelism basics
  11. Mixture of Experts (MoE)
  12. Number formats: BF16, FP8, FP4

  Then follow Tour: Life of a request. Then read the concept pages of the Engine group.

• B. You know how LLMs work#

  1. Follow Tour: Life of a request.
  2. Read the concepts, one group at a time:
     • Engine: step loop, scheduler, paged KV cache, ZOS
     • Model: model map, MLA, MoE routing
     • Decoding: sampler, speculative decoding
     • Scale: expert parallelism, DP votes
     • Platform: NVIDIA and Ascend, TileLang
  3. Then take the other tours: start-up, one step, one layer, one draft round, many GPUs.

• C. You study one part#

  • Engine loop: concept · llm_engine.py · tour
  • Scheduling and memory: scheduler · KV cache · scheduler.hpp
  • Attention: MLA · indexer · compression · attn.py
  • MoE: routing · mega-MoE · EPLB · mlp.py
  • Decoding: sampler · speculative · structured · sampler.py
  • Hardware: platforms · kernels · platform.py
  • Settings: Environment variables · Config classes
  • A file: Code map: the folder tree

How to use this site#

• Levels. Each page has a badge: Beginner, Intermediate or Advanced. A "Before you read" box lists the pages to read first.
• Two depths on each page. A page starts with one sentence, a plain explanation and a diagram. Then it shows the real code. Deep dive boxes give the full detail: tensor shapes, memory layouts, edge cases and environment variables. Click a deep dive to open it.
• Check yourself. Most pages end with 3 to 5 questions. Answer each one, then click "Show answer".
• Search. Press /, then type a word, a file name or an environment variable.
• Glossary. Words with a dotted line, such as KV cache, are glossary terms. Point to them, or tap them on a phone, to see a short definition. Glossary lists all terms.
• Code. Every code excerpt is copied from the repository by the build. The file path above an excerpt, and every file:line link, opens GitHub at commit 9bd9cf2.
• Chinese comments. Many comments in the code are in Chinese. The site translates the useful ones and marks them as translated.
• Grey entries in the menu are pages that are not written yet. Diagrams open at full size when you click them.

What is not in this release#

The README lists three parts that a complete inference service needs, but that this release does not include (README.md:38-42):

1. A KV cache system that keeps the KV cache of old requests for reuse. The Q&A says that this part is tied to DeepSeek's internal systems and is not open source yet (doc/inference_deployment_qa.md:21). The production service stores it on a 3FS storage cluster (doc/inference_deployment_qa.md:141).
2. VE/Prefill/Decode disaggregation (E/P/D): separate services for the image encoder, for prefill and for decode. The README says that "most required engine functionality is already implemented". You can see empty hook methods for it in LLMEngine.
3. Load balancing across data-parallel replicas, for prefill and for decode.

The picture below comes from the deployment Q&A in doc/. It shows the whole production system. This repository contains the engine that runs inside these services, with its expert-parallel load balancer (EPLB). In this release, one NORMAL engine does prefill and decode in the same process (deepinfer/engine/arg_utils.py:41), and the model runs the image encoder itself (deepinfer/model_executor/models/deepseek_vl.py:26-34). The entrypoint, the DeepSeek Recipe, the three load balancers and the external KV cache storage are not in this repository.

The code version this site explains#

This site explains one exact version of the DeepInfer repository. Use this version when you study the code with the site.

To get exactly this version, run these commands. The last command gets the kernel libraries at their pinned commits.

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
git checkout 9bd9cf243fc53ef9afbf8f04d391d989117d6ab9
git submodule update --init --recursive

The repository pins six kernel libraries as git submodules in vendor/ (.gitmodules). This table shows the commit of each one at 9bd9cf2: