DeepInfer Explained

Code map: the folder tree

Code mapabout 11 min read

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

Purpose#

This page is a map of the whole repository. Use it to find where a part of DeepInfer lives before you read its code. Click a folder to open it. Each folder shows its number of files, its number of lines, a short description and links to the pages that explain it.

  • Total: 342 files and 63,629 text lines at commit 9bd9cf2.
  • The engine package deepinfer/ has 223 files and 47,174 lines: three quarters of all lines.
  • The C++ code in csrc/ has 33 files and 5,908 lines.
  • The kernel libraries are not in the repository. vendor/ holds six empty git submodules.

The folder tree#

Counts include all files below a folder. "Lines" are text lines, as wc -l counts them. Images count as files with 0 lines.

  • DeepInfer/342 files · 63,629 lines

    The repository root. 14 files at the top level (889 lines): the build entry build.sh, setup.py, the Makefile, four requirements*.txt files, pyproject.toml, README.md, CONTRIBUTING.md, LICENSE, .gitmodules, .gitignore and .clang-format.

How the folders depend on each other#

Imports between the top-level folders (panel 1), and the three build steps (panel 2).
Open full size ↗

Read panel 1 from the top. The arrows go from the code that imports to the code that it imports. No arrow points up: deepinfer/ does not import examples/, eval/ or tests/. 77 files in deepinfer/ import tilelang or tile_kernels, and 18 files import a vendor library: 86 files in all, because some files import both.

Panel 2 shows ./build.sh. It installs the Python requirements, builds the vendor wheels, then builds the C++ extension (build.sh:13-25).

How the numbers were measured#

The counts come from the checkout at commit 9bd9cf2, without the .git folder. Run these commands in the repository root to check them:

find . -path ./.git -prune -o -type f -print | wc -l          # 342 files
find deepinfer -type f | wc -l                                # 223 files
find deepinfer -type f -print0 | xargs -0 grep -Il "" | tr '\n' '\0' | xargs -0 cat | wc -l   # 47174 text lines

The import counts come from grep -rlE "^\s*(from|import) (tilelang|tile_kernels)" deepinfer and the same search for deep_ep, deep_gemm, flash_mla and deep_select.

  • engine/llm_engine.py: the LLMEngine classLLMEngine starts the workers, sizes the KV cache, owns the C++ scheduler, and runs one engine step each time LLM calls step().
  • entrypoints/llm.py soon
  • Hardware platforms: NVIDIA and AscendDeepInfer runs on NVIDIA GPUs and on Huawei Ascend NPUs. At start-up it makes one Platform object, and the shared code reads all hardware choices from it.
  • The Python and C++ boundary soon
  • Tour: Life of a requestFollow one request from llm.generate() to its output tokens in 13 stops: tokenizer, admission, C++ scheduler, batch metadata, worker step, forward pass, KV cache, sampler, draft tokens and output objects.