Glossary

Referenceabout 17 min read

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

This page lists every term that has a tooltip on this site. Each entry gives a short definition. Many entries link to a page with the full explanation.

A B C D E F G H I K L M N P Q R S T V Z

A#

accelerator also: device

A chip that does AI math much faster than a CPU. In DeepInfer, an accelerator is an NVIDIA GPU or a Huawei Ascend NPU.

AFD also: Attention-FFN Disaggregation

Attention-FFN Disaggregation. A design that runs attention layers and FFN (expert) layers on different groups of devices. DeepInfer does not use it.

attention also: self-attention

The layer step where each token looks at the earlier tokens. It compares its query with their keys and makes a weighted sum of their values.

B#

block also: KV block, cache block, page, physical block

A fixed-size part of the KV cache that holds the keys and values of a fixed number of tokens. In DeepInfer one block holds 128 tokens for all layers.

Block 0 is never given to a sequence; the free list starts at block 1 (csrc/engine/block_manager.hpp:27-35).

Read more: Paged KV cache and block manager

block table also: block_table, page table

A list per sequence that maps its logical block 0, 1, 2 ... to physical block ids in the KV cache pool.

In DeepInfer it is std::vector<int> block_table in KvCacheState (csrc/engine/sequence.hpp:200-207).

Read more: Paged KV cache and block manager

BlockManager also: block manager, state_manager

The C++ object that owns the free list of KV cache blocks. It gives blocks to sequences and takes them back.

Defined in csrc/engine/block_manager.hpp:119-187. DeepInfer makes two: one for KV blocks and one for state slots.

Read more: Paged KV cache and block manager

C#

CANN

Compute Architecture for Neural Networks. Huawei's software stack (driver, runtime, libraries) for Ascend NPUs. It has the same role as CUDA.

The README asks for CANN 9.2 on Ascend (README.md:52).

chunked prefill also: chunk, chunked_prefill

Splitting a long prompt into parts that run in several prefill steps, so that one step never has more than max_num_batched_tokens tokens.

The cut is in csrc/engine/scheduler.hpp:470-475. Each chunk goes through the sampler, but only the token of the last chunk becomes an output (csrc/engine/prepare_sample.cpp:173-174, csrc/engine/scheduler.hpp:1011-1020).

D#

data parallelism also: DP

DP. Several devices run the same model part, and each device works on different requests.

Read more: Environment variables

DFlash also: MTP_module, draft model

The draft module of DeepInfer. In each decode step, the sampler calls it once after the draft check, and it proposes the drafts for the next step.

DFlash.decode_forward in deepinfer/model_executor/models/mtp/dflash.py:378-387.

E#

E/P/D disaggregation also: EPD, VE/Prefill/Decode

A deployment that runs the vision encoder (E), prefill (P) and decode (D) as separate model instances. It is not included in this DeepInfer release.

engine callbacks also: EngineCallbacks, post_process, pre_sampler_sync

Two engine functions that the driver worker calls during a step: pre_sampler_sync after sampling starts, and post_process when a step's results are ready.

The engine passes them in an EngineCallbacks object (deepinfer/engine/llm_engine.py:398-400, deepinfer/worker/worker.py:43-48).

engine step also: step, iteration

One call of LLMEngine.step(): schedule a batch, run the model once on the device, sample, and update the requests. Generation is a loop of steps.

LLM._run_engine() calls step() until no request is unfinished (deepinfer/entrypoints/llm.py:176-186).

Read more: Environment variables

EOS token also: EOS, BOS, end of sequence

End of sequence: a special token that tells the engine to stop a request. In the release tokenizer, EOS has id 1 and BOS (begin of sequence) has id 0.

Checked in deepinfer/hugging_face/tokenizer.py:36-39. The scheduler stops a request at EOS unless ignore_eos is set (csrc/engine/scheduler.hpp:1038-1041).

F#

FP8

An 8-bit floating-point number format. It uses half the memory of BF16. It needs scale factors to keep enough precision.

Read more: Number formats: BF16, FP8, FP4

fragmentation also: internal fragmentation, external fragmentation, wasted slots

Memory that is reserved but not used. With paging, only the last block of each sequence can be partly empty (internal fragmentation).

G#

git submodule also: submodule

A folder in a git repository that points to a commit of another repository. A plain clone leaves the folder empty until you run git submodule update.

DeepInfer has six submodules in vendor/ (.gitmodules). scripts/build_vendor.sh fetches them (scripts/build_vendor.sh:27-28).

H#

HCCL

Huawei Collective Communication Library. It moves tensors between Ascend NPUs. torch_npu adds it to PyTorch as the "hccl" backend.

hidden state also: hidden states, hidden_states

The vector that represents one token inside the model. The embedding makes it from the token id; each layer changes it.

I#

import hook

Code that changes what a Python import statement returns. DeepInfer's npu_ops package uses one to replace two vendor functions on Ascend.

import time also: at import, module level

The moment when Python runs the top level of a module for the first time: module code, class bodies and the default values of function arguments.

A value that code reads at import time does not change when you set the variable after the import. See the import-time table on Environment variables.

Read more: Environment variables

K#

kernel

A small program that runs on the GPU or NPU. One kernel does one operation, for example a matrix multiply, on many values in parallel.

L#

LLM also: large language model, language model

Large language model. A neural network that reads tokens and predicts the next token, one token at a time.

M#

memory profiling also: profile run, profile_run, profile_num_available_blocks

A test forward pass at start-up. It measures the peak device memory of the model, so that the engine knows how much memory is left for the KV cache.

Read more: Environment variables

MTP also: nextn, multi-token prediction

Multi-token prediction: a module that predicts more than one next token. DeepInfer uses DFlash as its MTP module. DEEPINFER_USE_NEXTN_DECODE sets the number of drafts (default 5).

The default is in deepinfer/environ.py:129.

N#

NCCL

NVIDIA Collective Communications Library. It moves tensors between NVIDIA GPUs (all-reduce, all-to-all and others). PyTorch calls it the "nccl" backend.

NPU also: Ascend

Neural Processing Unit. Huawei's name for its Ascend AI chips. In DeepInfer, an NPU does the same work as a GPU.

DeepInfer supports Huawei Ascend 950 NPUs (README.md:36).

P#

paged KV cache also: paging, paged attention, PagedAttention

A KV cache that is split into fixed-size blocks. A sequence gets blocks one at a time, from anywhere in the pool, and a block table records where they are.

Read more: Paged KV cache and block manager

pinned memory also: page-locked memory

CPU memory that the operating system cannot move or swap out. The GPU or NPU can copy from it directly (DMA), so copies are faster.

Platform object also: PLATFORM, platform

The object in deepinfer/platform.py that holds the hardware choices: kernel folder, communication backend, FP8 scale-factor type, memory pinning.

There is one Platform object per process. Code gets it as deepinfer.PLATFORM (deepinfer/__init__.py:53-64).

preemption also: preempt, preempted, recompute

The scheduler takes the KV blocks away from a running sequence because the pool is full. The sequence stops, or waits and is computed again later.

DeepInfer does this in Scheduler::_preempt (csrc/engine/scheduler.hpp:766-790).

Q#

query, key, value also: Q K V, qkv, key, value, query

Three vectors that attention makes for each token. The query asks, the keys of earlier tokens answer, and the values carry the content that is mixed.

R#

rank

The number of one process in a distributed job. In DeepInfer, one rank normally drives one GPU or NPU.

S#

SamplingParams also: sampling parameters, temperature, max_tokens

The settings of one request: temperature, top_p, top_k, max_tokens, seed and more. The engine copies them into the C++ sequence once.

Defined in deepinfer/sampling_params.py:13-53. The C++ side copies the values in SamplingScalars::capture (csrc/engine/sequence.hpp:103-115).

T#

tensor

A multi-dimensional array of numbers, for example a vector or a matrix. A PyTorch tensor is in CPU memory or in GPU/NPU memory.

V#

vendor library also: vendor/

An external kernel library in vendor/ (FlashMLA, DeepGEMM, DeepEP, DeepSelect). It is a git submodule. This site explains only how DeepInfer calls it.

visible devices also: CUDA_VISIBLE_DEVICES, ASCEND_RT_VISIBLE_DEVICES

The devices a process can use, set by CUDA_VISIBLE_DEVICES (NVIDIA) or ASCEND_RT_VISIBLE_DEVICES (Ascend). Device 0 is the first visible one.

Read more: Hardware discovery (deep_cluster)

Z#

ZOS also: zero-overhead scheduling, DEEPINFER_ZERO_OVERHEAD_SCHEDULING

Zero-overhead scheduling. The CPU prepares and post-processes one step while the device runs the next, so the CPU work does not stop the device.

On by default for the NORMAL engine (deepinfer/engine/arg_utils.py:51-55). The worker keeps a decode step and post-processes it during the next step (deepinfer/worker/worker.py:182-187).

Read more: Zero-overhead scheduling (ZOS)