Config classes

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

Use this page to look up one setting. The page deepinfer/config.py: the four config classes explains the code of deepinfer/config.py. The page Environment variables lists the DEEPINFER_* environment variables, which are a second, separate way to set the engine.

From keyword arguments to config objects#

1. LLM(model, **kwargs) adds its own defaults and makes an EngineArgs (deepinfer/entrypoints/llm.py:67-81).
2. EngineArgs.__post_init__ uses the model path as the tokenizer path if you give no tokenizer. It also sets two environment variables if they are not set (deepinfer/engine/arg_utils.py:48-60).
3. LLMEngine.from_engine_args calls create_engine_configs() (deepinfer/engine/llm_engine.py:460-469). That method makes the four objects in this order: ModelConfig, CacheConfig, ParallelConfig, SchedulerConfig (deepinfer/engine/arg_utils.py:62-114).
4. The engine gives the same four objects to the driver Worker. It gives copies to each CP follower process as spawn arguments (deepinfer/engine/llm_engine.py:377-413).
5. Some fields are empty at first. Other files fill them later, during start-up (the "Set later" rows below).

EngineArgs#

EngineArgs is a dataclass in deepinfer/engine/arg_utils.py (deepinfer/engine/arg_utils.py:17-46). The column "LLM default" shows the value that you get through the offline class LLM, when it is different.

LLM-only arguments. decode_sequence (default True) is not an engine setting. LLM keeps it and uses it to decode the output text (deepinfer/entrypoints/llm.py:64, deepinfer/entrypoints/llm.py:82). All other LLM keyword arguments go to EngineArgs through **kwargs. An unknown name raises TypeError from the dataclass constructor. LLM also sets TOKENIZERS_PARALLELISM=false if it is not set (deepinfer/entrypoints/llm.py:69).

Side effects of __post_init__. It sets DEEPINFER_ZERO_OVERHEAD_SCHEDULING to 1 for NORMAL and COMPLETION engines, and DEEPINFER_USE_MULTIMODAL_POOL to 1 except for COMPLETION. It sets each one only if the variable is not set, and then calls environ.rearm (deepinfer/engine/arg_utils.py:51-60, deepinfer/engine/engine_type.py:15-17). So ZOS is on when ModelConfig computes runtime_max_model_len.

ModelConfig#

Constructor: deepinfer/config.py:43-80. The constructor also loads the model config.json (the last row).

get_vocab_size() returns model_cfg.gpt.vocab_size, a value from the checkpoint, not from the repository (deepinfer/config.py:121-122).

Fields copied into the model config. _load_model_cfg writes max_num_seqs, max_num_batched_tokens, runtime_max_model_len and engine_type into model_cfg.gpt (deepinfer/config.py:82-94). The model layers get only model_cfg.gpt, so they read these copies.

Copies for CP followers. A CP follower gets ModelConfig as a spawn argument, so Python pickles it. __getstate__ leaves out model_cfg and keeps the four copied values. __setstate__ loads config.json again in the new process and writes the four values back:

    def __getstate__(self) -> dict:
        # model_cfg 不可 pickle，改为只传递必要的运行时标量，
        # 由 worker 进程在 __setstate__ 中重新 load model_cfg 后回填。
        state = self.__dict__.copy()
        model_cfg = state.pop("model_cfg", None)
        overrides: dict = {}
        if model_cfg is not None:
            overrides = {k: model_cfg.gpt[k] for k in self._GPT_RUNTIME_ATTRS}
        state["_gpt_runtime_overrides"] = overrides
        return state

    def __setstate__(self, newstate: dict) -> None:
        overrides = newstate.pop("_gpt_runtime_overrides", {})
        self.__dict__.update(newstate)
        self.model_cfg = load_model_config(self.model)
        self.architecture = self.model_cfg.architecture
        for key, value in overrides.items():
            setattr(self.model_cfg.gpt, key, value)

CacheConfig#

Constructor: deepinfer/config.py:125-143. Four fields are None at first. Two later start-up steps fill them in the same object, which the engine and the driver Worker share.

ParallelConfig#

Constructor: deepinfer/config.py:150-174. There is no tensor-parallel size: the "model parallel" size is the CP size.

Worked example. examples/offline_inference.py --ep 4 --cp 2 gives dp_size = max(4 // 2, 1) = 2 (examples/offline_inference.py:24-28). Then model_parallel_size = 2, engram_parallel_size = 2 × 2 = 4, world_size = max(4, 4) = 4. Engine 1 has data_parallel_rank = 1 % 2 = 1, and its ranks are 2 and 3.

SchedulerConfig#

Constructor: deepinfer/config.py:185-206. The C++ Scheduler reads the fields by name from the Python object, once, in its constructor.

Derived values#

runtime_max_model_len (deepinfer/config.py:11-16):

nextn              = DEEPINFER_USE_NEXTN_DECODE                 # default 5
scheduler_overhead = nextn + (nextn + 1 if ZOS on else 0)
draft_overhead     = nextn
runtime_max_model_len = max_model_len + scheduler_overhead + draft_overhead

max_num_batched_tokens (deepinfer/config.py:219-228):

max_num_decoding_tokens = max_num_seqs * (1 + nextn)
if max_num_batched_tokens is given:
    assert max_num_batched_tokens >= max_num_decoding_tokens
    use it
else:
    max(max_model_len, 4096, max_num_decoding_tokens)

Checks and their errors#

Some checks are assert statements. Python removes them when it runs with -O.

Where the docstrings and the code disagree#

• max_model_len. The docstring says: "If None, will be derived from the model" (deepinfer/config.py:39-40). The code uses 4096 and does not read the model (deepinfer/config.py:69). The checkpoint value max_position_embeddings is parsed, but ModelConfig does not use it (deepinfer/hugging_face/config.py:277).
• gpu_memory_utilization. The error text says "must be less than 1.0", but the test is > 1.0, so 1.0 passes (deepinfer/config.py:146-147). The LLM docstring says "between 0 and 1" (deepinfer/entrypoints/llm.py:44-48). No check refuses 0 or a negative value; then the memory profile finds no room, and start-up stops with "No available memory for the cache blocks" (deepinfer/engine/llm_engine.py:441-444).
• stop_preempted_request. Three defaults exist: SchedulerConfig False (deepinfer/config.py:191), EngineArgs True (deepinfer/engine/arg_utils.py:40), LLM False (deepinfer/entrypoints/llm.py:68). The EngineArgs value always reaches SchedulerConfig (deepinfer/engine/arg_utils.py:111).

Fields that nothing reads#

A search of deepinfer/, csrc/ and examples/ finds no reader for these fields. They are stored and then not used in this release.

• EngineArgs.offline_inference: stored in LLMEngine.offline_inference only (deepinfer/engine/llm_engine.py:195).
• ModelConfig.ignore_len (deepinfer/config.py:73) and ModelConfig.served_model_name (deepinfer/config.py:74).
• SchedulerConfig.driver_gpu_id: the C++ constructor does not read it (deepinfer/engine/llm_engine.py:253, csrc/engine/scheduler.hpp:94-108).
• CacheConfig.bytes_per_block appears only in a log line (deepinfer/engine/llm_engine.py:262-264).