deepinfer/entrypoints/llm.py: the offline LLM class

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

Purpose#

LLM is the class that a user script calls. It turns a list of prompts into a list of results, and hides the engine step loop.

Where it sits#

Panel 1 runs once per prompt and does no device work. Panel 2 is the loop. The flush runs only once, at the end.

Who calls it#

• The examples make the object in create_llm, with trust_remote_code=True, chunked_prefill=True, decode_sequence=False and statistics on (examples/_config.py:110-125). examples/offline_inference.py then calls generate (examples/offline_inference.py:95).
• The eval script uses decode_sequence=False and passes token ids (eval/run_eval.py:189).
• The benchmark uses the private method llm._add_request and then calls llm.llm_engine.step() itself (examples/benchmark.py:304-309, examples/benchmark.py:369).

What it calls#

• EngineArgs(...) and LLMEngine.from_engine_args (deepinfer/entrypoints/llm.py:70-84). See engine/llm_engine.py: the LLMEngine class.
• tokenizer.encode and tokenizer.decode (deepinfer/entrypoints/llm.py:156, deepinfer/entrypoints/llm.py:190).
• LLMEngine.add_request, has_unfinished_requests, step, flush_inflight_reqs, get_num_unfinished_requests and close.
• RequestOutput.clone from the C++ extension (deepinfer/entrypoints/llm.py:15). See csrc/engine/outputs.hpp: RequestOutput and its pool.

Key classes and functions#

Walk through the main methods#

__init__: defaults that differ from the engine#

    def __init__(
        self,
        model: str,
        tokenizer: str | None = None,
        trust_remote_code: bool = False,
        expert_parallel_size: int = 1,
        revision: str | None = None,
        tokenizer_revision: str | None = None,
        seed: int = 0,
        gpu_memory_utilization: float = 0.9,
        decode_sequence: bool = True,
        **kwargs: Any,
    ) -> None:
        kwargs.setdefault("disable_log_stats", True)
        kwargs.setdefault("stop_preempted_request", False)
        os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
        engine_args = EngineArgs(
            model=model,
            tokenizer=tokenizer,
            trust_remote_code=trust_remote_code,
            expert_parallel_size=expert_parallel_size,
            revision=revision,
            tokenizer_revision=tokenizer_revision,
            seed=seed,
            gpu_memory_utilization=gpu_memory_utilization,
            offline_inference=True,
            **kwargs,
        )
        self.decode_sequence = decode_sequence

        self.llm_engine = LLMEngine.from_engine_args(engine_args)
        self.request_counter = Counter()

1. Lines 56-64: nine named arguments. All other arguments go through **kwargs to EngineArgs, for example max_model_len, max_num_seqs or chunked_prefill. An unknown name raises TypeError, because EngineArgs is a dataclass.
2. Line 67: statistics logging is off, unless you pass disable_log_stats=False.
3. Line 68: stop_preempted_request=False. The EngineArgs default is True (deepinfer/engine/arg_utils.py:40). So with LLM, a preempted request is computed again, not stopped (see The KV cache).
4. Line 69: TOKENIZERS_PARALLELISM=false, unless the variable is already set.
5. Line 79: offline_inference=True. The engine stores it (deepinfer/engine/llm_engine.py:195); no other line in the release reads it.
6. Line 82: decode_sequence chooses the result type of generate() (text or outputs).

The docstring has a note in Chinese. Translated: "With CUDA graphs on, the error 'CUDA error: an illegal memory access was encountered' can mean that the CUDA graph ran out of memory. Lower gpu_memory_utilization and try again." (deepinfer/entrypoints/llm.py:50-51).

generate: the input rules#

        if prompts is None and prompt_token_ids is None:
            raise ValueError("Either prompts or prompt_token_ids must be provided.")
        if isinstance(prompts, str):
            # Convert a single prompt to a list.
            prompts = [prompts]
            multi_modal_data = [multi_modal_data]
        if prompts is not None and prompt_token_ids is not None and len(prompts) != len(prompt_token_ids):
            raise ValueError("The lengths of prompts and prompt_token_ids must be the same.")
        if sampling_params is None:
            # Use default sampling params.
            sampling_params = SamplingParams()

        # Add requests to the engine.
        num_requests = len(prompts) if prompts is not None else len(prompt_token_ids)
        for i in range(num_requests):
            prompt = prompts[i] if prompts is not None else None
            token_ids = None if prompt_token_ids is None else prompt_token_ids[i]

            self._add_request(
                prompt,
                # 共享同一个 SamplingParams 时无需在此复制：_add_request 内部会 deepcopy
                sampling_params if isinstance(sampling_params, SamplingParams) else sampling_params[i],
                token_ids,
                None if multi_modal_data is None else multi_modal_data[i],
            )
        return self._run_engine(use_tqdm)

1. Lines 117-118: you must give prompts, prompt_token_ids, or both.
2. Lines 119-122: one string becomes a list of one. multi_modal_data is wrapped too, so pass one dict for one string.
3. Lines 123-124: if you give both lists, they must have the same length. Then the token ids win; _add_request tokenizes only when the ids are None (deepinfer/entrypoints/llm.py:153-156).
4. Lines 125-127: no settings means one SamplingParams() for all requests: temperature 1.0, 16 tokens (deepinfer/sampling_params.py:49-53).
5. Line 138: one object for all, or a list with one object per prompt. The comment says, translated: "when the same SamplingParams is shared, no copy is needed here: _add_request does a deepcopy" (line 137).

_add_request: one request#

        sampling_params = copy.deepcopy(sampling_params)
        request_id = str(next(self.request_counter))
        if prompt_token_ids is None:
            if prompt is None:
                raise ValueError("prompt and prompt_token_ids cannot both be None")
            prompt_token_ids = self.llm_engine.tokenizer.encode(prompt)
        prompt_token_ids = np.asarray(prompt_token_ids, dtype=np.int32)
        if multi_modal_data is None and self.llm_engine.multi_modal_processor is not None:
            multi_modal_data = dict(image_bytes=[], image_sizes=[])
        return self.llm_engine.add_request(
            user_id="0",  # HACK 随便加一个 user_id
            request_id=request_id,
            sampling_params=sampling_params,
            prompt_token_ids=prompt_token_ids,
            multi_modal_data=multi_modal_data,
        )

1. Line 151: a deep copy, because the engine changes the object (it sets eos_id, see deepinfer/sampling_params.py: SamplingParams).
2. Line 152: the request id is the next value of a counter that starts at 0, as a string. The counter lives in the LLM object, so a second generate() call goes on from the last id.
3. Line 156: text goes through tokenizer.encode(prompt). With token ids, DeepInfer uses them as they are and adds no BOS token.
4. Line 157: the engine needs a 1-D np.int32 array; the C++ Sequence refuses other types (see deepinfer/_C/engine.pyi: the C++ engine API, as seen from Python).
5. Lines 158-159: if the engine has an image processor and you gave no image data, the request gets an empty image dict. The only registered model is a vision-language model, so the NORMAL engine always has that processor (deepinfer/model_executor/models/__init__.py:15-17, deepinfer/engine/llm_engine.py:209-221).
6. Line 160: the return value is a Sequence, or None for a dropped request. generate() ignores it.

_run_engine: the loop#

    def _run_engine(self, use_tqdm: bool) -> list[list[RequestOutput] | str]:
        # Initialize tqdm.
        if use_tqdm:
            num_requests = self.llm_engine.get_num_unfinished_requests()
            pbar = tqdm(total=num_requests, desc="Processed prompts")

        # Run the engine.
        outputs: dict[str, list[RequestOutput]] = defaultdict(list)
        while True:
            unfinished = self.llm_engine.has_unfinished_requests()
            step_outputs = self.llm_engine.step() if unfinished else self.llm_engine.flush_inflight_reqs()
            for raw_output in step_outputs:
                # clone step_outputs since step_outputs will be released in the next step
                output = RequestOutput.clone(raw_output)
                outputs[output.request_id].append(output)
                if output.finished and use_tqdm:
                    pbar.update(1)
            if not unfinished:
                break
        request_ids = sorted(outputs, key=int)
        results: list[list[RequestOutput] | str] = [outputs[i] for i in request_ids]
        if self.decode_sequence:
            results = [self.llm_engine.tokenizer.decode([x.token_id for x in output]) for output in results]

        if use_tqdm:
            pbar.close()

        return results

1. Line 171: the progress bar counts local unfinished requests at the start.
2. Line 177: has_unfinished_requests() asks all data-parallel ranks with an all-reduce MAX, when there is more than one rank (deepinfer/engine/llm_engine.py:665-679). So this rank goes on while any rank has work.
3. Line 178: work left: one engine step. No work left: one flush, which returns the outputs that the Worker still holds from the last ZOS decode step (deepinfer/worker/worker.py:274-283).
4. Lines 179-182: every handle is cloned at once. The engine releases the pool objects at the next pre_sampler_sync (deepinfer/engine/llm_engine.py:830-831).
5. Lines 185-186: the loop ends after the flush.
6. Line 187: the request ids are sorted as integers, so "10" comes after "9".
7. Lines 189-190: with decode_sequence=True, each request becomes one string: the decode of all its output token ids.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

Related concepts#

• Offline API and the step loop: the step loop from the user's view.
• engine/llm_engine.py: the LLMEngine class: the engine that LLM wraps.
• deepinfer/sampling_params.py: SamplingParams: the settings of each request.
• csrc/engine/outputs.hpp: RequestOutput and its pool: why each handle needs a clone.
• Zero-overhead scheduling (ZOS): why outputs come one call late and the flush exists.
• Tour: Life of a request: one request from generate() to text.