Offline API and the step loop

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

The idea#

An LLM makes one token at a time for each request. DeepInfer serves many requests together, so it works in rounds. One round is an engine step. In each step, the engine selects a batch of requests and runs the model once for the whole batch. Then it adds new tokens to each request in the batch.

The offline API is the class LLM in deepinfer/entrypoints/llm.py. It has one main method, generate(), which does three things:

1. It adds every prompt to the engine as a request.
2. It calls step() while a request is not finished. It copies the outputs of each step.
3. When all requests are finished, it calls flush_inflight_reqs() once, sorts the results, and returns them.

One step has fixed phases. The C++ scheduler selects the batch. The Worker runs the model on the device and samples the next tokens. The engine then updates each sequence and makes one RequestOutput for each new token. With zero-overhead scheduling (ZOS), the engine updates the sequences of a decode step during the next step. So the CPU work overlaps with the device work.

An everyday comparison. Think of a ferry that crosses a river again and again. On each trip it takes all cars that are ready. A car must make many trips to reach its goal, because each trip moves it only a short distance (one or a few tokens). The comparison stops being true in two places. A real ferry unloads at the end of the same trip. With ZOS, DeepInfer reports the result of a decode trip during the next trip. Also, all ferries of a fleet (the data-parallel ranks) must make the same number of trips, also when one of them is empty.

The picture#

Read panel 1 as a loop: test, step, copy, test again. Panel 2 shows that the Worker calls back into the engine twice in one step. Panel 3 is the most important one. A prefill step returns its tokens at once, but a decode step returns the tokens of the step before it.

In the code#

Step 1: generate() adds all requests first#

        # Add requests to the engine.
        num_requests = len(prompts) if prompts is not None else len(prompt_token_ids)
        for i in range(num_requests):
            prompt = prompts[i] if prompts is not None else None
            token_ids = None if prompt_token_ids is None else prompt_token_ids[i]

            self._add_request(
                prompt,
                # 共享同一个 SamplingParams 时无需在此复制：_add_request 内部会 deepcopy
                sampling_params if isinstance(sampling_params, SamplingParams) else sampling_params[i],
                token_ids,
                None if multi_modal_data is None else multi_modal_data[i],
            )
        return self._run_engine(use_tqdm)

Each _add_request call tokenizes the prompt (if the caller gave text), gives the request the next counter value as its id, and calls LLMEngine.add_request (deepinfer/entrypoints/llm.py:151-166). The engine makes a C++ sequence and puts it into the waiting queue of the scheduler (deepinfer/engine/llm_engine.py:620-630). No model work happens yet. The page Request and sequence lifecycle follows one sequence from here to its end.

Step 2: the loop in _run_engine()#

    def _run_engine(self, use_tqdm: bool) -> list[list[RequestOutput] | str]:
        # Initialize tqdm.
        if use_tqdm:
            num_requests = self.llm_engine.get_num_unfinished_requests()
            pbar = tqdm(total=num_requests, desc="Processed prompts")

        # Run the engine.
        outputs: dict[str, list[RequestOutput]] = defaultdict(list)
        while True:
            unfinished = self.llm_engine.has_unfinished_requests()
            step_outputs = self.llm_engine.step() if unfinished else self.llm_engine.flush_inflight_reqs()
            for raw_output in step_outputs:
                # clone step_outputs since step_outputs will be released in the next step
                output = RequestOutput.clone(raw_output)
                outputs[output.request_id].append(output)
                if output.finished and use_tqdm:
                    pbar.update(1)
            if not unfinished:
                break
        request_ids = sorted(outputs, key=int)
        results: list[list[RequestOutput] | str] = [outputs[i] for i in request_ids]
        if self.decode_sequence:
            results = [self.llm_engine.tokenizer.decode([x.token_id for x in output]) for output in results]

        if use_tqdm:
            pbar.close()

        return results

1. Line 177: the loop asks the engine if any request is unfinished.
2. Line 178: if yes, it runs one step. If no, it calls flush_inflight_reqs() instead.
3. Lines 179-182: each step returns an array of handles. The loop copies each output with RequestOutput.clone before the next step. The engine gives the original objects back to their pool one step later.
4. Lines 185-186: the loop ends after the flush.
5. Lines 187-190: the results are sorted by request id as an integer, so they come back in prompt order. With decode_sequence=True (the default), each list of outputs becomes a string (deepinfer/entrypoints/llm.py:64).

Step 3: is any request unfinished?#

    def has_unfinished_requests(self) -> bool:
        """Returns True if there are unfinished requests."""
        if envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING:
            dp_group = PG.cpu_data_parallel_group()
            device = "cpu"
        else:
            dp_group = PG.data_parallel_group()
            device = "cuda"
        unfinished = self.scheduler.has_unfinished_seqs()
        unfinished = unfinished or self.so_builder_has_pending()
        if dp_group.size() > 1:
            unfinished = torch.full((1,), unfinished, dtype=torch.int64, device=device)
            dist.all_reduce(unfinished, op=dist.ReduceOp.MAX, group=dp_group)
            unfinished = unfinished.item() > 0
        return unfinished

The local answer is true if a scheduler queue (waiting, running or prompted) holds a sequence (csrc/engine/scheduler.hpp:165-167). It is also true if a structured output grammar is still in the build queue. With more than one data-parallel rank, the engine takes the maximum over all ranks. So all ranks keep calling step() until the last rank is done. A rank with no work then runs a placeholder step.

Step 4: one call of step()#

LLMEngine.step() is short. It schedules, does some checks, and hands the batch to the Worker:

        with cpu_timer.record("schedule"):
            seq_metadata_list: list[SequenceMetadata]
            scheduler_outputs: SchedulerOutputs
            with cpu_timer.record("schedule:cpp"):
                seq_metadata_list, scheduler_outputs = self.scheduler.schedule(
                    self.sequence_metadata_pool
                )  # Will also free the finished sequence groups.

        with cpu_timer.record("multi_modal_fetch"):
            if self.multi_modal_processor is not None:
                self.multi_modal_processor.fetch()
                abort_requests = self.multi_modal_processor.fetch_to_abort_requests()
                if abort_requests:
                    logger.error(f"多模态处理失败（通常是损坏的图片/视频），即将 abort {len(abort_requests)} 个请求: {abort_requests}。")
                    self.abort_request(abort_requests)

        if not scheduler_outputs.is_empty() and self.engine_type == EngineType.PROMPT:
            assert scheduler_outputs.prompt_run, "Prompt engines should not perform decoding"
            assert seq_metadata_list[0].seqlen_q > 0, "seqlen_q should be > 0"

        # model execution
        request_outputs = self._execute_model_async_step(seq_metadata_list, scheduler_outputs)

        if self.log_stats:
            self.stats_logger.log_nccl_memory()

        self.generate_step += 1

        return request_outputs

The phases, in order:

1. Schedule (lines 885-887). The C++ scheduler picks a prefill batch or a decode batch. It returns one SequenceMetadata per sequence and a SchedulerOutputs object. The page The scheduler explains the rules.
2. Clone or zero (lines 889-905). Only for forked sequences. The offline API never forks.
3. Checks (lines 907-928). An empty hook, and a check of the structured-output state at the last prefill chunk.
4. Multimodal fetch (lines 939-945). The engine collects the results of the image worker. In the engine, a request becomes schedulable only here. A failed request is aborted.
5. Execute (line 952). The engine calls the Worker. The Worker runs the device work and calls back into the engine (next step).
6. Return (line 959). The handles of the outputs that are ready now.

Step 5: the Worker part of the step#

The Worker method execute_model_step has four parts. Its docstring lists them:

        """The unified entry for model execution: every step (prompt/decode, ZOS on/off,
        placeholder) goes through this Forward-Post_process-Sample round.
          1. Forward current step (main model)
          2. Post-process previous step
          3. Sample current step
          4. Post-process current step: in place for prompt / non-ZOS steps;
             ZOS decode steps defer it to the next step to overlap with forward

        # 4. prompt 步或配置 non-ZOS 时当场 post process；ZOS decode 步留到下一步（与 forward 重叠）
        if is_prompt or not envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING:
            request_outputs = np.concatenate([request_outputs, self._post_process_step(curr_step_state)])
            self.prev_step_state = None
        else:
            self.prev_step_state = curr_step_state

        return request_outputs

The post-process calls back into the engine. LLMEngine.post_process runs three C++ functions (deepinfer/engine/llm_engine.py:717-761). process_MTPInfo stores the new draft tokens. scheduler.process_outputs appends the tokens to each sequence and checks the stop conditions. build_request_outputs makes the output objects. The second callback, pre_sampler_sync, runs while the device samples. It gives back the buffers of the previous step and frees finished sequences (deepinfer/engine/llm_engine.py:830-846).

Step 6: what step() returns#

step() returns a NumPy array of uint64 values. Each value is the address of a RequestOutput object in the RequestOutputPool. The docstring gives the rule:

    def step(self) -> np.ndarray[np.uint64]:
        """Run one inference step and return the generated outputs.

        Returns:
            np.ndarray[np.uint64]: entrypoint -> a list of RequestOutput objects.

        Note:
            The returned RequestOutput objects are managed by RequestOutputPool.
            RequestOutput will be automatically released in the next step() call.
            If you need to retain them beyond the current step, you must call pool.clone(list[RequestOutput]) to obtain copies,
            and release them via pool.release(list[RequestOutput]).
        """

One request can get several outputs in one step, one per accepted token. A step can also return no output at all. Panel 3 of the figure shows steps with no output. Its footnote mentions the case with accepted drafts.

Step 7: the end of the loop and the flush#

    def flush_inflight_reqs(self) -> np.ndarray[np.uint64]:
        """flush 并 post process ZOS 中所有在途请求。"""
        request_outputs = self._run_workers("flush_inflight_reqs")
        self.deferred_release.collect_request_outputs(request_outputs)
        self.process_output(request_outputs)
        return request_outputs

    def flush_inflight_reqs(self) -> np.ndarray[np.uint64]:
        """
        Flush inflight requests running on GPU, useful for ZOS. It may cause CPU-GPU synchronization.

        Returns:
            An array of request outputs for the flushed requests.
        """
        request_outputs = self._post_process_step(self.prev_step_state)
        self.prev_step_state = None
        return request_outputs

With ZOS, the Worker keeps the last decode step in prev_step_state. No later step post-processes it, so the flush does it. In the offline loop, the kept step at that time has no real sequence, so the flush returns no output (deep dive below). A custom loop must still call it: if the loop stops earlier, the flush returns the last tokens.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

class DeferredRelease:
    """延后一轮释放 step 产出的资源，保证调用方读完。

    回收（collect）要释放的资源；在下一次的 pre_sampler_sync 时释放（release），与 GPU 计算 overlap。
    """

    def _placeholder_seq_metadata(self, is_prompt: bool) -> SequenceMetadata:
        nextn_pad_token_id = get_nextn_pad_token_ids(self.vocab_size)
        # mtp_info_write_slot defaults to the padding/dummy slot 0, which the store kernel skips.
        return SequenceMetadata(
            request_id=PLACEHOLDER_REQUEST_ID,
            seq_id=0,
            is_prompt=is_prompt,
            seq_data=SequenceData([0]),
            compressed_seq_data=SequenceData([0]),
            multi_modal_data=None,
            sampling_params=SamplingParams(temperature=0.0),
            block_table=[0],
            seqlen_q=1,
            seqlen_k=1,
            last_output_token=0,
            last_accepted_token=[],
            last_next_n_token=[] if is_prompt else nextn_pad_token_id,
        )

What can go wrong#

• A custom loop without the flush. With ZOS, the outputs of the last kept decode step come only from flush_inflight_reqs() (deepinfer/worker/worker.py:274-283).
• A loop on one data-parallel rank only. The scheduler votes with all-reduce calls in every step. If one rank stops calling step(), the other ranks wait in the vote. Use has_unfinished_requests(), which combines all ranks.
• Lost token ids. With decode_sequence=True (the default), generate() returns only text (deepinfer/entrypoints/llm.py:189-190). The examples set decode_sequence=False to keep the objects (examples/_config.py:121).
• A slow start. The first steps of a run can be empty, because requests wait for the multimodal worker and for an attempt point.
• A request that is never added. add_request returns None for a token id outside the vocabulary and writes only an info log line (deepinfer/engine/llm_engine.py:582-589). generate() then has no output for that prompt, and the result list is shorter than the prompt list.

Why it matters. The step loop is the heartbeat of the engine. A driver of your own (a server or a benchmark) must keep three rules. Copy the outputs before the next step. Flush at the end. Keep all data-parallel ranks in the same number of steps.