Request and sequence lifecycle

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

The idea#

A user sends a request: a prompt and its sampling settings. Inside the engine, the request becomes a sequence. The sequence is a C++ object. It holds the token ids, the settings, the KV block table, the draft tokens, and a status.

The status tells where the sequence is in its life (csrc/engine/sequence.hpp:30-41):

• WAITING: in the waiting queue. The sequence waits for its prefill, or for the next chunk of it.
• RUNNING: the prompt is in the KV cache. Each decode step adds one or more tokens.
• FINISHED_*: seven final values. Each one has a finish reason, for example "stop" for the end-of-sequence token or "length" for the token limit.

Two more things can happen on the way. A sequence can wait at the start, until the image worker marks it schedulable flag. And when the KV cache is full, the scheduler can preempt it. The sequence then stops, or it goes back to the queue and computes its KV cache again (recompute).

An everyday comparison. Think of a passenger at an airport. Check-in makes a boarding pass (the sequence). The passenger waits at the gate until the gate opens (schedulable, attempt point). Boarding is the prefill; the flight is the decode. The trip ends with a reason: arrived, cancelled, or bumped. A bumped passenger can get a seat on a later flight at the front of the line. The comparison stops being true here. A recomputed sequence keeps every token that it already made, but it must read all of them again in a new prefill.

The picture#

Read panel 1 from left to right. Amber arrows belong to the waiting queue, the dashed pink arrow is a recompute, and solid pink arrows end the sequence. The table in the finished box gives the reason string that the user sees in RequestOutput.finish_reason.

In the code#

Step 1: the engine makes the Sequence#

LLMEngine.add_request checks the prompt and makes the C++ object (deepinfer/engine/llm_engine.py:580-604). The binding fills a SequenceInit from the flat Python arguments:

SequenceInit make_sequence_init(int64_t seq_id, py::object multi_modal_data, int block_size, std::string request_id, std::string user_id,
    py::object sampling_params, double arrival_time, std::optional<std::vector<int64_t>> golden_token_ids) {
    SequenceInit init;
    init.request.seq_id = seq_id;
    init.input.multi_modal_data = std::move(multi_modal_data);
    init.cache.block_size = block_size;
    init.request.request_id = std::move(request_id);
    init.request.user_id = std::move(user_id);
    init.generation.params = std::move(sampling_params);
    init.request.arrival_time = arrival_time;
    init.schedule.schedulable = init.input.multi_modal_data.is_none();
    init.mtp.golden_tokens = std::move(golden_token_ids);
    return init;
}

Line 35 is important. A sequence with multimodal data starts as not schedulable. The offline LLM gives every request a multimodal dict when the engine has an image processor, also for text-only prompts (deepinfer/entrypoints/llm.py:158-159). The constructor then copies the sampling values and the stop values out of the Python object once (csrc/engine/sequence.hpp:276-283).

Two error cases give a sequence a final status before it enters a queue. Image bytes set to None give FINISHED_VE_FAILED (deepinfer/engine/llm_engine.py:603-604). An aborted grammar build gives FINISHED_SO_FAILED (deepinfer/engine/llm_engine.py:621-622).

Step 2: the sequence enters the waiting queue#

    void add_sequence(std::shared_ptr<Sequence> seq) {
        seq->request.arrival_time = get_timestamp_ms();
        if (engine_type == EngineType::NORMAL) {
            seq->reserve_output_tokens();
            seq->cache.block_table.reserve(_block_table_capacity(*seq));
        }
        waiting.push_back(std::move(seq));
    }

add_sequence sets the arrival time in milliseconds (the Python value is replaced). For a NORMAL engine it also reserves memory for max_tokens more token ids. It also reserves room in the block table for all blocks that the sequence can need (csrc/engine/scheduler.hpp:190-192). Then the sequence goes to the end of waiting. With an image processor, the engine also sends the multimodal dict to the image worker (deepinfer/engine/llm_engine.py:624-625).

Step 3: the image worker makes it schedulable#

The image worker runs in other processes. In every engine step, fetch() collects its results:

    def fetch(self) -> None:
        while True:
            try:
                id, multi_modal_data = self.output_queue.get_nowait()
            except queue.Empty:
                break
            # A returned result (even a None failure) proves the worker is alive.
            self._last_output_ts = time.time()
            self._pending.discard(id)
            if multi_modal_data is None:
                self.to_abort_ids.append(id)
                continue
            if id not in self.multi_modal_dict:
                free_multi_modal_pool_handles(multi_modal_data)  # 已 abort，结果不会再被消费
                continue
            if envs.DEEPINFER_USE_MULTIMODAL_POOL:
                # "handle" 保留在 dict 里标记池块所有权
                for key, (handle, shape, dtype) in multi_modal_data["handle"].items():
                    multi_modal_data[key] = get_multi_modal_pool().get_tensor(handle, shape, dtype)

            self.multi_modal_dict[id].clear()
            self.multi_modal_dict[id].update(multi_modal_data)
            self.scheduler.set_schedulable(id)

        self._check_worker_hang()

    void set_schedulable(const std::string &request_id) {
        for (auto &seq : waiting) {
            if (seq->request.request_id == request_id)
                seq->schedule.schedulable = true;
        }
    }

A failed result (None) goes to a list; the engine then aborts that request (deepinfer/engine/llm_engine.py:942-945). A sequence that is not schedulable stops the prefill loop when it is at the front of the queue (csrc/engine/scheduler.hpp:369-371).

Step 4: the prefill makes it RUNNING#

At an attempt point, the scheduler takes the sequence from the front of waiting. The first chunk takes the KV blocks and one state slot for the whole prompt. Only the last chunk changes the status:

            if (seq->cache.block_table.empty()) {
                acquire.allocate_seq(*seq);
            }

            if (!is_chunked) {
                num_curr_seqs += 1;
                seq->schedule.status = SequenceStatus::RUNNING;
                allocate_mtp_info_slot(*seq);
            }

After the step, process_outputs moves the cache counter forward by the chunk length. Then it decides:

• The whole prompt is in the cache: the sequence goes into running, and the sampled token becomes its first output token (csrc/engine/scheduler.hpp:937-953, csrc/engine/scheduler.hpp:969-990).
• More prompt is left: the sequence goes back to the front of waiting, with no output (csrc/engine/scheduler.hpp:1011-1020).

Step 5: decode steps add tokens#

In each decode step, a running sequence sends its last token plus its draft tokens. After the step, the accepted drafts and the new sampled token are appended one by one. The stop check runs after each token:

    void _check_stop(Sequence &seq) {
        int token_id = seq.get_last_token_id();

        if (!seq.generation.stopping.ignore_eos && token_id == eos_token_id) {
            seq.schedule.status = SequenceStatus::FINISHED_STOPPED;
            return;
        }

        if (seq.input.tokens.get_len() > max_model_len) {
            seq.schedule.status = SequenceStatus::FINISHED_LENGTH_CAPPED;
            return;
        }

        if (seq.get_output_len() == seq.generation.stopping.max_tokens) {
            seq.schedule.status = SequenceStatus::FINISHED_LENGTH_CAPPED;
            return;
        }
    }

So there are three stop conditions: the EOS token (unless ignore_eos), a length above max_model_len, and max_tokens output tokens. The EOS id is the tokenizer EOS that the engine gives to the scheduler (deepinfer/engine/llm_engine.py:312). SamplingParams has no stop strings and no list of stop ids (deepinfer/sampling_params.py:47-62).

                    for (auto token_id : token_ids) {
                        seq.append_token_id(token_id);
                        seq.append_compressed_token_id(map_engram(token_id));
                        token_cnt++;
                        _check_stop(seq);
                        if (seq.is_finished()) {
                            break;
                        }
                    }

Step 6: preemption: stop or recompute#

If a decode step finds no free block, the scheduler preempts the running sequence with the most output tokens (The scheduler). _preempt has two policies:

        const bool so_blocks_recompute = seq->structured_output.status != 0;
        if (stop_preempted_request || so_blocks_recompute) {
            if (so_blocks_recompute && !stop_preempted_request) {
                LOG_WARN("Request {} is preempted and stops: structured output (so_status {}) can not be recomputed.",
                    seq->request.request_id, seq->structured_output.status);
            } else {
                LOG_WARN("Request {} is preempted and stops.", seq->request.request_id);
            }
            seq->schedule.status = SequenceStatus::FINISHED_PREEMPTED;
            free_seq_nolock(*seq);
            ignored_seqs.push_back(seq);
        } else {
            LOG_WARN("Request {} is preempted and will be recomputed.", seq->request.request_id);
            seq->reset_for_recompute(); // 复位 seq 自身的滚动状态（cache.num_tokens、inflight 计数、MTP 状态），清单见
                                        // Sequence::reset_for_recompute
            waiting.push_front(seq);
            seq->schedule.status = SequenceStatus::WAITING;
            free_seq_nolock(*seq);
        }

• Stop. Status FINISHED_PREEMPTED, reason "insufficient_system_resource". This is the EngineArgs default (deepinfer/engine/arg_utils.py:40). A sequence with a structured-output status other than 0 always stops, because the code cannot recompute the grammar state (csrc/engine/scheduler.hpp:770-771).
• Recompute. The sequence keeps all its tokens, prompt and output. It loses its blocks, gets status WAITING, and goes to the front of the queue. Its next prefill reads the prompt and the outputs so far. The offline LLM uses this policy (deepinfer/entrypoints/llm.py:68).

reset_for_recompute resets the cache counters, the in-flight counters and the draft state (csrc/engine/sequence.hpp:307-311).

Step 7: a finished sequence leaves the engine#

A sequence that finishes in a decode step stays in running until the next decode schedule. Then free_finished_seqs takes it out. It also gives back the draft-info slot, if process_outputs did not give it back already (csrc/engine/scheduler.hpp:992-994):

    std::vector<SequencePtr> free_finished_seqs() {
        auto predicate = [this](SequencePtr &seq) { return seq->is_finished(); };
        std::vector<SequencePtr> result;
        for (auto &seq : running) {
            if (predicate(seq)) {
                release_mtp_info_slot(*seq);
                result.push_back(seq);
            }
        }
        running.erase(std::remove_if(running.begin(), running.end(), predicate), running.end());
        return result;
    }

The finished list goes out with that step. In pre_sampler_sync, the engine calls finalize_seq for each one, which frees the KV blocks and the state slot (deepinfer/engine/llm_engine.py:844-846, deepinfer/engine/llm_engine.py:804-806, csrc/engine/scheduler.hpp:738-745).

Step 8: the user sees RequestOutput objects#

For every new token, build_request_outputs makes one RequestOutput:

struct RequestOutput {
    RequestOutput(const std::string &request_id, int prompt_len, int token_id, const std::string &finish_reason, bool finished,
        int max_num_batched_tokens, float token_logprobs, float nextn_logprobs, float sum_smaller_probs, std::vector<float> &&top_logprobs,
        std::vector<int> &&top_token_ids, int num_output_tokens, PromptLogprobs &&prompt_logprobs)
        : request_id(request_id), prompt_len(prompt_len), token_id(token_id), finish_reason(finish_reason), finished(finished),
          max_num_batched_tokens(max_num_batched_tokens), token_logprobs(token_logprobs), nextn_logprobs(nextn_logprobs),
          sum_smaller_probs(sum_smaller_probs), top_logprobs(std::move(top_logprobs)), top_token_ids(std::move(top_token_ids)),
          num_output_tokens(num_output_tokens), prompt_logprobs(std::move(prompt_logprobs)) {}

    std::string request_id;
    int prompt_len;
    int token_id;
    std::string finish_reason;
    bool finished;
    int max_num_batched_tokens;
    float token_logprobs;
    float nextn_logprobs;
    float sum_smaller_probs;
    std::vector<float> top_logprobs;
    std::vector<int> top_token_ids;
    int num_output_tokens;
    PromptLogprobs prompt_logprobs;
    int draft_tok = 0;
    int accept_tok = 0;
    c10::optional<std::vector<float>> confidence_score = c10::nullopt;
};

finish_reason comes from the status (csrc/engine/sequence.hpp:45-64). It is an empty string while the sequence runs. Only the last output of a finished request has finished=True (csrc/engine/outputs.hpp:247-251). An ignored sequence (too long, aborted in the queue, failed, or preempted with stop) gets one output with its last token id and its reason (csrc/engine/outputs.hpp:297-304).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

            // ZOS 异步：本步尚在途时就被 _preempt 抢占重算的 seq（只在 decode 步出现——prompt 步的
            // 在途 seq 不在 running 里，不会成为抢占目标）。_preempt 已复位其计数并放回 waiting 队头，
            // 本步输出作废：跳过全部记账，避免重复入队与 cache.num_tokens 的错误推进。
            if (!scheduler_outputs.prompt_run && seq.schedule.status == SequenceStatus::WAITING) {
                pushed_tokens_before_finish.push_back(0);
                if (has_accepted_tokens)
                    flat_offset += scheduler_outputs.seqlens_q[i];
                continue;
            }

What can go wrong#

• Short outputs. The default max_tokens is 16 (deepinfer/sampling_params.py:53). generate() with no sampling_params uses this default.
• No stop strings. A stop word in the prompt template does not end the request. Only EOS and the two length limits do.
• Preempted requests that stop. With EngineArgs defaults, a preempted request ends with "insufficient_system_resource". A request with a structured-output status other than 0 ends this way with any setting (csrc/engine/scheduler.hpp:770-780).
• Recompute cost. A recomputed sequence reads all its tokens again. Many preemptions of long sequences can cost more device time than they save.
• Prompt too long. The request ends at its first prefill attempt with "prompt_length_exceeded" (csrc/engine/scheduler.hpp:377-385).
• No final output after an abort. A running sequence that is aborted gets no output with "abort" (deep dive "Every status").

Why it matters. The status and the finish reason are the only things that tell a user why a request ended. To read the logs, the metrics and the outputs, you must know which code path sets each value.