What an LLM does

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

The idea#

Text in, one token out#

A large language model (LLM) does not work on letters or words. It works on tokens: small pieces of text, each with an integer id. The page Tokens and tokenizers shows how text becomes ids.

The model is a mathematical function with a fixed list of inputs and outputs:

• Input: a list of token ids, for example the ids of "The cat sat on".
• Output: one probability for each token in the vocabulary. The vocabulary is the list of all tokens that the model knows. A modern vocabulary often has between 32,000 and 256,000 tokens.

Take "The cat sat on". The model can give "the" a probability of 0.41, "a" 0.22 and "his" 0.09. Every other token gets a very small probability. These numbers are made up, but real outputs look like this. The model does not give one answer. It gives a probability distribution over all possible next tokens.

Writing text is a loop#

To write an answer, an inference engine runs the model in a loop. This kind of generation is autoregressive:

1. Run the model on all tokens so far. Get the probabilities of the next token.
2. Select one token from these probabilities. This is sampling. The page Sampling: temperature, top-k, top-p explains the rules.
3. Add the selected token to the end of the input.
4. Stop if the token is the end-of-sequence token (EOS), or if the answer has the maximum number of tokens. Else go back to step 1.

So an answer of 200 tokens needs 200 runs of the model. Each run makes exactly one new token. (Speculative decoding can make more than one token per run. DeepInfer uses it, but the idea stays the same.)

Where the knowledge is#

The model has billions of learned numbers, the weights. Training set them: the model read a very large amount of text and learned to predict the next token. All the "knowledge" of the model is in these numbers.

Generation does not change the weights. Each run reads all of them from device memory and uses them for the math. This is why LLM inference needs large and fast accelerators. It is also why an engine such as DeepInfer uses each read of the weights for many requests at the same time.

An everyday comparison#

Think of the word suggestions on a phone keyboard. You type "See you", and the keyboard suggests "soon", "tomorrow" or "later". When you tap one word, the keyboard adds it and suggests the next word. An LLM does the same thing, one token at a time.

The comparison stops here. A keyboard looks at the last few words and shows three choices. An LLM looks at all the text so far, often thousands of tokens. It gives a probability to every token in its vocabulary. Then the sampler, not the user, makes the choice.

What an inference engine adds#

The loop above is simple. An engine such as DeepInfer makes it fast for many users at the same time:

• It keeps the intermediate results of earlier tokens, so each run computes only the new token. This memory is the KV cache (The KV cache).
• It splits the work into two phases: prefill reads the whole prompt once, and each decode step adds one token (Prefill and decode).
• It runs the tokens of many requests in one step, so one read of the weights serves all of them (Batching).
• A scheduler decides in each step which requests run.

The picture#

Read panel 1 from left to right: text, ids, probabilities, one token. Panel 2 shows the loop. Each row is one run of the model. The green token of each row becomes the last input token of the next row. The loop stops at EOS.

In DeepInfer#

This is a basics page, so this part is short. The tour Tour: Life of a request follows one request through all of these places.

Step 1: the user calls generate()#

The offline API is the class LLM. generate() adds one request per prompt. If you give no SamplingParams, it uses the default values (deepinfer/entrypoints/llm.py:125-127). Then it runs the loop:

        # Run the engine.
        outputs: dict[str, list[RequestOutput]] = defaultdict(list)
        while True:
            unfinished = self.llm_engine.has_unfinished_requests()
            step_outputs = self.llm_engine.step() if unfinished else self.llm_engine.flush_inflight_reqs()
            for raw_output in step_outputs:
                # clone step_outputs since step_outputs will be released in the next step
                output = RequestOutput.clone(raw_output)
                outputs[output.request_id].append(output)
                if output.finished and use_tqdm:
                    pbar.update(1)
            if not unfinished:
                break

Step 2: the engine stops a request#

After each step, C++ code checks each sequence for the stop rules of the loop:

    void _check_stop(Sequence &seq) {
        int token_id = seq.get_last_token_id();

        if (!seq.generation.stopping.ignore_eos && token_id == eos_token_id) {
            seq.schedule.status = SequenceStatus::FINISHED_STOPPED;
            return;
        }

        if (seq.input.tokens.get_len() > max_model_len) {
            seq.schedule.status = SequenceStatus::FINISHED_LENGTH_CAPPED;
            return;
        }

        if (seq.get_output_len() == seq.generation.stopping.max_tokens) {
            seq.schedule.status = SequenceStatus::FINISHED_LENGTH_CAPPED;
            return;
        }

The status becomes a finish reason in the output: "stop" for EOS and "length" for the two length limits (csrc/engine/sequence.hpp:45-50).

Step 3: ids become text again#

At the end, LLM collects the tokens of each request. With decode_sequence=True (the default), it turns the token ids back into text with the same tokenizer (deepinfer/entrypoints/llm.py:187-190, deepinfer/entrypoints/llm.py:64).

Deep dive#

The sections below give the full detail. Open them one at a time.

P(x1, x2, ..., xn) = P(x1) · P(x2 | x1) · P(x3 | x1, x2) · ... · P(xn | x1 ... xn-1)

P("The cat sat on") = 0.02 × 0.01 × 0.10 × 0.50 = 0.00001
log P               = ln 0.02 + ln 0.01 + ln 0.10 + ln 0.50
                    = −3.912 − 4.605 − 2.303 − 0.693 = −11.513

        if prompts is None and prompt_token_ids is None:
            raise ValueError("Either prompts or prompt_token_ids must be provided.")
        if isinstance(prompts, str):
            # Convert a single prompt to a list.
            prompts = [prompts]
            multi_modal_data = [multi_modal_data]
        if prompts is not None and prompt_token_ids is not None and len(prompts) != len(prompt_token_ids):
            raise ValueError("The lengths of prompts and prompt_token_ids must be the same.")
        if sampling_params is None:
            # Use default sampling params.
            sampling_params = SamplingParams()

        # Add requests to the engine.
        num_requests = len(prompts) if prompts is not None else len(prompt_token_ids)
        for i in range(num_requests):
            prompt = prompts[i] if prompts is not None else None
            token_ids = None if prompt_token_ids is None else prompt_token_ids[i]

            self._add_request(
                prompt,
                # 共享同一个 SamplingParams 时无需在此复制：_add_request 内部会 deepcopy
                sampling_params if isinstance(sampling_params, SamplingParams) else sampling_params[i],
                token_ids,
                None if multi_modal_data is None else multi_modal_data[i],
            )
        return self._run_engine(use_tqdm)

What can go wrong#

• Very short answers. The default max_tokens is 16 (deepinfer/sampling_params.py:53). Set a larger value for real answers.
• Answers that never end early. With ignore_eos=True, every request runs to max_tokens. Benchmarks use this on purpose; normal use does not.
• An empty prompt. add_request raises ValueError("prompt must not be empty") (deepinfer/engine/llm_engine.py:553-554).
• A bad token id. If you pass a token id below 0, or at or above the valid limit, add_request logs an info message and returns None. It does not raise (deepinfer/engine/llm_engine.py:580-589). The limit is vocab_size, plus 5 image ids if the request has multimodal data (deepinfer/multimodal_tokens.py:28).

Why it matters. Every later page builds on this loop. Prefill and decode are the first run and the later runs. The KV cache saves the work of earlier tokens. Batching and the scheduler decide which loops share a step. Sampling is step 2 of the loop.