Figure: the path of each field. C++ reads the numbers once, when it makes the sequence (panel 1). Each step copies them into flat arrays for the sampler (panel 2). Click to zoom.Open full size ↗
User scripts make one object per request, or one for all. Example: the eval script sets temperature, top-p, max_tokens and a fixed seq_seed per sample (eval/run_eval.py:225-234).
@staticmethod
def _make_seed(seed: int | None) -> int:
if seed is None:
return random.getrandbits(64)
if isinstance(seed, int):
return seed & 0xFFFF_FFFF_FFFF_FFFF
raise TypeError()
Line 88: no seed gives 64 random bits from Python's random module. So two runs give different tokens, unless you pass a seed or seed the random module.
Line 90: an int is masked to 64 bits. A negative seed is valid: -1 becomes 18,446,744,073,709,551,615. True is an int in Python, so it gives seed 1.
Line 91: any other type raises TypeError() with an empty message. A NumPy integer is not a Python int, so np.int64(7) raises too.
Because of the mask, the range check on seq_seed at lines 104-105 can never fail.
Optional "trigger_token_ids" (one list[int]): the token sequence that turns the grammar on. Before the full sequence appears, the output is free.
Optional "bound_token_id": a non-negative integer; 0 is valid. Missing or None means no bound. A list is not valid. It needs a non-empty trigger_token_ids.
When the bound is active, free tokens are no longer allowed. The output first follows the trigger sequence token by token, then enters the grammar.
The page Structured outputs explains the grammar matcher. examples/benchmark.py shows a regex example: {"schema_type": "regex", "schema": r"Hello, my name is [a-z]+"} (examples/benchmark.py:280).
if isinstance(sampling_params, dict):
sampling_params = SamplingParams(**{k: v for k, v in sampling_params.items() if v is not None})
sampling_params.eos_id = self.eos_id
if self.yoco_prefill_truncation_mode != "off" and sampling_params.prompt_logprobs:
# mask / compact 任一模式下显式关闭 prompt logprobs。
logger.warning(
"Request %s: prompt_logprobs is forced to False under YoCo prefill truncation "
"(DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE != 'off'): compaction cannot compute clipped rows; "
"mask returns degenerate logp for pre-rectangle tokens",
request_id,
)
sampling_params.prompt_logprobs = False
Line 527: eos_id always becomes the tokenizer's EOS id. The docstring calls the field a "HACK" with default 0 "only for warm up phase" (deepinfer/sampling_params.py:36-38). So a value that you set has no effect through add_request.
Lines 529-537: with DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE not off, prompt_logprobs becomes False, with a warning in the log.
The engine changes the object that it gets. LLM passes a deep copy, so your object stays as it was. A direct caller of LLMEngine.add_request sees its own object change.
The text form shows 9 of the 13 fields (deepinfer/sampling_params.py:107-119). It leaves out first_n_token_no_eos, eos_id and golden_token_ids. logprobs is not stored, so it cannot show.
and max_tokens and ignore_eos into StoppingCriteria (csrc/engine/sequence.hpp:181-190). The Python object stays in the sequence as generation.params, but C++ never reads these numbers from it again.
Each step, in the metadata. Each SequenceMetadata copies the SamplingScalars struct (csrc/engine/sequence.hpp:380).
Each step, in prepare_sampler_inputs. The function fills one entry per sequence in pinned CPU tensors: temperatures and top_ps as float32, top_ks as int32, seq_seeds as uint64 (csrc/engine/prepare_sample.cpp:82-88, csrc/engine/prepare_sample.cpp:157-161). max_top_logprobs is the largest top_logprobs of the batch (line 157).
Stop check. After the step, the scheduler stops a sequence on EOS (unless ignore_eos), at max_model_len, or at max_tokens output tokens (csrc/engine/scheduler.hpp:1035-1052).
Worked example. R1 has temperature=0.6, top_p=0.95, seq_seed=7. R2 has temperature=0 and no seed. In one decode step: temperatures = [0.6, 1e-5], top_ps = [0.95, 1.0], top_ks = [-1, -1], seq_seeds = [7, <64 random bits>]. Note that 0.6 as float32 is 0.60000002384.
You write SamplingParams(temperature=0, top_p=0.5, top_k=40). Which values does the sampler get?
medium
Show answer
Temperature 1e-5, top-p 1.0, top-k -1. The checks pass first (0 ≥ 0, 0.5 in range, 40 in range), then lines 76-79 replace the three values (deepinfer/sampling_params.py:75-79).
3
You set eos_id=42 and call LLM.generate. Which EOS id does the request use?
The samplerThe sampler turns the hidden states of the rows that need a token into float32 logits, applies temperature, top-p and top-k, and draws one token per request with a seeded hash. In decode steps it also verifies the draft tokens.
deepinfer/entrypoints/llm.py: the offline LLM classLLM is DeepInfer's offline Python API. It makes one LLMEngine, adds every prompt as a request, calls step() until no data-parallel rank has work, clones the outputs, and returns token lists or decoded text.
engine/llm_engine.py: the LLMEngine classLLMEngine starts the workers, sizes the KV cache, owns the C++ scheduler, and runs one engine step each time LLM calls step().
engine/prepare_sample.cpp: the sampler inputsprepare_sampler_inputs reads the scheduled sequences once and builds the arrays that the sampler needs: which model-output rows to sample, per-sequence temperature, top-p, top-k and seed, and the draft-token bookkeeping of speculative decoding.
csrc/engine/sequence.hpp: Sequence and SequenceMetadatasequence.hpp defines the C++ Sequence (one record per request, owned by the scheduler) and SequenceMetadata (a per-step view of one scheduled sequence that the worker reads). It also defines the status enum, the finish reasons and the image token offsets.
Structured outputsA structured-output request carries a grammar (regex, JSON schema or EBNF). DeepInfer compiles it with xgrammar on a pinned CPU thread pool, then masks every token that the grammar does not allow, also on each draft row.
Determinism and batch invarianceFloating-point sums depend on their order, and many kernels choose the order from the batch. DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) fixes this. Deterministic sampling also needs DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed.