C++ engine API (deepinfer._C)

Code version: DeepInfer @ 9bd9cf2 · 2026-10-21

Use this page to find a name fast. For the rules that apply to many names (memory that a call borrows, calls that release the GIL, loose types), read deepinfer/_C/engine.pyi: the C++ engine API, as seen from Python. This page does not repeat them.

The module and its three submodules#

• One module, three parts. csrc/pybind.cpp makes the pybind11 module and adds the submodules cuda_utils, engine and graph_timer (csrc/pybind.cpp:7-17). The stub of the package lists the same three (deepinfer/_C/__init__.pyi:1-5).
• One build per platform. setup.py compiles the same engine sources on both platforms. It adds cuda_utils.cpp and graph_timer_cuda.cpp on NVIDIA, and asc_utils.cpp and graph_timer_asc.cpp on Ascend (setup.py:22-34, setup.py:43-47, setup.py:61-65).
• Where the file is. The compiled .so is in deepinfer/_so/<accelerator>-<arch>/. The package adds that folder to its search path at import (deepinfer/__init__.py:28-32).
• The stubs. The .pyi files are type stubs that make stubgen generates with pybind11_stubgen (Makefile:27-28, scripts/gen_stub.sh:11-14). Python never imports them. The "Stub" column below gives the line in the stub.

Submodule engine (34 names)#

The stub lists the names in __all__ (deepinfer/_C/engine.pyi:8): 19 classes, 2 enums, 12 functions and 1 constant. The groups are the same as on the engine.pyi page, which also gives the binding and the C++ line of each name.

Scheduling and blocks#

Sequences and ZOS inputs#

Outputs#

Input metadata#

Sampler inputs#

Submodule cuda_utils (3 names)#

Device memory outside the PyTorch caching allocator. The NVIDIA binding is in csrc/cuda_utils.cpp:98-108. The Ascend build binds the same three names, with the same Python signatures, in csrc/asc_utils.cpp:108-113.

In this release, only a test uses CudaIPCWrapper (tests/distributed/test_expert_shuffle.py:17, tests/distributed/test_expert_shuffle.py:89). The package itself does not call the two IPC functions.

Submodule graph_timer (3 names)#

Low-level handles for the graph timer, which measures regions inside a replayed CUDA graph. The binding is csrc/graph_timer/pybind.cpp:5-29. The Python side is CUDAGraphTimer in deepinfer/timer.py:78-217. CudaGraphRunner wraps each captured graph in a GraphHandle and registers it (deepinfer/worker/cuda_graph_runner.py:130-136).

Python files that import deepinfer._C#

Examples also import from it, for example RequestOutput in examples/offline_inference.py:61 and Sequence in examples/benchmark.py:270. A comment in sampling_inputs.py says that torch must load first, because the extension needs the libtorch shared libraries (deepinfer/model_executor/sampling_inputs.py:16).

Alphabetical index#

alloc_tensor (cuda_utils) · BlockManager (engine) · build_request_outputs (engine) · clone_and_zero_states_and_blocks (engine) · DecodeSamplingInputs (engine) · DriverSamplingInputs (engine) · encode_compressed (engine) · EventHandle (graph_timer) · EventRecordNodeHandle (graph_timer) · export_ipc_tensor (cuda_utils) · FlopsEstimator (engine) · get_finished_seqs_info (engine) · GraphHandle (graph_timer) · import_ipc_tensor (cuda_utils) · LogitsTransformInputs (engine) · PLACEHOLDER_REQUEST_ID (engine) · PrefillSamplingInputs (engine) · prepare_decode_v4 (engine) · prepare_decode_zos_inputs (engine) · prepare_prompt_v4 (engine) · prepare_sampler_inputs (engine) · prepare_zos_inputs (engine) · PrepareArgs (engine) · process_MTPInfo (engine) · PromptLogprobInputs (engine) · RequestOutput (engine) · RequestOutputPool (engine) · SamplerInputs (engine) · SamplingKernelInputs (engine) · Scheduler (engine) · SchedulerOutputs (engine) · Sequence (engine) · SequenceData (engine) · SequenceMetadata (engine) · SequenceMetadataPool (engine) · SequenceStatus (engine) · set_engram_lookup_table (engine) · SpeculativeSamplingInputs (engine) · update_prompt_logprobs (engine) · YocoPrefillTruncationMode (engine)