sdkagent

sglang API reference

400 public APIs from sglang (sgl-project/sglang) — 199 classes, 125 functions, 76 methods. Signatures extracted by static analysis of the actual source.

Repository: sgl-project/sglang

KindCount
Classes199
Functions125
Methods76

API list

classpython.sglang._mps_stub.Event
Minimal stand-in for ``torch.cuda.Event``.
classpython.sglang._mps_stub.Stream
Minimal stand-in for ``torch.cuda.Stream``.
classpython.sglang._mps_stub.StreamContext
Minimal stand-in for ``torch.cuda.StreamContext``.
funcpython.sglang._mps_stub.current_device() -> int
Return the index of the current MPS device (always 0).
funcpython.sglang._mps_stub.current_stream(device:Any=None) -> Stream
Return the default (and only) MPS stream.
funcpython.sglang._mps_stub.device_count() -> int
Return the number of available MPS devices (always 1).
funcpython.sglang._mps_stub.get_device_properties(device:Any=0) -> _MPSDeviceProperties
Return the properties of the MPS device.
funcpython.sglang._mps_stub.install() -> None
Patch ``torch.mps`` with the stubs above.
funcpython.sglang._mps_stub.set_device(device:Any) -> None
Set the current device.
funcpython.sglang._mps_stub.stream(s:Any) -> Stream
Return a context manager that is a no-op on MPS.
funcpython.sglang._triton_stub.install() -> None
Register a mock ``triton`` package in *sys.modules*.
classpython.sglang.check_env.BaseEnv
Base class for environment check
methodpython.sglang.check_env.BaseEnv.get_info() -> dict
Get CUDA-related information if available.
methodpython.sglang.check_env.BaseEnv.get_package_versions() -> dict
Get versions of specified packages.
classpython.sglang.check_env.GPUEnv
Environment checker for Nvidia GPU
methodpython.sglang.check_env.GPUEnv.get_topology()
Get GPU topology information.
classpython.sglang.check_env.HIPEnv
Environment checker for ROCm/HIP
classpython.sglang.check_env.MPSEnv
Environment checker for Apple Silicon MPS
classpython.sglang.check_env.MUSAEnv
Environment checker for MThreads GPU
methodpython.sglang.check_env.MUSAEnv.get_topology()
Get GPU topology information.
classpython.sglang.check_env.NPUEnv
Environment checker for Ascend NPU
funcpython.sglang.cli.utils.get_is_diffusion_model(model_path:str) -> bool
Detect whether model_path points to a diffusion model.
classpython.sglang.global_config.GlobalConfig
Store some global constants.
funcpython.sglang.kernel_api_logging.debug_torch_op(op_func:Callable, op_name:str, *namespace:str='sglang') -> Callable
NOTE: For internal use.
funcpython.sglang.kernels.aot.python.sgl_kernel.elementwise.rmsnorm(input:torch.Tensor, weight:torch.Tensor, eps:float=1e-06, out:Optional[torch.Tensor]=None, enable_pdl:Optional[bool]=None) -> torch.Tensor
Root mean square normalization.
funcpython.sglang.kernels.aot.python.sgl_kernel.spatial.create_greenctx_stream_by_value(SM_a:int, SM_b:int, device_id:int=None) -> tuple[ExternalStream, ExternalStream]
Create two streams for greenctx.
funcpython.sglang.kernels.aot.python.sgl_kernel.spatial.get_sm_available(device_id:int=None) -> int
Get the SMs available on the device.
funcpython.sglang.kernels.jit.utils.arch.get_jit_cuda_arch() -> ArchInfo
Get the current CUDA architecture info.
classpython.sglang.kernels.ops.activation.ReLU2Op
``out = relu(input) ** 2`` (single-input, not gated).
funcpython.sglang.kernels.ops.activation.gelu_and_mul(input:torch.Tensor, out:Optional[torch.Tensor]=None) -> torch.Tensor
``out = gelu(input[..., :d]) * input[..., d:]``.
funcpython.sglang.kernels.ops.activation.gelu_quick(input:torch.Tensor, out:Optional[torch.Tensor]=None) -> torch.Tensor
``out = input * sigmoid(1.702 * input)``.
funcpython.sglang.kernels.ops.activation.relu2(input:torch.Tensor, out:Optional[torch.Tensor]=None) -> torch.Tensor
``out = relu(input) ** 2``.
funcpython.sglang.kernels.ops.attention.dsa.tilelang_kernel.fp8_index(q:torch.Tensor, q_s:torch.Tensor, k:torch.Tensor, k_s:torch.Tensor) -> torch.Tensor
Perform index score using FP8 precision.
funcpython.sglang.kernels.ops.attention.dsv4.elementwise.fused_q_indexer_rope_first_quant(q_input:torch.Tensor, weight:torch.Tensor, weight_scale:float, cos_sin_cache:torch.Tensor, positions:torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]
DeepSeek-V3.2 only.
funcpython.sglang.kernels.ops.attention.fla.kda.cdiv(a:int, b:int) -> int
Ceiling division.
funcpython.sglang.kernels.ops.attention.fla.kda.next_power_of_2(n:int) -> int
The next power of 2 (inclusive)
classpython.sglang.kernels.ops.attention.flash_attn.cute.cache_utils.JITCache
In-memory cache for compiled functions.
methodpython.sglang.kernels.ops.attention.flash_attn.cute.cache_utils.JITCache.clear() -> None
Clear in-memory cache of compiled functions
funcpython.sglang.kernels.ops.attention.flash_attn.cute.cache_utils.get_jit_cache(name:str | None=None) -> JITCache
JIT cache factory.
funcpython.sglang.kernels.ops.attention.flash_attn.cute.cute_dsl_ptxas.patch()
Install system ptxas hook.
funcpython.sglang.kernels.ops.attention.flash_attn.cute.fa_logging.set_fa_log_level(level:int | str) -> None
Set the FA log level programmatically.
classpython.sglang.kernels.ops.attention.flash_attn.cute.mask.Sm100MaskEnum
Enumeration of mask types for FMHA operations.
funcpython.sglang.kernels.ops.attention.flash_attn.cute.pipeline.make_pipeline_state(type:PipelineUserType, stages:int)
Creates a pipeline state.
funcpython.sglang.kernels.ops.attention.flash_attn.cute.softmax.floor_if_packed(q_idx, qhead_per_kvhead:cutlass.Constexpr[int]) -> cute.Tensor
Convert q_idx to packed format for Pack-GQA.
funcpython.sglang.kernels.ops.attention.position.compute_position_triton(extend_prefix_lens:torch.Tensor, extend_seq_lens:torch.Tensor, extend_seq_lens_sum)
Compute positions.
funcpython.sglang.kernels.ops.diffusion.residual_gate_add(residual:torch.Tensor, update:torch.Tensor, gate:torch.Tensor) -> torch.Tensor
Fused ``residual + gate * update``.
funcpython.sglang.kernels.ops.diffusion.triton.mps_fallback.triton_one_pass_rms_norm_native(x:torch.Tensor, w:torch.Tensor, eps:float=1e-06) -> torch.Tensor
MLX-accelerated triton_one_pass_rms_norm.
funcpython.sglang.kernels.ops.diffusion.triton.sana_wm_gdn.fused_qk_inv_rms(qkv:torch.Tensor, eps:float=1e-05) -> tuple[torch.Tensor, torch.Tensor]
Single-pass Triton fused Q+K inverse-RMS.
funcpython.sglang.kernels.ops.diffusion.usp_relayout.usp_merge_heads(x:torch.Tensor) -> torch.Tensor
Merge Ulysses output heads with an exact eager fallback.
funcpython.sglang.kernels.ops.elementwise.elementwise.fused_sigmoid_mul(attn_output:torch.Tensor, gate:torch.Tensor, inplace:bool=False) -> torch.Tensor
Fused sigmoid-mul for attention output gating.
classpython.sglang.kernels.ops.gemm.cutedsl_bf16_gemm.WorkTileInfo
Which output tile this CTA processes.
funcpython.sglang.kernels.ops.gemm.dsv3_fused_a_gemm(mat_a:torch.Tensor, mat_b:torch.Tensor, output:Optional[torch.Tensor]=None) -> torch.Tensor
DeepSeek-V3 fused QKV-A GEMM.
funcpython.sglang.kernels.ops.gemm.kv_b_lora_absorbed.step_a_q_fwd(q_nope:torch.Tensor, B_buf:torch.Tensor, batch_info:LoRABatchInfo, full_K_per_head:int) -> torch.Tensor
Step A of the q-side correction.
funcpython.sglang.kernels.ops.gemm.kv_b_lora_absorbed.step_a_v_fwd(attn_output:torch.Tensor, A_buf:torch.Tensor, batch_info:LoRABatchInfo) -> torch.Tensor
Step A of the v-side correction.
funcpython.sglang.kernels.ops.gemm.trtllm_lora_temp.kv_b_lora_absorbed.step_a_q_fwd(q_nope:torch.Tensor, B_buf:torch.Tensor, batch_info:LoRABatchInfo, full_K_per_head:int) -> torch.Tensor
Step A of the q-side correction.
funcpython.sglang.kernels.ops.gemm.trtllm_lora_temp.kv_b_lora_absorbed.step_a_v_fwd(attn_output:torch.Tensor, A_buf:torch.Tensor, batch_info:LoRABatchInfo) -> torch.Tensor
Step A of the v-side correction.
funcpython.sglang.kernels.ops.kimi_k3.all_reduce.register_comm(comm:Communicator, *pull_sem_mc_ptr:int=0) -> None
Register the CustomAllReduceV2 storage plane.
funcpython.sglang.kernels.ops.kimi_k3.attn_res_hip.supports_attn_res_hip(hidden_size:int, nvb:int) -> bool
Whether this shape fits the register budget.
funcpython.sglang.kernels.ops.kimi_k3.gemm_ar.fits(x:torch.Tensor) -> bool
Whether this o_proj input can take the fused GEMM+AR path.
funcpython.sglang.kernels.ops.layernorm.gemma4_fused_ops.gemma4_fused_routing(gating_output:torch.Tensor, per_expert_scale:torch.Tensor, topk:int) -> tuple[torch.Tensor, torch.Tensor]
One-launch Gemma4 router.
funcpython.sglang.kernels.ops.lplb.cuda_solver.solve_ipm(A:torch.Tensor, b:torch.Tensor, c:torch.Tensor, num_iters:int=DEFAULT_NUM_ITERS, result:torch.Tensor | None=None) -> torch.Tensor
Run the fused single-SM IPM kernel.
funcpython.sglang.kernels.ops.lplb.shmem_budget.assert_fits(nc:int, nv:int, gpu:str='h100') -> None
Raise if the fused kernel will not fit on the target GPU.
funcpython.sglang.kernels.ops.lplb.shmem_budget.max_nc_for_nv(nv:int, gpu:str='h100') -> int
Largest NC that fits for a given NV.
funcpython.sglang.kernels.ops.lplb.shmem_budget.report(nc:int, nv:int, gpu:str='h100') -> str
Human-readable summary — used by kernels on init for logging.
classpython.sglang.kernels.ops.mamba.triton_ops.ssu_dispatch.TritonSSUBackend
Triton-based selective-state-update backend.
funcpython.sglang.kernels.ops.moe.ep_moe_kernels.get_tma_aligned_size(x:int, element_size:int) -> int
Global memory address of TMA must be 16-byte aligned.
funcpython.sglang.kernels.ops.moe.moe_front.get_front_strategy(num_tokens:int, device) -> str
Return the measured front strategy for this exact workload.
funcpython.sglang.kernels.ops.sampling.murmur_hash.fmix32(h:tl.uint32) -> tl.uint32
final mix of 32-bit hash value for MurmurHash
funcpython.sglang.kernels.ops.sampling.murmur_hash.murmur3_mix(h:tl.uint32, k:tl.uint32) -> tl.uint32
Mixes a 32-bit key into the hash state.
funcpython.sglang.kernels.ops.sampling.murmur_hash.rotl32(x, r:tl.constexpr) -> tl.uint32
rotate left 32-bit integer x by r bits e.g.
funcpython.sglang.kernels.ops.sampling.top_k_renorm_probs(probs:torch.Tensor, top_k:Union[torch.Tensor, int]) -> torch.Tensor
Renormalize ``probs`` by top-k thresholding.
funcpython.sglang.kernels.ops.sampling.top_p_renorm_probs(probs:torch.Tensor, top_p:Union[torch.Tensor, float]) -> torch.Tensor
Renormalize ``probs`` by top-p thresholding.
funcpython.sglang.kernels.registry.register_kernel(spec:KernelSpec) -> KernelSpec
Register ``spec`` in the process-wide :data:`registry`.
funcpython.sglang.kernels.selector.clear_cache() -> None
Drop the resolved-callable cache (used by tests).
funcpython.sglang.kernels.selector.get_kernel(op:str, backend:Optional[KernelBackend]=None) -> Callable
Resolve ``op`` to a callable kernel and cache it.
classpython.sglang.kernels.spec.DeviceType
Accelerator device family a kernel can run on.
classpython.sglang.kernels.spec.FormatSignature
A light description of a kernel's data contract.
classpython.sglang.kernels.spec.KernelSpec
A single callable kernel implementation and its metadata.
methodpython.sglang.kernels.spec.KernelSpec.is_available(platform:PlatformInfo) -> bool
Whether this backend can run on ``platform`` (metadata-only check).
methodpython.sglang.kernels.spec.KernelSpec.load() -> Callable
Import and return the backing callable.
classpython.sglang.kernels.spec.PlatformInfo
A minimal snapshot of the runtime accelerator platform.
funcpython.sglang.kernels.spec.capabilities_satisfied(capabilities:Union[FrozenSet[CapabilityRequirement], Tuple[CapabilityRequirement, ...], CapabilityRequirement], platform:PlatformInfo) -> bool
OR over ``capabilities`` (empty = unrestricted).
classpython.sglang.lang.backend.crusoe.Crusoe
SGLang backend for Crusoe managed inference.
classpython.sglang.lang.backend.runtime_endpoint.Runtime
A wrapper for the HTTP server.
classpython.sglang.lang.interpreter.ProgramState
The state of an SGL program.
classpython.sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.flux.FluxExecutor
Executor for Flux models in ComfyUI.
classpython.sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.zimage.ZImageExecutor
Executor for ZImage models in ComfyUI.
methodpython.sglang.multimodal_gen.configs.models.base.ModelConfig.update_model_arch(source_model_dict:dict[str, Any]) -> None
Update arch_config with source_model_dict
classpython.sglang.multimodal_gen.configs.models.dits.cosmos3video.Cosmos3VideoConfig
DiT config wrapper for Cosmos3 Video model.
classpython.sglang.multimodal_gen.configs.models.dits.ltx_2.LTX2Config
Configuration for LTX-2 Video Transformer.
classpython.sglang.multimodal_gen.configs.models.encoders.qwen3.Qwen3TextArchConfig
Architecture config for Qwen3 text encoder.
classpython.sglang.multimodal_gen.configs.models.encoders.qwen3.Qwen3TextConfig
Top-level config for Qwen3 text encoder.
classpython.sglang.multimodal_gen.configs.models.encoders.qwen3vl.Qwen3VLConfig
Configuration for Qwen3-VL text encoder.
classpython.sglang.multimodal_gen.configs.models.vaes.hunyuan3d.Hunyuan3DVAEArchConfig
Architecture config for Hunyuan3D VAE.
classpython.sglang.multimodal_gen.configs.models.vaes.hunyuan3d.Hunyuan3DVAEConfig
VAE configuration for Hunyuan3D.
classpython.sglang.multimodal_gen.configs.models.vaes.stablediffusion3.StableDiffusion3VAEConfig
Configuration for StableDiffusion3 VAE.
classpython.sglang.multimodal_gen.configs.pipeline_configs.base.PipelineConfig
The base configuration class for a generation pipeline.
classpython.sglang.multimodal_gen.configs.pipeline_configs.base.STA_Mode
STA (Sliding Tile Attention) modes.
classpython.sglang.multimodal_gen.configs.pipeline_configs.base.SlidingTileAttnConfig
Configuration for sliding tile attention.
classpython.sglang.multimodal_gen.configs.pipeline_configs.base.SpatialImagePipelineConfig
Base config for spatial image pipelines (e.g.
funcpython.sglang.multimodal_gen.configs.pipeline_configs.base.parse_int_list(value:str) -> list[int]
Parse a comma-separated string of integers into a list.
classpython.sglang.multimodal_gen.configs.pipeline_configs.cosmos3.Cosmos3Config
Cosmos3 unified pipeline config.
classpython.sglang.multimodal_gen.configs.pipeline_configs.flux.FluxPipelineConfig
Configuration for the FLUX pipeline.
classpython.sglang.multimodal_gen.configs.pipeline_configs.glm_image.GlmImagePipelineConfig
Configuration for the GlmImage pipeline.
classpython.sglang.multimodal_gen.configs.pipeline_configs.helios.HeliosT2VConfig
Configuration for the Helios T2V pipeline.
classpython.sglang.multimodal_gen.configs.pipeline_configs.krea2.Krea2PipelineConfig
Krea-2 single-stream MMDiT, text-to-image.
classpython.sglang.multimodal_gen.configs.pipeline_configs.ltx_2.LTX23PipelineConfig
Configuration overrides for LTX-2.3.
classpython.sglang.multimodal_gen.configs.pipeline_configs.ltx_2.LTX2PipelineConfig
Configuration for LTX-Video pipeline.
classpython.sglang.multimodal_gen.configs.pipeline_configs.qwen_image.QwenImagePipelineConfig
Configuration for the QwenImage pipeline.
funcpython.sglang.multimodal_gen.configs.sample.glmimage.align_glm_image_dimension(value:int) -> int
Round a GLM-Image dimension up to a supported multiple.
classpython.sglang.multimodal_gen.configs.sample.joy_echo.JoyEchoSamplingParams
Sampling parameters for JoyEcho DMD inference.
classpython.sglang.multimodal_gen.configs.sample.ltx_2.LTX2SamplingParams
Sampling parameters for LTX-2.
classpython.sglang.multimodal_gen.configs.sample.sampling_params.SamplingParams
Sampling parameters for generation.
methodpython.sglang.multimodal_gen.configs.sample.sampling_params.SamplingParams.add_cli_args(parser:Any) -> Any
Add CLI arguments for SamplingParam fields
classpython.sglang.multimodal_gen.configs.sample.sana.SanaSamplingParams
Defaults for SANA 1.5 1024px variant.
classpython.sglang.multimodal_gen.configs.sample.wan.Wan2_2_Base_SamplingParams
Sampling parameters for Wan2.2 TI2V 5B model.
classpython.sglang.multimodal_gen.configs.sample.wan.Wan2_2_TI2V_5B_SamplingParam
Sampling parameters for Wan2.2 TI2V 5B model.
funcpython.sglang.multimodal_gen.registry.get_non_diffusers_pipeline_name(model_path:str) -> Optional[str]
Get the pipeline name for a known non-diffusers model.
funcpython.sglang.multimodal_gen.registry.get_pipeline_config_classes(pipeline_class_name:str) -> Tuple[Type[PipelineConfig], Type[Any]] | None
Get the configuration classes for a pipeline.
classpython.sglang.multimodal_gen.runtime.cache.cache_dit_integration.CacheDitConfig
Configuration for cache-dit integration.
classpython.sglang.multimodal_gen.runtime.cache.spectrum.SpectrumForecaster
Chebyshev + discrete Taylor blend forecaster.
funcpython.sglang.multimodal_gen.runtime.disaggregation.roles.get_module_role(module_name:str) -> 'RoleType | None'
Classify a module name to its primary role.
funcpython.sglang.multimodal_gen.runtime.disaggregation.transport.protocol.encode_transfer_msg(msg:Any) -> list[bytes]
Encode as [TRANSFER_MAGIC, json_payload_bytes].
funcpython.sglang.multimodal_gen.runtime.distributed.group_coordinator.get_local_torch_device() -> torch.device
Return the torch device for the current rank.
funcpython.sglang.multimodal_gen.runtime.distributed.parallel_state.destroy_model_parallel() -> None
Set the groups to none and destroy them.
funcpython.sglang.multimodal_gen.runtime.distributed.parallel_state.get_dp_rank() -> int
Return my rank for the data parallel group.
funcpython.sglang.multimodal_gen.runtime.distributed.parallel_state.get_world_rank() -> int
Return my rank for the world group.
funcpython.sglang.multimodal_gen.runtime.distributed.parallel_state.get_world_size() -> int
Return world size for the world group.
classpython.sglang.multimodal_gen.runtime.entrypoints.cli.cli_types.CLISubcommand
Base class for CLI subcommands
methodpython.sglang.multimodal_gen.runtime.entrypoints.cli.cli_types.CLISubcommand.validate(args:argparse.Namespace) -> None
Validate the arguments for this command
funcpython.sglang.multimodal_gen.runtime.entrypoints.http_server.model_info_endpoint(request:Request)
Get model information.
funcpython.sglang.multimodal_gen.runtime.entrypoints.http_server.server_info_endpoint(request:Request)
Get server information.
classpython.sglang.multimodal_gen.runtime.entrypoints.openai.common_api.ModelCard
Model cards.
funcpython.sglang.multimodal_gen.runtime.entrypoints.openai.common_api.available_models()
Show available models.
funcpython.sglang.multimodal_gen.runtime.entrypoints.openai.common_api.retrieve_model(model:str)
Retrieve a model instance.
classpython.sglang.multimodal_gen.runtime.entrypoints.openai.realtime.generate_session.GenerateSession
A realtime generation session
funcpython.sglang.multimodal_gen.runtime.entrypoints.openai.utils.build_sampling_params(request_id:str, **kwargs) -> SamplingParams
Build SamplingParams from request parameters.
funcpython.sglang.multimodal_gen.runtime.launch_server.parse_url_string(url_str:str) -> list[str]
Parse a semicolon-separated URL string into a list.
classpython.sglang.multimodal_gen.runtime.layers.activation.GeluAndMul
An activation function for GeGLU.
classpython.sglang.multimodal_gen.runtime.layers.activation.SiluAndMul
An activation function for SwiGLU.
methodpython.sglang.multimodal_gen.runtime.layers.activation.SiluAndMul.forward_native(x:torch.Tensor) -> torch.Tensor
PyTorch-native implementation equivalent to forward().
funcpython.sglang.multimodal_gen.runtime.layers.activation.get_act_fn(act_fn_name:str) -> nn.Module
Get an activation function by name.
classpython.sglang.multimodal_gen.runtime.layers.attention.backends.attention_backend.AttentionBackend
Abstract class for attention backends.
classpython.sglang.multimodal_gen.runtime.layers.attention.layer.LocalAttention
Attention layer.
classpython.sglang.multimodal_gen.runtime.layers.attention.turbo_layer.DistributedAttention
Initialization.
classpython.sglang.multimodal_gen.runtime.layers.custom_op.CustomOp
Base class for custom ops.
classpython.sglang.multimodal_gen.runtime.layers.layernorm.RMSNorm
Root mean square normalization.
classpython.sglang.multimodal_gen.runtime.layers.linear.ColumnParallelLinear
Linear layer with column parallelism.
classpython.sglang.multimodal_gen.runtime.layers.linear.LinearBase
Base linear layer.
classpython.sglang.multimodal_gen.runtime.layers.linear.LinearMethodBase
Base class for different (maybe quantized) linear methods.
methodpython.sglang.multimodal_gen.runtime.layers.linear.LinearMethodBase.apply(layer:torch.nn.Module, x:torch.Tensor, bias:torch.Tensor | None=None) -> torch.Tensor
Apply the weights in layer to the input tensor.
classpython.sglang.multimodal_gen.runtime.layers.linear.MergedColumnParallelLinear
Packed linear layers with column parallelism.
classpython.sglang.multimodal_gen.runtime.layers.linear.ReplicatedLinear
Replicated linear layer.
classpython.sglang.multimodal_gen.runtime.layers.linear.RowParallelLinear
Linear layer with row parallelism.
classpython.sglang.multimodal_gen.runtime.layers.linear.UnquantizedLinearMethod
Linear method without quantization.
classpython.sglang.multimodal_gen.runtime.layers.mlp.FeedForward
A feed-forward layer.
classpython.sglang.multimodal_gen.runtime.layers.mlp.MLP
MLP for DiT blocks, NO gated linear units
funcpython.sglang.multimodal_gen.runtime.layers.parallel_conv.gather_height_sizes(x:torch.Tensor) -> list[int]
gather heights of sharded feature_maps from peers
classpython.sglang.multimodal_gen.runtime.layers.quantization.configs.base_config.QuantizationConfig
Base class for quantization configs.
classpython.sglang.multimodal_gen.runtime.layers.quantization.configs.base_config.QuantizeMethodBase
Base class for different quantized methods.
methodpython.sglang.multimodal_gen.runtime.layers.quantization.configs.base_config.QuantizeMethodBase.process_weights_after_loading(layer:nn.Module) -> None
Process the weight after loading.
classpython.sglang.multimodal_gen.runtime.layers.quantization.fp8.Fp8Config
Config class for FP8.
classpython.sglang.multimodal_gen.runtime.layers.quantization.fp8.Fp8LinearMethod
Linear method for FP8.
classpython.sglang.multimodal_gen.runtime.layers.quantization.modelopt_quant.ModelOptFp4Config
Config class for NVFP4.
classpython.sglang.multimodal_gen.runtime.layers.rotary_embedding.base.RotaryEmbedding
Original rotary positional embedding.
classpython.sglang.multimodal_gen.runtime.layers.rotary_embedding.mrope.NDRotaryEmbedding
N-dimensional rotary positional embedding.
classpython.sglang.multimodal_gen.runtime.layers.rotary_embedding.mrope.OneDRotaryEmbedding
1D rotary positional embedding with caching.
classpython.sglang.multimodal_gen.runtime.layers.visual_embedding.ModulateProjection
Modulation layer for DiT blocks.
classpython.sglang.multimodal_gen.runtime.layers.vocab_parallel_embedding.UnquantizedEmbeddingMethod
Unquantized method for embeddings.
funcpython.sglang.multimodal_gen.runtime.layers.vocab_parallel_embedding.pad_vocab_size(vocab_size:int, pad_to:int=DEFAULT_VOCAB_PADDING_SIZE) -> int
Pad the vocab size to the given value.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.component_loader.AutoProcessorLoader
Loader for auto processor.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.component_loader.ImageProcessorLoader
Loader for image processor.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.component_loader.TokenizerLoader
Loader for tokenizers.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.scheduler_loader.SchedulerLoader
Loader for scheduler.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.text_encoder_loader.TextEncoderLoader
Loader for text encoders.
classpython.sglang.multimodal_gen.runtime.loader.component_loaders.text_encoder_loader.TextEncoderLoader.Source
A source for weights.
funcpython.sglang.multimodal_gen.runtime.loader.weight_utils.default_weight_loader(param:torch.Tensor, loaded_weight:torch.Tensor) -> None
Default weight loader.
funcpython.sglang.multimodal_gen.runtime.loader.weight_utils.maybe_remap_kv_scale_name(name:str, params_dict:dict) -> str | None
Remap the name of FP8 k/v_scale parameters.
funcpython.sglang.multimodal_gen.runtime.managers.forward_context.get_forward_context() -> 'ForwardContext'
Get the current forward context.
classpython.sglang.multimodal_gen.runtime.managers.gpu_worker.GPUWorker
A worker that executes the model on a single GPU.
methodpython.sglang.multimodal_gen.runtime.managers.gpu_worker.GPUWorker.execute_forward(batch:List[Req], return_req:bool=False) -> OutputBatch | Req
Execute a forward pass.
methodpython.sglang.multimodal_gen.runtime.managers.gpu_worker.GPUWorker.merge_lora_weights(target:str='all', strength:float=1.0) -> OutputBatch
Merge LoRA weights.
methodpython.sglang.multimodal_gen.runtime.managers.gpu_worker.GPUWorker.release_realtime_session(session_id:str) -> OutputBatch
release the session of a realtime connection
methodpython.sglang.multimodal_gen.runtime.managers.gpu_worker.GPUWorker.unmerge_lora_weights(target:str='all') -> OutputBatch
Unmerge LoRA weights.
classpython.sglang.multimodal_gen.runtime.managers.scheduler.Scheduler
Runs the main event loop for the rank 0 worker.
classpython.sglang.multimodal_gen.runtime.models.bridges.mova_dual_tower.PerFrameAttentionPooling
Per-frame multi-head attention pooling.
methodpython.sglang.multimodal_gen.runtime.models.bridges.mova_dual_tower.PerFrameAttentionPooling.forward(x:torch.Tensor, grid_size:Tuple[int, int, int]) -> torch.Tensor
Forward pass.
funcpython.sglang.multimodal_gen.runtime.models.dits.common.modulate(x:torch.Tensor, shift:torch.Tensor | None=None, scale:torch.Tensor | None=None) -> torch.Tensor
Modulate by shift and scale.
classpython.sglang.multimodal_gen.runtime.models.dits.cosmos3video.Cosmos3GatedMLP
Gated MLP (SwiGLU-style) for Cosmos3.
classpython.sglang.multimodal_gen.runtime.models.dits.cosmos3video.Cosmos3OmniTransformer
Cosmos3 Omni transformer.
classpython.sglang.multimodal_gen.runtime.models.dits.flux.FluxTransformer2DModel
The Transformer model introduced in Flux.
classpython.sglang.multimodal_gen.runtime.models.dits.flux_2.Flux2Transformer2DModel
The Transformer model introduced in Flux 2.
classpython.sglang.multimodal_gen.runtime.models.dits.glm_image.GlmImageKVCache
Container for all layers' KV caches.
classpython.sglang.multimodal_gen.runtime.models.dits.glm_image.GlmImageLayerKVCache
KV cache for GlmImage model.
classpython.sglang.multimodal_gen.runtime.models.dits.joy_image.ModulateWan
Modulation layer for WanX.
funcpython.sglang.multimodal_gen.runtime.models.dits.joy_image.fused_add_gate(residual:torch.Tensor, x:torch.Tensor, gate:torch.Tensor) -> torch.Tensor
Fused residual addition with gate.
classpython.sglang.multimodal_gen.runtime.models.dits.mova_audio_dit.Conv1dLocalIsland
Inherits from Conv1d and overrides forward.
classpython.sglang.multimodal_gen.runtime.models.dits.mova_video_dit.CrossAttention
Cross-Attention module for MOVA DiT.
classpython.sglang.multimodal_gen.runtime.models.dits.qwen_image.QwenImageTransformer2DModel
The Transformer model introduced in Qwen.
classpython.sglang.multimodal_gen.runtime.models.dits.sana.GLUMBConv
Gated Linear Unit with Multi-Branch Convolution.
classpython.sglang.multimodal_gen.runtime.models.dits.sana_wm.SanaWMBlock
One transformer block of SANA-WM.
classpython.sglang.multimodal_gen.runtime.models.dits.sana_wm.SanaWMTransformer3DModel
SANA-WM 2.6B TI2V world model.
classpython.sglang.multimodal_gen.runtime.models.dits.sana_wm_components.TimestepEmbedder
Upstream ``TimestepEmbedder``.
classpython.sglang.multimodal_gen.runtime.models.dits.sana_wm_refiner_transformer.SanaWMRefinerBlock
Video-only LTX-2 transformer block.
classpython.sglang.multimodal_gen.runtime.models.encoders.gemma2.Gemma2Model
Gemma2 text encoder model for SANA pipeline.
classpython.sglang.multimodal_gen.runtime.models.encoders.gemma_3.Gemma3MultiModalProjector
Projector for Gemma3 multimodal.
classpython.sglang.multimodal_gen.runtime.models.encoders.qwen3.Qwen3DecoderLayer
Qwen3 transformer decoder layer.
classpython.sglang.multimodal_gen.runtime.models.encoders.t5.AttentionType
Attention type.
classpython.sglang.multimodal_gen.runtime.models.parameter.BasevLLMParameter
Base parameter for vLLM linear layers.
classpython.sglang.multimodal_gen.runtime.models.parameter.ModelWeightParameter
Parameter class for linear layer weights.
classpython.sglang.multimodal_gen.runtime.models.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler
Euler scheduler.
classpython.sglang.multimodal_gen.runtime.models.vaes.autoencoder_kl_qwenimage.QwenImageDecoder3d
A 3D decoder module.
classpython.sglang.multimodal_gen.runtime.models.vaes.autoencoder_kl_qwenimage.QwenImageEncoder3d
A 3D encoder module.
classpython.sglang.multimodal_gen.runtime.models.vaes.autoencoder_kl_qwenimage.QwenImageRMS_norm
A custom RMS normalization layer.
classpython.sglang.multimodal_gen.runtime.models.vaes.autoencoder_kl_qwenimage.QwenImageResidualBlock
A custom residual block module.
classpython.sglang.multimodal_gen.runtime.models.vaes.hunyuan3d_vae.Latent2MeshOutput
Container for mesh output from VAE decoder.
classpython.sglang.multimodal_gen.runtime.models.vaes.hunyuan3d_vae.MCSurfaceExtractor
Marching Cubes surface extractor.
classpython.sglang.multimodal_gen.runtime.models.vaes.hunyuan3d_vae.SurfaceExtractor
Base class for surface extraction algorithms.
classpython.sglang.multimodal_gen.runtime.models.vaes.hunyuan3d_vae.VectsetVAE
Base VAE class for vector set encoding.
methodpython.sglang.multimodal_gen.runtime.models.vaes.hunyuan3d_vae.VectsetVAE.latents2mesh(latents:torch.FloatTensor, **kwargs)
Convert latents to mesh.
classpython.sglang.multimodal_gen.runtime.models.vaes.ltx_2_audio.LTX2AudioAudioPatchifier
Patchifier for spectrogram/audio latents.
classpython.sglang.multimodal_gen.runtime.models.vaes.ltx_2_vae.AutoencoderKLCausalLTX2Video
Streaming causal LTX-2 VAE.
classpython.sglang.multimodal_gen.runtime.models.vaes.ltx_2_vae.LTX2VideoDownBlock3D
Down block used in the LTXVideo model.
classpython.sglang.multimodal_gen.runtime.models.vaes.ltx_2_vae.LTX2VideoMidBlock3d
A middle block used in the LTXVideo model.
classpython.sglang.multimodal_gen.runtime.models.vaes.ltx_2_vae.LTX2VideoUpBlock3d
Up block used in the LTXVideo model.
funcpython.sglang.multimodal_gen.runtime.models.vaes.minimax_h3_audio_vae.alias_free.sinc(x:torch.Tensor)
Implementation of sinc, i.e.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanAttentionBlock
Causal self-attention with a single head.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanDecoder3d
A 3D decoder module.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanEncoder3d
A 3D encoder module.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanMidBlock
Middle block for WanVAE encoder and decoder.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanRMS_norm
A custom RMS normalization layer.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanResample
A custom resampling module for 2D and 3D data.
classpython.sglang.multimodal_gen.runtime.models.vaes.wanvae.WanResidualBlock
A custom residual block module.
classpython.sglang.multimodal_gen.runtime.pipelines.hunyuan3d_pipeline.Hunyuan3D2Pipeline
Hunyuan3D 2.0 image-to-mesh pipeline.
classpython.sglang.multimodal_gen.runtime.pipelines.ltx_2_pipeline.LTX2TwoStageResidencyController
LTX-2.3 two-stage residency controller.
classpython.sglang.multimodal_gen.runtime.pipelines.mova_pipeline.MOVAPipeline
MOVA pipeline with SGLang stage orchestration.
methodpython.sglang.multimodal_gen.runtime.pipelines.mova_pipeline.MOVAPipeline.initialize_pipeline(server_args:ServerArgs) -> None
Initialize the pipeline.
classpython.sglang.multimodal_gen.runtime.pipelines.sana_wm_pipeline.SanaWMPipeline
SANA-WM TI2V pipeline (single-stage).
classpython.sglang.multimodal_gen.runtime.pipelines.sana_wm_realtime_pipeline.SanaWMRealtimePipeline
SANA-WM realtime interactive pipeline.
classpython.sglang.multimodal_gen.runtime.pipelines.wan_pipeline.WanPipeline
Wan video diffusion pipeline with LoRA support.
funcpython.sglang.multimodal_gen.runtime.pipelines_core.build_pipeline(server_args:ServerArgs) -> PipelineWithLoRA
Only works with valid hf diffusers configs.
classpython.sglang.multimodal_gen.runtime.pipelines_core.schedule_batch.OutputBatch
Final output (after pipeline completion)
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.base.PipelineStage
Abstract base class for all pipeline stages.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.base.PipelineStage.device() -> torch.device
Get the device for this stage.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.base.PipelineStage.forward(batch:Req, server_args:ServerArgs) -> Req
Forward pass of the stage's processing.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.base.PipelineStage.verify_input(batch:Req, server_args:ServerArgs) -> VerificationResult
Verify the input for the stage.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.base.PipelineStage.verify_output(batch:Req, server_args:ServerArgs) -> VerificationResult
Verify the output for the stage.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.causal_denoising.CausalDMDDenoisingStage
Denoising stage for causal diffusion.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.causal_denoising.CausalDMDDenoisingStage.verify_input(batch:Req, server_args:ServerArgs) -> VerificationResult
Verify denoising stage inputs.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.denoising_dmd.DmdDenoisingStage
Denoising stage for DMD.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.denoising_dmd.DmdDenoisingStage.forward(batch:Req, server_args:ServerArgs) -> Req
Run the denoising loop.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.cosmos3.Cosmos3DecodingStage
VAE decoding stage for Cosmos3.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.cosmos3.Cosmos3TokenizationStage
Tokenization stage for Cosmos3.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.hunyuan3d.paint.DDIMSolver
DDIM solver for fast sampling.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.task_profiles.MiniMaxH3TaskProfile
One row of the task table.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.mova.MOVADenoisingStage
Run MOVA dual-tower denoising loop.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.mova.MOVADenoisingStage.verify_input(batch:Req, server_args:ServerArgs) -> VerificationResult
Verify denoising stage inputs.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.mova.MOVADenoisingStage.verify_output(batch:Req, server_args:ServerArgs) -> VerificationResult
Verify denoising stage outputs.
funcpython.sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.sana_wm.parity_probe.dump_obj(dirpath:Path | str | None, name:str, obj) -> None
Save a picklable object (e.g.
funcpython.sglang.multimodal_gen.runtime.pipelines_core.stages.progressive_resolution.spectral_ops.dct_1d(x:torch.Tensor, norm:str='ortho') -> torch.Tensor
1-D DCT-II via torch.fft.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators
Common validators for pipeline stages.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.bool_value(value:Any) -> bool
Check if value is a boolean.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.divisible_by(value:Any, divisor:int) -> bool
Check if value is divisible by divisor.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.is_list(value:Any) -> bool
Check if value is a list (can be empty).
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.is_tuple(value:Any) -> bool
Check if value is a tuple.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.list_length(value:Any, length:int) -> bool
Check if list has specific length.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.list_not_empty(value:Any) -> bool
Check if value is a non-empty list.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.non_negative_float(value:Any) -> bool
Check if value is a non-negative float.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.non_negative_int(value:Any) -> bool
Check if value is a non-negative float.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.none_or_list(value:Any) -> bool
Check if value is None or a list.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.none_or_positive_int(value:Any) -> bool
Check if value is None or a positive integer.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.not_none(value:Any) -> bool
Check if value is not None.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.positive_float(value:Any) -> bool
Check if value is a positive float.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.positive_int(value:Any) -> bool
Check if value is a positive integer.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.string_not_empty(value:Any) -> bool
Check if value is a non-empty string.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.StageValidators.string_not_none(value:Any) -> bool
Check if value is a non-empty string.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.ValidationFailure
Details about a specific validation failure.
classpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.VerificationResult
Wrapper class for stage verification results.
methodpython.sglang.multimodal_gen.runtime.pipelines_core.stages.validators.VerificationResult.is_valid() -> bool
Check if all validations passed.
funcpython.sglang.multimodal_gen.runtime.platforms.cpu_platform_plugin() -> str | None
Detect if CPU platform should be used.
methodpython.sglang.multimodal_gen.runtime.platforms.interface.Platform.get_current_memory_usage(device:torch.types.Device | None=None) -> float
Return the memory usage in bytes.
methodpython.sglang.multimodal_gen.runtime.platforms.interface.Platform.get_device_name(device_id:int=0) -> str
Get the name of a device.
methodpython.sglang.multimodal_gen.runtime.platforms.interface.Platform.get_device_total_memory(device_id:int=0) -> int
Get the total memory of a device in bytes.
methodpython.sglang.multimodal_gen.runtime.platforms.interface.Platform.get_device_uuid(device_id:int=0) -> str
Get the uuid of a device, e.g.
methodpython.sglang.multimodal_gen.runtime.platforms.interface.Platform.seed_everything(seed:int | None=None) -> None
Set the seed of each random module.
funcpython.sglang.multimodal_gen.runtime.platforms.xpu_platform_plugin() -> str | None
Detect if Intel XPU platform is available.
classpython.sglang.multimodal_gen.runtime.postprocess.realesrgan_upscaler.ImageUpscaler
Lazy-loaded Real-ESRGAN upscaler.
methodpython.sglang.multimodal_gen.runtime.postprocess.realesrgan_upscaler.ImageUpscaler.upscale(frames:list[np.ndarray]) -> list[np.ndarray]
Upscale a list of HWC uint8 frames.
classpython.sglang.multimodal_gen.runtime.postprocess.realesrgan_upscaler.RRDB
Residual in Residual Dense Block.
methodpython.sglang.multimodal_gen.runtime.postprocess.rife_interpolator.Model.load_model(path:str, strip_module_prefix:bool=True) -> None
Load weights from {path}/flownet.pkl.
funcpython.sglang.multimodal_gen.runtime.postprocess.rife_interpolator.warp(tenInput:torch.Tensor, tenFlow:torch.Tensor) -> torch.Tensor
Warp tenInput by tenFlow using grid_sample.
classpython.sglang.multimodal_gen.runtime.realtime.session.BaseRealtimeState
per-session state owned by pipeline stages
classpython.sglang.multimodal_gen.runtime.server_args.server_args.Backend
Enumeration for different model backends.
methodpython.sglang.multimodal_gen.runtime.server_args.server_args.Backend.from_string(value:str) -> 'Backend'
Convert string to Backend enum.
funcpython.sglang.multimodal_gen.runtime.server_warmup.get_first_generation_req(req_or_group:Any) -> Req | None
Extract the first req
funcpython.sglang.multimodal_gen.runtime.server_warmup.is_realtime_serving(server_args:ServerArgs) -> bool
Synthetic warmup has no realtime session state.
funcpython.sglang.multimodal_gen.runtime.server_warmup.is_warmup_req(req_or_group:Any) -> bool
either server-based or req-based
funcpython.sglang.multimodal_gen.runtime.utils.common.add_prefix(name:str, prefix:str) -> str
Add a weight path prefix to a module name.
funcpython.sglang.multimodal_gen.runtime.utils.hf_diffusers_utils.check_gguf_file(model:str | os.PathLike) -> bool
Check if the file is a GGUF model.
methodpython.sglang.multimodal_gen.runtime.utils.logging_utils.ColoredFormatter.format(record:logging.LogRecord) -> str
Adds color to the log
classpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.ImageProcessorV2
Image processor for Hunyuan3D single-view input.
classpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.MVImageProcessorV2
Multi-view image processor for Hunyuan3D.
funcpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.export_to_trimesh(mesh_output:Any) -> Any
Convert mesh output to trimesh format.
funcpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.get_perspective_projection_matrix(fovy:float, aspect_wh:float, near:float, far:float) -> np.ndarray
Compute perspective projection matrix.
funcpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.mesh_uv_wrap(mesh:Any) -> Any
Apply UV unwrapping to mesh.
funcpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.stride_from_shape(shape:Tuple[int, ...]) -> List[int]
Compute stride from shape for scatter operations.
funcpython.sglang.multimodal_gen.runtime.utils.mesh3d_utils.transform_pos(mtx:Union[np.ndarray, torch.Tensor], pos:torch.Tensor, keepdim:bool=False) -> torch.Tensor
Transform positions by a matrix.
classpython.sglang.multimodal_gen.runtime.utils.trace_wrapper.DiffStage
Named trace stages for the diffusion pipeline.
funcpython.sglang.multimodal_gen.runtime.utils.vision.load_image(image:str | bytes | PIL.Image.Image, convert_method:Callable[[PIL.Image.Image], PIL.Image.Image] | None=None) -> PIL.Image.Image
Loads `image` to a PIL Image.
funcpython.sglang.multimodal_gen.runtime.utils.vision.normalize(images:np.ndarray | torch.Tensor) -> np.ndarray | torch.Tensor
Normalize an image array to [-1,1].
funcpython.sglang.multimodal_gen.runtime.utils.vision.numpy_to_pt(images:np.ndarray) -> torch.Tensor
Convert a NumPy image to a PyTorch tensor.
funcpython.sglang.multimodal_gen.runtime.utils.vision.resize(image:PIL.Image.Image | np.ndarray | torch.Tensor, height:int, width:int, resize_mode:str='default', resample:str='lanczos') -> PIL.Image.Image | np.ndarray | torch.Tensor
Resize image.
funcpython.sglang.multimodal_gen.runtime.vla.observation.tensor_fingerprint(tensor:torch.Tensor) -> str
Hash tensor content with SRT's CPU/CUDA implementation.
funcpython.sglang.multimodal_gen.utils.get_mixed_precision_state() -> MixedPrecisionState
Get the current mixed precision state.
funcpython.sglang.multimodal_gen.utils.resolve_obj_by_qualname(qualname:str) -> Any
Resolve an object by its fully qualified name.
funcpython.sglang.multimodal_gen.utils.warn_for_unimplemented_methods(cls:type[T]) -> type[T]
A replacement for `abc.ABC`.
funcpython.sglang.srt.arg_groups.deepseek_v4_hook.validate_deepseek_v4_cp(server_args:ServerArgs) -> None
Validate DeepSeek V4 context-parallel configuration.
funcpython.sglang.srt.arg_groups.overrides.run_post_process_pass(server_args:Any, fn:Callable[..., dict]) -> None
Invoke one pass at its legacy handler slot.
funcpython.sglang.srt.batch_invariant_ops.batch_invariant_ops.log_softmax(input:torch.Tensor, dim:int=-1) -> torch.Tensor
Compute log_softmax using Triton kernel.
funcpython.sglang.srt.batch_invariant_ops.batch_invariant_ops.rms_norm(input:torch.Tensor, weight:torch.Tensor, eps:float=1e-06) -> torch.Tensor
Compute RMS normalization using Triton kernel.
funcpython.sglang.srt.compilation.backend.set_model_tag(tag:str)
Context manager to set the model tag.
funcpython.sglang.srt.compilation.inductor_pass.get_pass_context() -> PassContext
Get the current pass context.
classpython.sglang.srt.compilation.pass_manager.PostGradPassManager
The pass manager for post-grad passes.
classpython.sglang.srt.configs.dbrx.DbrxAttentionConfig
Configuration class for Dbrx Attention.
classpython.sglang.srt.configs.dbrx.DbrxConfig
Configuration class for Dbrx.
classpython.sglang.srt.configs.dbrx.DbrxFFNConfig
Configuration class for Dbrx FFN.
classpython.sglang.srt.configs.embedding_model_spec.EmbeddingExecution
The model path that produces an embedding.
classpython.sglang.srt.configs.internvl.InternLM2Tokenizer
Construct a InternLM2 tokenizer.
methodpython.sglang.srt.configs.internvl.InternLM2Tokenizer.get_vocab()
Returns vocab as a dict
methodpython.sglang.srt.configs.internvl.InternLM2Tokenizer.vocab_size()
Returns vocab size
classpython.sglang.srt.configs.kimi_k25.KimiK25Config
K2-VL model configuration.
methodpython.sglang.srt.configs.kimi_k25.KimiK25Config.vocab_size() -> int
Get vocab size from text config for compatibility.
classpython.sglang.srt.configs.lfm2.Lfm2Config
SGLang configuration for LFM2 models.
methodpython.sglang.srt.configs.lfm2.Lfm2Config.mamba_chunk_size() -> int
Return chunk size for Mamba2 backend.
classpython.sglang.srt.configs.lfm2_vl.Lfm2VlConfig
SGLang configuration for LFM2-VL models.
methodpython.sglang.srt.configs.lfm2_vl.Lfm2VlConfig.mamba_chunk_size() -> int
Return chunk size for Mamba2 backend.
methodpython.sglang.srt.configs.model_config.ModelConfig.get_num_kv_heads(tensor_parallel_size) -> int
Returns the number of KV heads per GPU.
methodpython.sglang.srt.configs.model_config.ModelConfig.get_total_num_kv_heads() -> int
Returns the total number of KV heads.
classpython.sglang.srt.connector.azure.AzureBlobConnector
File connector for Azure Blob Storage.
methodpython.sglang.srt.connector.serde.serde.Deserializer.from_bytes(bs:bytes) -> torch.Tensor
Deserialize a pytorch tensor from bytes.
methodpython.sglang.srt.connector.serde.serde.Serializer.to_bytes(t:torch.Tensor) -> bytes
Serialize a pytorch tensor to bytes.
funcpython.sglang.srt.connector.utils.parse_model_name(url:str) -> str
Parse the model name from the url.
classpython.sglang.srt.debug_utils.comparator.aligner.token_aligner.smart.types.TokenAlignerPlan
Token alignment plan.
funcpython.sglang.srt.debug_utils.comparator.dims_spec.tensor_naming.find_dim_index(dim_specs:list[DimSpec], name:str) -> Optional[int]
Find index by name.
funcpython.sglang.srt.debug_utils.comparator.dims_spec.tensor_naming.get_dim_names(tensor:torch.Tensor) -> tuple[Optional[str], ...]
Get dimension names attached to a tensor.
methodpython.sglang.srt.debug_utils.comparator.dims_spec.types.DimSpec.sub_dims() -> list[str]
Sub-dim names.
funcpython.sglang.srt.debug_utils.comparator.entrypoint.parse_args(argv:list[str]) -> argparse.Namespace
Parse CLI arguments from an argv list.
classpython.sglang.srt.debug_utils.comparator.report_sink.ReportSink
Unified entry point for all record output.
funcpython.sglang.srt.debug_utils.comparator.utils.calc_per_token_rel_diff(x:torch.Tensor, y:torch.Tensor, *seq_dim:int) -> torch.Tensor
Cosine-distance-like metric per token position.
classpython.sglang.srt.debug_utils.source_patcher.types.PatchApplicationError
match text not found or not unique in source.
classpython.sglang.srt.disaggregation.base.conn.BaseKVManager
Base class for managing transfer states
classpython.sglang.srt.disaggregation.common.staging_handler.DecodeStagingContext
Staging-specific context for decode mode.
methodpython.sglang.srt.disaggregation.common.staging_handler.DecodeStagingHandler.create(kv_manager, scheduler, tp_rank:int) -> DecodeStagingHandler
Factory: create handler.
methodpython.sglang.srt.disaggregation.common.staging_handler.DecodeStagingHandler.is_staging_room(room:int) -> bool
Check if a room is registered for staging scatter.
methodpython.sglang.srt.disaggregation.common.staging_handler.DecodeStagingHandler.submit_last_scatter_async(room:int) -> bool
Record all-ranks Success.
classpython.sglang.srt.disaggregation.common.staging_handler.PrefillStagingContext
Staging-specific context for prefill mode.
classpython.sglang.srt.disaggregation.decode.DecodePreallocQueue
Store the requests that are preallocating.
methodpython.sglang.srt.disaggregation.decode.DecodePreallocQueue.add(req:Req, is_retracted:bool=False, is_rebootstrap:bool=False) -> None
Add a request to the pending queue.
methodpython.sglang.srt.disaggregation.decode.DecodePreallocQueue.extend(reqs:List[Req], is_retracted:bool=False) -> None
Add a request to the pending queue.
classpython.sglang.srt.disaggregation.decode.DecodeTransferQueue
Store the requests that is polling kv
classpython.sglang.srt.disaggregation.encode_grpc_server.SGLangEncoderServer
gRPC service implementation for SGLang encoder.
funcpython.sglang.srt.disaggregation.encode_receiver.extract_original_req_id(part_req_id:str) -> str
Extract the original request ID from a part request ID.
funcpython.sglang.srt.disaggregation.encode_receiver.video_meta_attrs_for(model_type:Optional[str]) -> tuple
Video-meta attrs for model_type.
classpython.sglang.srt.disaggregation.kv_events.KVCacheEvent
Base class for all KV cache-related events
classpython.sglang.srt.disaggregation.kv_events.KVEventsConfig
Configuration for KV event publishing.
methodpython.sglang.srt.disaggregation.kv_events.KVEventsConfig.from_cli(cli_value:str) -> 'KVEventsConfig'
Parse the CLI value for the event publisher config.
classpython.sglang.srt.disaggregation.kv_events.NullEventPublisher
No-op implementation (default when disabled).
classpython.sglang.srt.disaggregation.kv_events.StorageMedium
Storage tier for KV cache events.
classpython.sglang.srt.disaggregation.nixl.conn.TransferInfo
Contains indices for a transfer, sent by KVReceiver.
classpython.sglang.srt.disaggregation.nixl.conn.TransferStatus
Used by KV Receiver to know when a transfer is done.
classpython.sglang.srt.disaggregation.prefill.PrefillBootstrapQueue
Store the requests in bootstrapping
methodpython.sglang.srt.disaggregation.prefill.PrefillBootstrapQueue.finalize_bootstrap(req:Req) -> bool
Initialize the sender after bootstrap completes.
funcpython.sglang.srt.disaggregation.prefill.should_force_retry(req:Req) -> bool
Test hook to force a request into optimistic prefill retry.
funcpython.sglang.srt.distributed.device_communicators.custom_all_reduce_utils.gpu_p2p_access_check(src:int, tgt:int) -> bool
Check if GPU src can access GPU tgt.
funcpython.sglang.srt.distributed.utils.get_global_tcp_store() -> Optional[TCPStore]
Get the existing global TCPStore.
funcpython.sglang.srt.distributed.utils.get_pp_indices(num_hidden_layers:int, pp_rank:int, pp_size:int) -> Tuple[int, int]
Try to evenly distribute layers across partitions.
classpython.sglang.srt.dllm.mixin.scheduler.DllmManager
Manager for Diffusion LLM request scheduling.
methodpython.sglang.srt.dllm.mixin.scheduler.DllmManager.add_waiting_reqs(reqs:Union[Req, List[Req]]) -> None
Add requests to waiting queue with redundancy check.
methodpython.sglang.srt.dllm.mixin.scheduler.DllmManager.any_staging_reqs() -> bool
Check if there are requests in staging queue.
methodpython.sglang.srt.dllm.mixin.scheduler.DllmManager.filter_finished_reqs() -> None
Remove finished requests from both queues.
methodpython.sglang.srt.dllm.mixin.scheduler.DllmManager.get_decode_requests() -> List[Req]
Get all decode requests from waiting queue.
methodpython.sglang.srt.dllm.mixin.scheduler.DllmManager.get_prefill_requests() -> List[Req]
Get all prefill requests from waiting queue.
funcpython.sglang.srt.elastic_ep.elastic_ep.try_admit_scale_ranks(global_ranks:List[int]) -> bool
Admit append-only ranks into the expandable WORLD group.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicBashTool
Anthropic ``bash_*`` server tool family.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicComputerTool
Anthropic ``computer_*`` server tool family.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicCountTokensRequest
Anthropic count_tokens API request.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicCountTokensResponse
Anthropic count_tokens API response.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicError
Error structure for Anthropic API.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicErrorResponse
Error response structure for Anthropic API.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicMessageEndDelta
Delta carried on ``message_delta`` events.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicMessagesRequest
Anthropic Messages API request.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicMessagesResponse
Anthropic Messages API response.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicOutputConfig
Claude 4.7 ``output_config`` block.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicToolChoice
Tool choice strategy.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicUsage
Token usage information.
classpython.sglang.srt.entrypoints.anthropic.protocol.AnthropicWebSearchTool
Anthropic ``web_search_*`` server tool family.
classpython.sglang.srt.entrypoints.anthropic.serving.AnthropicServing
Handler for Anthropic Messages API requests.
funcpython.sglang.srt.entrypoints.elastic_ep.scale_elastic_ep(raw_request:Request)
Request an asynchronous EP scale-up.
classpython.sglang.srt.entrypoints.engine.Engine
The entry point to the inference engine.
methodpython.sglang.srt.entrypoints.engine.Engine.close_session(session_id:str) -> None
Close a session and release its resources.

About this data

These signatures were extracted from the public source of sgl-project/sglang using Python's ast module. Argument names, default values, type annotations and return types are taken verbatim from the code. Implementation bodies are never stored. See how it works for details.

Back to all 805 libraries