Win11 Docker vLLM RTX5090 32G显存 32G内存 推荐部署方案
docker pull --platform linux/amd64 vllm/vllm-openai:qwen38-x86_64-cu130
docker run -d --gpus all --shm-size 24G -p 8000:8000 `
-v E:\.cache\huggingface:/root/.cache/huggingface `
-e HF_ENDPOINT=https://hf-mirror.com `
--name vllm-qwen3.8-nvfp4 `
vllm/vllm-openai:qwen38-x86_64-cu130 `
RadixArk/Qwen3.8-27B-NVFP4 `
--enable-prompt-tokens-details `
--quantization modelopt `
--trust-remote-code `
--served-model-name qwen3.8-27b `
--max-model-len 163840 `
--max-num-seqs 3 `
--max-num-batched-tokens 4096 `
--kv-offloading-size 16 `
--gpu-memory-utilization 0.90 `
--kv-cache-dtype fp8_e4m3 `
--enable-prefix-caching `
--enable-chunked-prefill `
--reasoning-parser qwen3 `
--enable-auto-tool-choice `
--tool-call-parser qwen3_coder `
--chat-template /root/.cache/huggingface/fixed_chat_template.jinja
fixed_chat_template.jinjia用的是froggeric/Qwen-Fixed-Chat-Templates
没开MTP是因为高负荷下会崩溃,所以没开MTP,目前最高3并发,每个并发60+t/s。
Win11 Docker SGLang 目前尝试了很多方法和参数都无法做到 3并发 160k上下文 一直崩溃无法工作
如果有成功的大神,求指教如何部署
感谢分享 vLLM 配置。3 并发 × 160k 在 32GB 显存上 SGLang 是可以跑的,关键是 fp8 KV + HiCache 分层缓存。
先算一下显存:这个模型是 hybrid 架构:64 层里每 4 层有 1 层 full attention(共 16 层),其余 linear attention 的状态大小固定、不随上下文增长。KV cache 只来自这 16 个 full attention 层(4 KV head × 256 head_dim),即每 token 约 64 KB(bf16)/ 32 KB(fp8)。3 并发 × 160k = 48 万 token 全部打满时:
- bf16 KV:约 30 GB——单这一项就塞不下 32 GB 显存
- fp8 KV:约 15 GB——加上 ~15 GB NVFP4 权重、CUDA graph 和激活,刚好卡在 OOM 边缘
vLLM 那边的 --kv-offloading-size 16 对应 SGLang 的 HiCache:--enable-hierarchical-cache 把 GPU 放不下的 KV(包括 hybrid 模型的 linear-attn state)分层卸载到 host 内存,前缀命中时直接取回而不是重算。
5090 上的推荐配置(需要 CUDA ≥ 12.8 的较新镜像,支持 sm_120):
sglang serve \
--model-path RadixArk/Qwen3.8-27B-NVFP4 \
--trust-remote-code \
--mem-fraction-static 0.85 \
--kv-cache-dtype fp8_e4m3 \
--context-length 163840 \
--max-running-requests 3 \
--chunked-prefill-size 4096 \
--cuda-graph-max-bs 4 \
--enable-hierarchical-cache \
--hicache-size 16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
几点说明:
--kv-cache-dtype fp8_e4m3把 KV 显存减半,是这个量级显存下跑长上下文的关键;代价是长程 agentic 任务上有轻微精度损失(日常对话类任务基本无感)。上下文不长且对精度敏感的话可以去掉。--hicache-size单位是 GB,指 host 内存池大小;32GB 内存的机器建议留余量,12~16 比较稳妥。不给的话默认按--hicache-ratio 2.0(host 池 = 2× GPU KV 池)。- 注意运行中请求的 live KV 仍然要在 GPU 上(这一点 vLLM 也一样),fp8 KV 下 480k token 约 15 GB,扣除权重后余量不大;如果实际负载经常打满,可以把
--max-running-requests降到 2。 - MTP 在 SGLang 上是验证过的(model card 推荐配置就开 NEXTN):追加
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4。但 MTP 额外吃显存,建议先不开验证稳定后再叠加。你在 vLLM 上高负载开 MTP 崩溃的问题,欢迎开 issue 附日志,我们也确认下 SGLang 侧有没有同样现象。 - 如果启动阶段就直接崩(而不是跑起来后 OOM),请到 sglang GitHub 开 issue 附完整日志,我们看下 sm_120 上的 kernel 路径。
另外好奇:用 froggeric 的 fixed chat template 是遇到了什么问题?模型自带 template 在 SGLang 上应该可以直接用,如果官方 template 有 bug 欢迎反馈。
非常感谢您的回答,不过我看您提供的命令应该是linux操作系统环境下的RTX 5090的SGLang的启动命令
我目前的操作系统环境是 Windows 11 使用Docker的 lmsysorg/sglang:qwen38-27b 这个镜像 根据您的建议用下面命令启动
docker run -d --gpus all --shm-size 24G -p 8000:30000 `
-v E:\.cache\huggingface:/root/.cache/huggingface `
--name sglang-qwen3.8-nvfp4 `
-e HF_ENDPOINT=https://hf-mirror.com `
-e SGLANG_USE_IPC_POOL_HANDLE_CACHE=0 `
--ipc=host `
lmsysorg/sglang:qwen38-27b `
sglang serve `
--model-path RadixArk/Qwen3.8-27B-NVFP4 `
--trust-remote-code `
--mem-fraction-static 0.85 `
--kv-cache-dtype fp8_e4m3 `
--context-length 65536 `
--max-running-requests 3 `
--chunked-prefill-size 4096 `
--cuda-graph-max-bs-decode 4 `
--enable-hierarchical-cache `
--hicache-size 16 `
--reasoning-parser qwen3 `
--tool-call-parser qwen3_coder `
--served-model-name qwen3.8-27b `
--host 0.0.0.0 `
--port 8000
启动阶段就直接崩溃
这是完整日志
2026-08-25 20:32:55.854 |
2026-08-25 20:32:55.854 | ==========
2026-08-25 20:32:55.854 | == CUDA ==
2026-08-25 20:32:55.854 | ==========
2026-08-25 20:32:55.856 |
2026-08-25 20:32:55.856 | CUDA Version 13.0.3
2026-08-25 20:32:55.857 |
2026-08-25 20:32:55.857 | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2026-08-25 20:32:55.858 |
2026-08-25 20:32:55.858 | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2026-08-25 20:32:55.858 | By pulling and using the container, you accept the terms and conditions of this license:
2026-08-25 20:32:55.858 | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2026-08-25 20:32:55.858 |
2026-08-25 20:32:55.858 | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2026-08-25 20:32:55.867 |
2026-08-25 20:32:55.892 | /sgl-workspace/sglang/python/sglang/srt/environ.py:1402: UserWarning: Environment variable SGLANG_FLASHINFER_PR4266_SOURCE has been deprecated.
2026-08-25 20:32:55.892 | warnings.warn(f"Environment variable {old_name} has been deprecated.")
2026-08-25 20:32:59.861 | W0825 12:32:59.860000 1 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:32:59.883 | W0825 12:32:59.882000 1 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:33:03.270 | Unable to import `torchao` Tensor objects. This may affect loading checkpoints serialized with `torchao`
2026-08-25 20:33:09.210 | [2026-08-25 12:33:09] Attention backend not specified. Use flashinfer backend by default.
2026-08-25 20:33:09.645 | [2026-08-25 12:33:09] Multimodal feature transport auto-resolved to cuda_ipc (single-node CUDA). Pass --mm-feature-transport=cpu to opt out.
2026-08-25 20:33:09.645 | [2026-08-25 12:33:09] Using CUDA IPC for multimodal features: reserving up to 1024 MiB on base GPU 0 across 1 tokenizer worker(s). This reduces KV cache headroom; a full pool falls back to CPU transport.
2026-08-25 20:33:09.645 | [2026-08-25 12:33:09] CUDA IPC pool-handle caching is disabled. It reuses mappings to the existing bounded pool without reserving another pool; set SGLANG_USE_IPC_POOL_HANDLE_CACHE=0 to disable it.
2026-08-25 20:33:11.344 | [2026-08-25 12:33:11] Compiled-kernel caches now live under SGLANG_CACHE_DIR (/root/.cache/sglang). These older directories are no longer used by sglang, but may still be used by other frameworks on this machine, so they were left alone: /root/.cache/flashinfer. Remove them yourself if nothing else needs them.
2026-08-25 20:33:11.347 | [2026-08-25 12:33:11] server_args=ServerArgs(model_path='RadixArk/Qwen3.8-27B-NVFP4', tokenizer_path='RadixArk/Qwen3.8-27B-NVFP4', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=65536, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.85, max_running_requests=3, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, retraction_policy='length', schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr=None, nnodes=1, node_rank=0, tp_size=1, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dwdp_size=1, dcp_comm_backend='ag_rs', dcp_replicate_q_proj=None, enable_prefill_cp=False, cp_strategy=None, enable_dsa_cache_layer_split=False, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_cp_decode_attn_tp=False, enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=811275110, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='0.0.0.0', port=8000, fastapi_root_path='', smg_grpc_mode=False, grpc_mode=False, grpc_port=None, sidecar=None, sidecar_args=None, skip_server_warmup=False, warmups=None, enable_http2=False, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='qwen3.8-27b', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='qwen3', default_chat_template_kwargs=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser='qwen3_coder', tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=False, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, smg_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, flashinfer_autotune_skip_ops=None, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='full', max_bs=4, bs=[1, 2, 4], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None), prefill=PhaseConfig(backend='breakable', max_bs=4096, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None)), cuda_graph_backend_decode=None, cuda_graph_backend_prefill=None, cuda_graph_max_bs_decode=4, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, enable_scattered_sconv=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, torchao_config='', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_dflash_block_size=None, speculative_dspark_block_size=None, speculative_dspark_sps_table_path=None, speculative_dspark_confidence_sts_path=None, speculative_dspark_align_verify_tokens_to_graph_tier=False, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_window_size=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, _speculative_draft_quantization_explicitly_set=False, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=1, moe_a2a_backend='none', deepep_v2_mode='direct', deepep_v2_dispatcher_output_dtype='auto', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', fuseep_mode=2, deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_waterfill=False, ep_join_mode=None, ep_join_rank_offset=0, elastic_ep_initial_size=None, max_ep_size=None, elastic_ep_scale_timeout=600, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_max_states_per_path=-1, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='extra_buffer', uses_mamba_radix_cache=True, mamba_track_interval=256, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, linear_attn_verify_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_linear_replayssm_spec=False, enable_hierarchical_cache=True, hicache_ratio=2.0, hicache_size=16, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, mm_processor_worker_num=0, mm_io_worker_num=0, limit_mm_data_per_request=None, enable_mm_global_cache=False, disable_fast_image_processor=False, mm_feature_transport='cuda_ipc', keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, enable_flexkv=False, flexkv_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, dllm_fdfo=True, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_attempts=0, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, return_hidden_states_mode=None, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, weight_cache_mode='off', weight_cache_socket=None, weight_cache_timeout=1800, forward_hooks=None, msprobe_dump_config=None)
2026-08-25 20:33:11.375 | /sgl-workspace/sglang/python/sglang/srt/environ.py:1402: UserWarning: Environment variable SGLANG_FLASHINFER_PR4266_SOURCE has been deprecated.
2026-08-25 20:33:11.375 | warnings.warn(f"Environment variable {old_name} has been deprecated.")
2026-08-25 20:33:14.965 | W0825 12:33:14.964000 81 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:33:14.981 | W0825 12:33:14.980000 81 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:33:15.464 | /sgl-workspace/sglang/python/sglang/srt/environ.py:1402: UserWarning: Environment variable SGLANG_FLASHINFER_PR4266_SOURCE has been deprecated.
2026-08-25 20:33:15.464 | warnings.warn(f"Environment variable {old_name} has been deprecated.")
2026-08-25 20:33:19.062 | W0825 12:33:19.061000 124 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:33:19.076 | W0825 12:33:19.075000 124 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
2026-08-25 20:33:20.683 | [2026-08-25 12:33:20] Ignore import error when loading sglang.srt.multimodal.processors.inkling: No module named 'helion'
2026-08-25 20:33:21.423 | [2026-08-25 12:33:21] Ignore import error when loading sglang.srt.multimodal.processors.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
2026-08-25 20:33:29.316 | [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
2026-08-25 20:33:30.286 | [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
2026-08-25 20:33:38.084 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.multimodal.processors.inkling: No module named 'helion'
2026-08-25 20:33:38.239 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.multimodal.processors.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
2026-08-25 20:33:38.245 | [2026-08-25 12:33:38] Multimodal data loading enabled with 16 worker threads (auto).
2026-08-25 20:33:38.356 | [2026-08-25 12:33:38] Init torch distributed begin.
2026-08-25 20:33:38.415 | [2026-08-25 12:33:38] Init torch distributed ends. elapsed=0.06 s, mem usage=0.11 GB
2026-08-25 20:33:38.565 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.models.inkling: No module named 'helion'
2026-08-25 20:33:38.669 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.models.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
2026-08-25 20:33:38.703 | [2026-08-25 12:33:38] Load weight begin. avail mem=30.08 GB
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Using ModelOptModelLoader due to ModelOpt quantization config.
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] ModelOptModelLoader: Loading base model...
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Model is already quantized, loading directly...
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Detected ModelOpt FP8 checkpoint. The format is experimental and subject to change.
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Detected fp8 checkpoint.
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Detected nvfp4 checkpoint. Please note that the format is experimental and subject to change.
2026-08-25 20:33:38.704 | [2026-08-25 12:33:38] Detected nvfp4 checkpoint. Please note that the format is experimental and subject to change.
2026-08-25 20:33:38.797 | [2026-08-25 12:33:38] Multimodal attention backend not set. Use triton_attn.
2026-08-25 20:33:38.797 | [2026-08-25 12:33:38] Using triton_attn as multimodal attention backend.
2026-08-25 20:33:38.822 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.models.inkling: No module named 'helion'
2026-08-25 20:33:38.840 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.845 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.850 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.851 | [2026-08-25 12:33:38] using attn output gate!
2026-08-25 20:33:38.855 | [2026-08-25 12:33:38] Ignore import error when loading sglang.srt.models.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
2026-08-25 20:33:38.861 | [2026-08-25 12:33:38] Multimodal data loading enabled with 16 worker threads (auto).
2026-08-25 20:33:38.868 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.873 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.881 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.890 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.894 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.898 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.907 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.911 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.917 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.927 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.931 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.935 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.943 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.948 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.954 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.962 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.967 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.972 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.984 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.991 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:38.996 | [2026-08-25 12:33:38] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.004 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.009 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.012 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.020 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.026 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.031 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.041 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.046 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.052 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.060 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.064 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.067 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.078 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.082 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.086 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.093 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.097 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.101 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.110 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.116 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.121 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.129 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.133 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.138 | [2026-08-25 12:33:39] Enabled fused SiLU+mul+FP4-quant for dense MLP down_proj input.
2026-08-25 20:33:39.189 | [2026-08-25 12:33:39] Found local HF snapshot for RadixArk/Qwen3.8-27B-NVFP4 at /root/.cache/huggingface/hub/models--RadixArk--Qwen3.8-27B-NVFP4/snapshots/319f741cce68d7914884900c138a1fbb70a42f30; skipping download.
2026-08-25 20:33:40.709 |
2026-08-25 20:33:40.709 | Multi-thread loading shards: 0% Completed | 0/3 [00:00<?, ?it/s][2026-08-25 12:33:40] Multimodal processor concurrency enabled with 2 isolated worker threads (auto).
2026-08-25 20:33:40.711 | [2026-08-25 12:33:40] CUDA IPC multimodal feature pools reserve 1024 MiB total on GPU 0 (1024 MiB per tokenizer worker × 1; configured budget 1024 MiB).
2026-08-25 20:33:40.827 | [2026-08-25 12:33:40] Using default HuggingFace chat template with detected content format: openai
2026-08-25 20:33:41.089 | [2026-08-25 12:33:41] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen3_coder
2026-08-25 20:37:40.116 |
2026-08-25 20:37:40.116 | Multi-thread loading shards: 33% Completed | 1/3 [01:52<03:45, 112.54s/it]
2026-08-25 20:37:40.116 | Multi-thread loading shards: 67% Completed | 2/3 [03:45<01:52, 112.98s/it]
2026-08-25 20:37:40.116 | Multi-thread loading shards: 100% Completed | 3/3 [03:59<00:00, 67.85s/it]
2026-08-25 20:37:40.116 | Multi-thread loading shards: 100% Completed | 3/3 [03:59<00:00, 79.99s/it]
2026-08-25 20:37:46.380 | [2026-08-25 12:37:46] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results!
2026-08-25 20:37:46.383 | [2026-08-25 12:37:46] Load weight end. elapsed=247.69 s, type=Qwen3_5ForConditionalGeneration, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=9.94 GB, mem usage=20.14 GB.
2026-08-25 20:38:06.893 | [2026-08-25 12:38:06] Mamba Cache is allocated. max_mamba_cache_size: 16, conv_state size: 0.05GB, ssm_state size: 2.39GB
2026-08-25 20:38:07.226 | [2026-08-25 12:38:07] KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 97319, K size: 1.48 GB, V size: 1.48 GB
2026-08-25 20:38:07.227 | [2026-08-25 12:38:07] Memory pool end. avail mem=2.96 GB
2026-08-25 20:38:07.440 | [2026-08-25 12:38:07] Linear attention kernel backend: decode=triton, prefill=triton, verify=triton
2026-08-25 20:38:07.440 | [2026-08-25 12:38:07] Using hybrid linear attention backend for hybrid GDN models.
2026-08-25 20:38:07.441 | [2026-08-25 12:38:07] GDN kernel dispatcher: decode=TritonGDNKernel, extend=TritonGDNKernel, verify=TritonGDNKernel packed_decode=True
2026-08-25 20:38:07.445 | [2026-08-25 12:38:07] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.18/sm120/772fea630fdf214a/rank_tp0_pp0_dp0.json
2026-08-25 20:38:08.239 |
2026-08-25 20:38:08.239 | [AutoTuner]: Tuning fp8_gemm: 0%| | 0/3 [00:00<?, ?profile/s]
2026-08-25 20:38:08.239 | [AutoTuner]: Tuning fp8_gemm: 100%|██████████| 3/3 [00:00<00:00, 50.34profile/s]
2026-08-25 20:38:08.348 | [2026-08-25 12:38:08] Using fused GDN decode QKVZ/BA unpack + indexed Conv1D.
2026-08-25 20:38:08.927 |
2026-08-25 20:38:08.927 | [AutoTuner]: Tuning fp8_gemm: 0%| | 0/3 [00:00<?, ?profile/s]
2026-08-25 20:38:08.927 | [AutoTuner]: Tuning fp8_gemm: 100%|██████████| 3/3 [00:00<00:00, 62.44profile/s]
2026-08-25 20:38:09.423 |
2026-08-25 20:38:09.423 | [AutoTuner]: Tuning fp4_gemm: 0%| | 0/3 [00:00<?, ?profile/s]
2026-08-25 20:38:09.423 | [AutoTuner]: Tuning fp4_gemm: 67%|██████▋ | 2/3 [00:00<00:00, 11.91profile/s]
2026-08-25 20:38:09.423 | [AutoTuner]: Tuning fp4_gemm: 100%|██████████| 3/3 [00:00<00:00, 12.02profile/s]
2026-08-25 20:38:09.589 |
2026-08-25 20:38:09.589 | [AutoTuner]: Tuning fp4_gemm: 0%| | 0/3 [00:00<?, ?profile/s]
2026-08-25 20:38:09.589 | [AutoTuner]: Tuning fp4_gemm: 67%|██████▋ | 2/3 [00:00<00:00, 18.43profile/s]
2026-08-25 20:38:09.589 | [AutoTuner]: Tuning fp4_gemm: 100%|██████████| 3/3 [00:00<00:00, 18.26profile/s]
2026-08-25 20:38:09.623 |
2026-08-25 20:38:09.623 | [AutoTuner]: Tuning fp8_gemm: 0%| | 0/3 [00:00<?, ?profile/s]
2026-08-25 20:38:09.623 | [AutoTuner]: Tuning fp8_gemm: 100%|██████████| 3/3 [00:00<00:00, 123.89profile/s]
2026-08-25 20:38:18.548 | [2026-08-25 12:38:18] FlashInfer autotune completed.
2026-08-25 20:38:18.549 | [2026-08-25 12:38:18] Disabling auto-selected prefill CUDA graph: only 2.36 GiB is free after model/KV/eager-buffer allocation; at least 4.00 GiB is required for capture. Set an explicit prefill CUDA graph backend to override this safety gate.
2026-08-25 20:38:18.550 | [2026-08-25 12:38:18] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 3], avail mem=2.36 GB
2026-08-25 20:38:31.060 |
2026-08-25 20:38:31.060 | 0%| | 0/3 [00:00<?, ?it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=3 avail_mem=2.36 GB): 0%| | 0/3 [00:00<?, ?it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=3 avail_mem=2.36 GB): 33%|███▎ | 1/3 [00:00<00:00, 2.49it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=2 avail_mem=2.25 GB): 33%|███▎ | 1/3 [00:00<00:00, 2.49it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=2 avail_mem=2.25 GB): 67%|██████▋ | 2/3 [00:00<00:00, 2.74it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=1 avail_mem=2.24 GB): 67%|██████▋ | 2/3 [00:00<00:00, 2.74it/s]
2026-08-25 20:38:31.060 | Capturing batches (bs=1 avail_mem=2.24 GB): 100%|██████████| 3/3 [00:12<00:00, 5.46s/it]
2026-08-25 20:38:31.060 | Capturing batches (bs=1 avail_mem=2.24 GB): 100%|██████████| 3/3 [00:12<00:00, 4.09s/it]
2026-08-25 20:38:31.298 | [2026-08-25 12:38:31] Capture target decode CUDA graph end. elapsed=12.75 s, mem usage=0.14 GB, avail mem=2.22 GB.
2026-08-25 20:38:31.298 | [2026-08-25 12:38:31] max_total_num_tokens=97319, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=3, context_len=65536, available_gpu_mem=2.22 GB
2026-08-25 20:38:31.299 | [2026-08-25 12:38:31] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>, <ComponentType.MAMBA: 2>). Tree Core: UnifiedTreeCore
2026-08-25 20:38:31.307 | [2026-08-25 12:38:31] attach_hybrid_pool_to_unified_cache failed
2026-08-25 20:38:31.307 | Traceback (most recent call last):
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 1651, in attach_hybrid_pool_to_unified_cache
2026-08-25 20:38:31.307 | result = strategy.build(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 1218, in build
2026-08-25 20:38:31.307 | host_pool_group, cache_controller = build_hybrid_mamba_stack(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 640, in build_hybrid_mamba_stack
2026-08-25 20:38:31.307 | kv_host_pool = build_kv_host_pool(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 101, in build_kv_host_pool
2026-08-25 20:38:31.307 | return kv_host_pool_cls(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/pool_host/mha.py", line 90, in __init__
2026-08-25 20:38:31.307 | super().__init__(
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/pool_host/base.py", line 146, in __init__
2026-08-25 20:38:31.307 | raise ValueError(
2026-08-25 20:38:31.307 | ValueError: Not enough host memory available. Requesting 8.79 GB but only have 8.78 GB free. Please reduce the size of the hierarchical cache.
2026-08-25 20:38:31.307 | [2026-08-25 12:38:31] Scheduler hit an exception: Traceback (most recent call last):
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4966, in run_scheduler_process
2026-08-25 20:38:31.307 | scheduler = Scheduler(
2026-08-25 20:38:31.307 | ^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 511, in __init__
2026-08-25 20:38:31.307 | result = kv_cache_builder.build_kv_cache(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/kv_cache_builder.py", line 268, in build_kv_cache
2026-08-25 20:38:31.307 | tree_cache = create_tree_cache(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/registry.py", line 210, in create_tree_cache
2026-08-25 20:38:31.307 | cache = default_radix_cache_factory(ctx)
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/registry.py", line 115, in default_radix_cache_factory
2026-08-25 20:38:31.307 | return _create_unified_radix_cache(ctx, server_args, params)
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/registry.py", line 189, in _create_unified_radix_cache
2026-08-25 20:38:31.307 | cache.init_hicache(server_args, params)
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/unified_radix_cache.py", line 349, in init_hicache
2026-08-25 20:38:31.307 | attach_hybrid_pool_to_unified_cache(
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 1651, in attach_hybrid_pool_to_unified_cache
2026-08-25 20:38:31.307 | result = strategy.build(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 1218, in build
2026-08-25 20:38:31.307 | host_pool_group, cache_controller = build_hybrid_mamba_stack(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 640, in build_hybrid_mamba_stack
2026-08-25 20:38:31.307 | kv_host_pool = build_kv_host_pool(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 101, in build_kv_host_pool
2026-08-25 20:38:31.307 | return kv_host_pool_cls(
2026-08-25 20:38:31.307 | ^^^^^^^^^^^^^^^^^
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/pool_host/mha.py", line 90, in __init__
2026-08-25 20:38:31.307 | super().__init__(
2026-08-25 20:38:31.307 | File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/pool_host/base.py", line 146, in __init__
2026-08-25 20:38:31.307 | raise ValueError(
2026-08-25 20:38:31.307 | ValueError: Not enough host memory available. Requesting 8.79 GB but only have 8.78 GB free. Please reduce the size of the hierarchical cache.
2026-08-25 20:38:31.307 |
2026-08-25 20:38:31.308 | [2026-08-25 12:38:31] Received sigquit from a child process. It usually means the child failed.
2026-08-25 20:38:31.308 | [2026-08-25 12:38:31] kill_process_tree called: parent_pid=1, include_parent=True, pid=1
2026-08-25 20:38:31.312 | [2026-08-25 12:38:31] kill_process_tree called: parent_pid=1, include_parent=False, pid=1
2026-08-25 20:38:33.178 | [W825 12:38:33.893667129 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
尝试了几次都会崩溃报错,我担心是上下文长度太高导致,所以改为64k上下文,依然如此。还请大神帮忙看下是不是Windows11操作系统下的docker的lmsysorg/sglang:qwen38-27b镜像下无法达成160k上下文 3并发这个目标