Instructions to use kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ", trust_remote_code=True) model = AutoModel.from_pretrained("kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Kimi-K2.5 ยท REAP (192/384 experts) + MoE-SQ paired 4:8 NVFP4
Rebuttal checkpoint for Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts (MoE-SQ, arXiv 2610.02241): a matched-bit-budget comparison of expert pruning (REAP) and MoE-SQ.
| routed experts | 192 of 384 per layer, top-8 routing unchanged |
| bits per routed-expert weight | 1.375 (counted against all 384 experts' weights, scales and metadata included) |
| size | ~199 GB |
| left in bf16 | attention, shared expert, dense first layer, router, embeddings, lm_head, vision tower |
How it was made
A slice of ISTA-DASLab/Kimi-K2.5-P48NVFP4-MoESQ: the REAP-kept experts' tensors are copied byte-for-byte (paired-4:8 sparse storage), renumbered in order, and the router rows / score-correction bias are sliced; 192 of 384 experts removed per layer. Nothing is recompressed, so the MoE-SQ compression never saw the pruned model. Kept indices from REAP (llm-compressor 0.13.0, mixed calibration, 512 ร 4096) are in reap_kept_experts.jsonl.
Notes:
- Stage 1 and the slicing ran on 8ร NVIDIA B300 (SM103).
Serving
Same stack as the other MoE-SQ / REAP Kimi checkpoints: vLLM v0.30.0 with the MoE-SQ patch
(IST-DASLab/MoESQ, integrations/vllm/install.sh). Multimodal paths are untested; all evaluations are text-only.
License and attribution
Base model moonshotai/Kimi-K2.5 under a Modified MIT License
(see LICENSE, which applies to this checkpoint); upstream notices in THIRD_PARTY_NOTICES.md.
REAP is due to Cerebras Research; the implementation used is llm-compressor's REAPPruningModifier.
- Downloads last month
- 36
Model tree for kwanhee/Kimi-K2.5-REAP-E192-P48NVFP4-MoESQ
Base model
moonshotai/Kimi-K2.5