Qwen3.8-9B-Distill (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of empero-ai/Qwen3.8-9B-Distill, quantized using Intel's AutoRound algorithm.

⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (<think> blocks) and multimodal capabilities:

Algorithm: Intel AutoRound
Precision / Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 32 (fine-grained reconstruction fidelity)
Symmetric (sym): True
Calibration Samples (nsamples): 512
Sequence Length (seqlen): 4096
Tuning Iterations (iters): 1000 (Production-grade accuracy)
Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

AutoRound Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound
AutoGPTQ Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-GPTQ
LLM-Compressor / Compressed-Tensors Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor

🚀 Usage & Quickstart

1. High-Throughput Serving via vLLM

# Using the LLM-Compressor / Compressed-Tensors build
vllm serve Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

📊 VRAM & Performance Benefits

Original Model (BF16): ~8–10 GB VRAM required for full context inference
Quantized Model (W4A16 Group 32): ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
Throughput: Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (<think>) reasoning.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.15 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.15-runpod`	`0rsw686010`
PyTorch 2.15 (CUDA 13.4)	13.4	`vishva123/cuda-13.4-pytorch-2.15-runpod`	`0s0rhdfaqy`

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`	`yk0y6j6rpg`
PyTorch 2.14 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.14-runpod`	`gsp4gwx0nw`

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`