Instructions to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

SGLang

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4
```

Gemma 4 E4B Uncensored NVFP4

EAGLE speculative decoding drafter for Gemma 4 26B-A4B Uncensored (TrevorJS) — a 42-layer E4B (EAGLE for Blackwell) model quantized to NVFP4 AWQ using NVIDIA ModelOpt 0.42.0.

Designed for EAGLE-based speculative decoding on NVIDIA DGX Spark (GB10, SM 12.1) and other Blackwell-architecture GPUs.

GitHub Repo — patches, deployment configs, Dockerfile

Model Details

Property	Value
Architecture	Gemma 4 (E4B EAGLE Drafter)
Target Model	TrevorJS/gemma-4-26B-A4B-it-uncensored (26B MoE)
Layers	42 (35 sliding-window + 7 full-attention)
Hidden Size	2560
Attention Heads	8 (2 KV heads), head_dim=256, global_head_dim=512
Sliding Window	512 tokens
Max Context	131,072 tokens
Quantization	NVFP4 AWQ (ModelOpt 0.42.0)
Model Size	9.6 GB
Vocabulary	262,144 tokens

Quick Start

Prerequisites

Target model — Any NVFP4-quantized Gemma 4 26B MoE (e.g., AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4)
This drafter model — Download below
Three vLLM patches — Required for Gemma 4 speculative decoding (see Required Patches)
Pre-built container — ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

1. Download both models

pip install -U huggingface-hub

# Target model (26B MoE)
huggingface-cli download AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/trevorjs-26b

# This drafter model (E4B)
huggingface-cli download AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/e4b-drafter

2. Get the patched vLLM files

Three patches to vLLM 0.19.1 are required for Gemma 4 speculative decoding. Download from the DECKARD 31B GitHub repo:

for f in eagle_patched.py serving_chat_patched.py modelopt_patched.py; do
  curl -LO https://raw.githubusercontent.com/AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4/main/$f
done

3. Launch with Docker Compose

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest
    container_name: vllm-trevorjs-26b-spec
    restart: unless-stopped
    network_mode: host
    volumes:
      - ~/models/trevorjs-26b:/models/target
      - ~/models/e4b-drafter:/models/e4b-drafter
      - ./modelopt_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
      - ./eagle_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py
    environment:
      - VLLM_TEST_FORCE_FP8_MARLIN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_USE_FLASHINFER_MOE_FP4=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - bash
      - -c
      - |
        exec vllm serve /models/target \
          --served-model-name trevorjs-26b \
          --quantization modelopt \
          --dtype auto \
          --kv-cache-dtype fp8 \
          --tensor-parallel-size 1 \
          --max-model-len 131072 \
          --max-num-seqs 4 \
          --gpu-memory-utilization 0.65 \
          --trust-remote-code \
          --host 0.0.0.0 --port 8000 \
          --enable-chunked-prefill \
          --enable-prefix-caching \
          --enable-auto-tool-choice \
          --tool-call-parser gemma4 \
          --reasoning-parser gemma4 \
          --speculative-config '{"method":"draft_model","model":"/models/e4b-drafter","num_speculative_tokens":5,"quantization":"modelopt"}'
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box, and a spec-decode drafter's verify buffers are not counted by the fraction so leave headroom (0.65 here). Discrete-VRAM GPUs can run higher.

Speculative Config Parameters

Parameter	Value	Description
`method`	`draft_model`	EAGLE-based draft model speculative decoding
`model`	`/models/e4b-drafter`	Path to this E4B drafter model
`num_speculative_tokens`	`5`	Number of tokens the drafter proposes per step
`quantization`	`modelopt`	Required — drafter uses NVFP4 ModelOpt format

Required vLLM Patches

Speculative decoding with Gemma 4 requires three patches to vLLM 0.19.1. Without these, the server will crash on startup.

Patch 1: `eagle_patched.py` — Gemma 4 spec decode support

File: vllm/v1/spec_decode/eagle.py

Three fixes are needed:

1a. Remove multimodal guard

vLLM 0.19.1 calls _raise_if_multimodal() which blocks ALL multimodal targets from speculative decoding, even when the drafter is text-only. Remove this call — the downstream code already handles text-only drafters with multimodal targets correctly.

# In initialize() method — REMOVE this line:
# self._raise_if_multimodal()

1b. Add Gemma4 to model whitelist

Gemma 4 uses image_token_id (258880) but NOT image_token_index. The spec decode framework needs an explicit mapping:

# In the model whitelist check, add Gemma4:
if self.get_model_name(target_model) in [
    "Qwen2_5_VLForConditionalGeneration",
    # ... existing models ...
    "Gemma4ForConditionalGeneration",  # ADD THIS
]:
    self.model.config.image_token_index = target_model.config.image_token_id

1c. Multi-group KV cache support

Gemma 4 uses heterogeneous attention: head_dim=256 for sliding-window layers and head_dim=512 for global attention layers. This creates two distinct KV cache groups. The spec decode framework assumes a single group.

Fix validate_same_kv_cache_group to log a warning instead of asserting, and rewrite initialize_attn_backend to key attention groups by (backend_class, kv_cache_group_id) instead of just backend_class, mapping each draft layer to its correct KV cache group.

Patch 2: `serving_chat_patched.py` — Non-streaming reasoning parser

File: vllm/entrypoints/openai/chat_completion/serving.py

Gemma 4's reasoning parser uses <|channel> (token 100) and <channel|> (token 101) delimiters. With skip_special_tokens=True (the default for non-streaming), these are stripped, causing extract_reasoning() to return None — thinking content lands in the content field.

The fix re-decodes from raw token_ids with skip_special_tokens=False when text-based extraction fails:

if reasoning is None and token_ids and hasattr(reasoning_parser, 'start_token_id'):
    token_ids_list = list(token_ids)
    if reasoning_parser.start_token_id in token_ids_list:
        full_text = reasoning_parser.model_tokenizer.decode(
            token_ids_list, skip_special_tokens=False
        )
        reasoning, content = reasoning_parser.extract_reasoning(full_text, request=request)
        if content:
            _tok = reasoning_parser.model_tokenizer
            content = _tok.decode(_tok.encode(content), skip_special_tokens=True)

Patch 3: `modelopt_patched.py` — NVFP4 AWQ support

File: vllm/model_executor/layers/quantization/modelopt.py

Three fixes:

FP8 NaN scrubbing — ModelOpt 0.42.0 produces ~60 FP8 NaN values (0x7F/0xFF) in weight_scale tensors. Scrubs to zero at load time.
NVFP4_AWQ quant_algo — Registers NVFP4_AWQ (upstream only handles NVFP4).
AWQ pre_quant_scale — Loads and applies per-channel pre_quant_scale tensors for AWQ weight redistribution.

Applying the Patches

Mount as volume binds in Docker Compose (shown above), or copy manually:

VLLM_PATH=$(python3 -c "import vllm; print(vllm.__path__[0])")
cp eagle_patched.py    $VLLM_PATH/v1/spec_decode/eagle.py
cp serving_chat_patched.py $VLLM_PATH/entrypoints/openai/chat_completion/serving.py
cp modelopt_patched.py $VLLM_PATH/model_executor/layers/quantization/modelopt.py

Heterogeneous Attention Architecture

This E4B drafter mirrors the Gemma 4 heterogeneous attention design:

35 sliding-window layers — head_dim=256, window of 512 tokens, default RoPE (theta=10000)
7 full-attention layers — head_dim=512, global attention, proportional RoPE (theta=1M, partial_rotary_factor=0.25)

This creates two distinct KV cache groups within the drafter, handled by the multi-group KV cache fix in eagle_patched.py.

Cross-Model Drafter Compatibility

This drafter was derived from the TrevorJS uncensored fine-tune. It can also be used with other Gemma 4 targets that share the same vocabulary (262K tokens):

Target Model	Expected Compatibility
TrevorJS 26B Uncensored	Best — same base model
DECKARD 31B Dense	Good — same vocab, different fine-tune
SuperGemma4 26B MoE	Good — same vocab, different abliteration
Official Gemma 4 26B	Moderate — base model, no uncensoring

Acceptance rate will be highest with the matching base model and lower with mismatched fine-tunes, but all combinations will function correctly.

Related Models

Model	Type	Size	Link
TrevorJS 26B MoE NVFP4 (target)	MoE NVFP4	16 GB	HuggingFace
DECKARD 31B AWQ_FULL	Dense NVFP4	20.5 GB	HuggingFace \| GitHub
DECKARD E4B Drafter	EAGLE NVFP4	9.6 GB	HuggingFace \| GitHub
SuperGemma4 26B MoE	MoE NVFP4	~14 GB	HuggingFace \| GitHub
vLLM AWQ Container	Docker	—	GHCR

Hardware Requirements

Target + Drafter combined: ~26 GB (16 GB target + 9.6 GB drafter)
Recommended: NVIDIA DGX Spark (128 GB unified memory) or any GPU with >= 40 GB VRAM
Required: Blackwell architecture (SM 10.0+) for native FP4

License

This model inherits the Gemma license from Google.

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

₿ Bitcoin (BTC)

_{bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4}

Ξ Ethereum (ETH)

_{0x1512667F6D61454ad531d2E45C0a5d1fd82D0500}

◎ Solana (SOL)

_{DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t}

ⓜ Monero (XMR)

_{836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd}

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.

Downloads last month: 2,648

Safetensors

Model size

6B params

Tensor type

BF16

Model tree for AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Finetuned

TrevorJS/gemma-4-26B-A4B-it-uncensored

Quantized

(22)

this model

Collection including AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

Gemma 4 — DECKARD HERETIC, Multimodal & Speculators

Collection

Gemma 4 abliterated/quantized — DECKARD HERETIC 31B, SuperGemma4-26B multimodal, 26B-A4B MoE, plus EAGLE3/DFlash drafters. • 13 items • Updated Aug 12 • 8