Instructions to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
Use Docker
docker model run hf.co/InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with Ollama:
ollama run hf.co/InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with Docker Model Runner:
docker model run hf.co/InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
- Lemonade
How to use InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.L3-70B-Euryale-v2.1-iMat-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
L3-70B-Euryale-v2.1-iMat-GGUF
Quantized from fp16.
- Weighted quantizations were creating using fp16 GGUF and groups_merged.txt in 88 chunks and n_ctx=512
Recommended Sampler Settings (From Original Model Card)
Temperature - 1.17
min_p - 0.075
Repetition Penalty - 1.10
SillyTavern Instruct Settings:
Context Template: Llama-3-Instruct-Names
Instruct Presets: Euryale-v2.1-Llama-3-Instruct
For a brief rundown of iMatrix quant performance please see this PR
All quants are verified working prior to uploading to repo for your safety and convenience.
Tip: Pick a file size under your GPU's VRAM while still allowing some room for context for best speed. You may need to pad this further depending on if you are running image gen or TTS as well.
Original model card can be found here
- Downloads last month
- 1,278
1-bit
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Model tree for InferenceIllusionist/L3-70B-Euryale-v2.1-iMat-GGUF
Base model
Sao10K/L3-70B-Euryale-v2.1