How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16# Run inference directly in the terminal:
llama cli -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16# Run inference directly in the terminal:
./build/bin/llama cli -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16Use Docker
docker model run hf.co/thomas-yanxin/Qwen2-VL-7B-GGUF:F16Quick Links
How to use it
- To get the Code:
git clone https://github.com/HimariO/llama.cpp.git
cd llama.cpp
git switch qwen2-vl
- nano Makefile #(to add llama-qwen2vl-cli)
diff --git a/Makefile b/Makefile
index 8a903d7e..51403be2 100644
--- a/Makefile
+++ b/Makefile
@@ -1485,6 +1485,14 @@ libllava.a: examples/llava/llava.cpp \
$(OBJ_ALL)
$(CXX) $(CXXFLAGS) -static -fPIC -c $< -o $@ -Wno-cast-qual
+llama-qwen2vl-cli: examples/llava/qwen2vl-cli.cpp \
+ examples/llava/llava.cpp \
+ examples/llava/llava.h \
+ examples/llava/clip.cpp \
+ examples/llava/clip.h \
+ $(OBJ_ALL)
+ $(CXX) $(CXXFLAGS) $< $(filter-out %.h $<,$^) -o $@ $(LDFLAGS) -Wno-cast-qual
+
llama-llava-cli: examples/llava/llava-cli.cpp \
examples/llava/llava.cpp \
examples/llava/llava.h \
- Metal Build
cmake . -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=$(which nvcc) -DTCNN_CUDA_ARCHITECTURES=61
make -j35
- RUN
./bin/llama-qwen2vl-cli -m ./thomas-yanxin/Qwen2-VL-7B-GGUF/Qwen2-VL-7B-GGUF-Q4_K_M.gguf --mmproj ./thomas-yanxin/Qwen2-VL-7B-GGUF/qwen2vl-vision.gguf -p "Describe the image" --image "./thomas-yanxin/Qwen2-VL-7B-GGUF/1.png"
- Downloads last month
- 132
Hardware compatibility
Log In to add your hardware
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16# Run inference directly in the terminal: llama cli -hf thomas-yanxin/Qwen2-VL-7B-GGUF:F16