Run Qwen 35B model on RTX4070 in Win11 via llama.cpp
Clone the code
git clone https://github.com/ggml-org/llama.cpp.git
Follow the guide to run the command below to build the docker
docker build -t local/llama.cpp:full-cuda --target full -f .devops/cuda.Dockerfile .Download models from huggingface, for rtx4070 12g, Q3 is a better option
curl.exe -L -o Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q3_K_P.gguf "https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive/resolve/main/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q3_K_P.gguf?download=true"curl.exe -L -o mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf "https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive/resolve/main/mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf?download=true"rtx4070 roughly can set 16k context size, run with command below (make sure your docker desktop is running first)
docker run --gpus all -p 8090:8080 -v F:\AI\llama.cpp\models:/models --entrypoint /app/llama-server local/llama.cpp:full-cuda -m /models/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q3_K_P.gguf --mmproj /models/mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf -ngl 14 --ctx-size 16384 -c 16384 --parallel 1 -fa 1 --cache-type-k q4_0 --cache-type-v q4_0 --host 0.0.0.0 --port 8080token speed is around 15 t/s, my desktop env:
intel i13700k, ram ddr5 64gb, graphic rtx4070 12g, 1t ssd
sometimes it’s a bit slow to use it. then we can try some light model:
curl.exe -L -o Qwen3.5-9B-Coder.Q4_K_M.gguf https://huggingface.co/mradermacher/Qwen3.5-9B-Coder-GGUF/resolve/main/Qwen3.5-9B-Coder.Q4_K_M.gguf
and also add a flag to let is start new session when content size limit is hit:
docker run –gpus all -p 8090:8080 -v F:\AI\llama.cpp\models:/models –entrypoint /app/llama-server local/llama.cpp:full-cuda -m /models/Qwen3.5-9B-Coder.Q4_K_M.gguf -ngl 99 –parallel 1 -fa 1 –cache-type-k q4_0 –cache-type-v q4_0 –host 0.0.0.0 –port 8080 –cont-batching –context-shift
or you can also download other models:
curl.exe -L -o Qwen3.5-9B-Q4_K_M.gguf https://huggingface.co/jc-builds/Qwen3.5-9B-VLM-Q4_K_M-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf
curl.exe -L -o mmproj-F16.gguf https://huggingface.co/jc-builds/Qwen3.5-9B-VLM-Q4_K_M-GGUF/resolve/main/mmproj-F16.gguf
docker run –gpus all -p 8090:8080 -v F:\AI\llama.cpp\models:/models –entrypoint /app/llama-server local/llama.cpp:full-cuda -m /models/Qwen3.5-9B-Q4_K_M.gguf –mmproj /models/mmproj-F16.gguf -ngl 99 –parallel 1 -fa 1 –cache-type-k q4_0 –cache-type-v q4_0 –host 0.0.0.0 –port 8080 –cont-batching –context-shift