--- name: llama-cpp description: llama.cpp local GGUF inference + HF Hub model discovery. version: 2.1.3 author: Orchestra Research license: MIT dependencies: [llama-cpp-python>=0.2.0] platforms: [linux, macos, windows] metadata: hermes: tags: [llama.cpp, GGUF, Quantization, Hugging Face Hub, CPU Inference, Apple Silicon, Edge Deployment, AMD GPUs, Intel GPUs, NVIDIA, URL-first] --- # llama.cpp + GGUF Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp. ## When to use - Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs - Find the right GGUF for a specific Hugging Face repo - Build a `llama-server` or `llama-cli` command from the Hub - Search the Hub for models that already support llama.cpp - Enumerate available `.gguf` files and sizes for a repo - Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM ## Model Discovery workflow Prefer URL workflows before asking for `hf`, Python, or custom scripts. 1. Search for candidate repos on the Hub: - Base: `https://huggingface.co/models?apps=llama.cpp&sort=trending` - Add `search=` for a model family - Add `num_parameters=min:0,max:24B` or similar when the user has size constraints 2. Open the repo with the llama.cpp local-app view: - `https://huggingface.co/?local-app=llama.cpp` - When multiple providers offer the same model, compare download counts at `https://huggingface.co/api/models?search=&sort=downloads` and use the tree API on each repo to compare actual file sizes (see `references/cross-provider-comparison.md`) 3. Treat the local-app snippet as the source of truth when it is visible: - copy the exact `llama-server` or `llama-cli` command - report the recommended quant exactly as HF shows it 4. Read the same `?local-app=llama.cpp` URL as page text or HTML and extract the section under `Hardware compatibility`: - prefer its exact quant labels and sizes over generic tables - keep repo-specific labels such as `UD-Q4_K_M` or `IQ4_NL_XL` - if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance 5. Query the tree API to confirm what actually exists: - `https://huggingface.co/api/models//tree/main?recursive=true` - keep entries where `type` is `file` and `path` ends with `.gguf` - use `path` and `size` as the source of truth for filenames and byte sizes - separate quantized checkpoints from `mmproj-*.gguf` projector files and `BF16/` shard files - use `https://huggingface.co//tree/main` only as a human fallback 6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant: - shorthand quant selection: `llama-server -hf :` - exact-file fallback: `llama-server --hf-repo --hf-file ` 7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files. ## Quick start ### Install llama.cpp ```bash # macOS / Linux (simplest) brew install llama.cpp ``` ```bash winget install llama.cpp ``` ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release ``` ### Install from source with CUDA (Debian / GCC 14) Building with CUDA on Debian with GCC 14 requires a compatibility workaround — CUDA 12.x `nvcc` rejects GCC >13. **Before updating, always backup existing binaries:** ```bash # Backup to source tree for future reference mkdir -p ~/src/llama.cpp/backups-sm61 sudo cp /usr/local/bin/llama-server /usr/local/bin/llama-server.bak \ /usr/local/bin/llama-server-turbo /usr/local/bin/llama-server-turbo.bak \ ~/src/llama.cpp/backups-sm61/ ``` **CUDA 12.6 with nvcc not on PATH (Debian Hermes host):** When CUDA toolkit is at `/usr/local/cuda-12.6` and nvcc isn't in `PATH`, cmake's `enable_language(CUDA)` fails because it can't find the compiler. Set all three: ```bash CUDAHOSTCXX=/usr/bin/g++-13 cmake .. \ -DCMAKE_C_COMPILER=/usr/bin/gcc-13 \ -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \ -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc \ -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \ -DCMAKE_CUDA_ARCHITECTURES=75 \ -DGGML_CUDA=ON \ -DGGML_CUDA_FA=ON \ -DGGML_CUDA_GRAPHS=ON \ -DGGML_CPU=ON \ -DGGML_OPENMP=ON \ -DGGML_NATIVE=ON \ -DGGML_LLAMAFILE=ON \ -DGGML_BUILD_EXAMPLES=OFF \ -DGGML_BUILD_TESTS=OFF \ -DGGML_CCACHE=OFF \ -DCMAKE_BUILD_TYPE=Release ``` The key difference from the simpler `CUDAHOSTCXX=...` variant is that when nvcc isn't discoverable via `PATH`, cmake needs `CMAKE_CUDA_COMPILER` (to find nvcc) AND `CMAKE_CUDA_HOST_COMPILER` (to tell nvcc which g++ to use). `CUDAHOSTCXX` alone only works if cmake can already find nvcc. **Cmake flag migration (upstream change around May 2026):** Upstream ggml-org renamed cmake flags from `GGML_CUDA`/`GGML_CUDA_FA` etc. to `LLAMA_CUDA`/`LLAMA_CUDA_FA`. If `-DGGML_CUDA=ON` is silently ignored during cmake configuration (the flag shows up as unused in cache), the build used the newer `LLAMA_*` prefix. Run `cmake -LAH 2>/dev/null | grep -E "LLAMA_CUDA|GGML_CUDA"` on a configured build dir to check which set is active. Forks (EsmaeelNabil MTP-TurboQuant, BoFan MTP-TurboQuant) still use the `GGML_*` prefix as of June 2026. **Clone and build — preferred (static, self-contained):** ```bash git clone https://github.com/ggml-org/llama.cpp /tmp/llama.cpp cd /tmp/llama.cpp && mkdir -p build && cd build CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \ -DGGML_CUDA=ON \ -DGGML_CUDA_FA=ON \ -DCMAKE_BUILD_TYPE=Release \ -DCUDAToolkit_ROOT=/usr/local/cuda \ -DBUILD_SHARED_LIBS=OFF # ⚠️ Required for static binary cmake --build . --target llama-server --config Release -j$(nproc) # Install (keep backups just in case) sudo cp bin/llama-server /usr/local/bin/llama-server-cuda ``` `BUILD_SHARED_LIBS=OFF` produces a fully static 100+ MB binary. Without it, cmake may default to a 9 MB dynamically-linked binary that depends on `.so` files from the build tree (`libllama.so`, `libggml-cuda.so`, etc.) — if `/tmp/llama.cpp/` gets cleaned up later, the installed binary silently fails. **Clone and build — fallback (CUDAHOSTCXX only, may produce dynamic binary):** ```bash git clone https://github.com/ggml-org/llama.cpp /tmp/llama.cpp cd /tmp/llama.cpp && mkdir -p build && cd build # Create nvcc wrapper with GCC 13 compatibility cat > /tmp/nvcc-wrapper.sh << 'WRAPPER' #!/bin/bash exec /usr/local/cuda/bin/nvcc -allow-unsupported-compiler -ccbin /usr/bin/gcc-13 "$@" WRAPPER chmod +x /tmp/nvcc-wrapper.sh cmake .. \ -DGGML_CUDA=ON \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_CUDA_COMPILER=/tmp/nvcc-wrapper.sh \ -DCUDAToolkit_ROOT=/usr/local/cuda cmake --build . --target llama-server --config Release -j$(nproc) # Install (keep backups just in case) sudo cp bin/llama-server /usr/local/bin/llama-server-cuda ``` **Method 2 — CUDAHOSTCXX env var (simpler):** For older fork checkouts where cmake may not find CUDA, set the env var directly instead of using a wrapper: ```bash cd /tmp/llama.cpp && mkdir -p build && cd build CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \ -DGGML_CUDA=ON \ -DGGML_CUDA_FA=ON \ -DCMAKE_BUILD_TYPE=Release \ -DCUDAToolkit_ROOT=/usr/local/cuda cmake --build . --target llama-server --config Release -j$(nproc) ``` This works because `CUDAHOSTCXX` tells nvcc which host compiler to use, bypassing the gcc version check without needing a shell wrapper. **Verify the new build:** ```bash /usr/local/bin/llama-server-cuda --version # Output: version: 1 (XXXXXXX) — new commit hash ``` **KV cache type note:** The default `llama-server` binary (BoFan's MTP-TurboQuant fork) supports TurboQuant KV cache types: `turbo2`, `turbo3`, `turbo4`. Upstream ggml-org builds do NOT have these — they only support `f32`, `f16`, `bf16`, `q8_0`, `q4_0`, `q4_1`, `iq4_nl`, `q5_0`, `q5_1`. Use `f16` or `iq4_nl` as TurboQuant replacements on vanilla builds. **TurboQuant KV cache (turbo2/turbo3/turbo4):** Google Research's KV cache compression (ICLR 2026) using PolarQuant + QJL. ~2-4x compression. Default in the installed `llama-server`. Recommended flags: `-ctk turbo4 -ctv turbo2`. **Pascal + TurboQuant Flash Attention interaction (important):** On Ray's current `/usr/local/bin/llama-server` build (`3c7616b`, BoFan TurboQuant fork), Flash Attention is often auto-enabled on GTX 1080 Ti (CC 6.1), and TurboQuant KV can require it. In the Qwen3.6-35B-A3B benchmark, the log showed: ```text llama_init_from_model: turbo cache types require flash_attn — enabling automatically llama_context: flash_attn = enabled ``` This happened even though the command passed `-fa off`. Dense-model logs for Qwen3-14B, Qwen3.5-9B, DeepSeek-R1-8B, Gemma 3, Gemma 4, and Ministral also showed Flash Attention auto-enabled. Practical rule: with TurboQuant KV, trust the server log over the CLI flag and default to omitted/`auto` unless you are explicitly benchmarking a non-TurboQuant path. **Qwen3 8B/14B long context:** Qwen3 config.json shows `max_position_embeddings: 40960` (40K native) with `rope_theta: 1000000` and no RoPE scaling. For Hermes' 65K minimum on dense Qwen3 models, use llama.cpp's YaRN flags: ```bash --rope-scaling yarn --rope-scale 1.6 --yarn-orig-ctx 40960 \ -c 65536 --ctx-size 65536 ``` For 128K, use `--rope-scale 3.2` with `--yarn-orig-ctx 40960`. This is aggressive (3.2x extrapolation) — quality may degrade at the far end, but many reports confirm it works. Test before committing. **Qwen thinking-mode blank-content pitfall:** Hybrid-thinking Qwen-family GGUFs may return empty `message.content` and put text under `reasoning_content` when thinking is enabled. If the consumer expects OpenAI `message.content` (Hermes/gateway/tools), add: ```bash --chat-template-kwargs '{"enable_thinking":false}' ``` Retest with `/v1/chat/completions`; this fixed Qwen3.5-9B uncensored returning blank content while text completions worked. **Fork status (updated May 2026):** - **EsmaeelNabil/llama.cpp-mtp-turbo-quant** (Debian Hermes host, RTX 2080 Ti) — all-in-one merge of MTP + TurboQuant, branch `feat/mtp-turboquant-kv-cache`. Based on the newest upstream (May 12, commit `a578bb6b`). Turbo2/3/4 KV cache types, CUDA FA kernels for all turbo combinations, MTP speculative decoding (`--spec-type mtp`). Conflict-resolved merge of am17an MTP + TheTom TurboQuant. - **BoFan-tunning/llama.cpp-MTP-TurboQuant** (Ray's CachyOS machine, GTX 1080 Ti) — based on ~May 4 upstream, MTP fused, crash fixes, works with SWA (Gemma 3/4 ✅), no RDNA2 baggage - **Stormrage34/llama.cpp-turboquant-hip** (legacy) — has RDNA2 baggage, SWA models hang, base too old for Gemma 3. Even the pre-RDNA2 commit (`ba62fc7`) crashes Gemma 3 (SIGSEGV, base too old) **Which fork on which machine:** Debian Hermes host (RTX 2080 Ti, sm_75) → EsmaeelNabil. Ray's CachyOS (GTX 1080 Ti, sm_61) → BoFan. Different SM targets, different fork bases — don't cross-install. **Running two builds side by side:** When you need MTP support for a new model architecture (e.g. Gemma 4 QAT's `gemma4-assistant` MTP drafter, added Jun 7 PR #23398) but also want to keep an existing fork's features (TurboQuant KV cache), build a fresh upstream binary alongside the fork. Clone from `ggml-org/llama.cpp`, build with `cmake -DLLAMA_CUDA=ON` (new prefix), and install as a separate binary like `llama-server-upstream`. Use it only for the models that need the newer arch support. The fork handles the rest. This avoids messy cherry-pick conflicts across divergent codebases. Build recipe (static binary): `BUILD_SHARED_LIBS=OFF` + `CUDAHOSTCXX=/usr/bin/gcc-13`. For CUDA 12.6 where nvcc isn't on PATH, also set `CMAKE_CUDA_COMPILER` and `CMAKE_CUDA_HOST_COMPILER` explicitly. See `references/turboquant.md` for full build instructions, fork comparison, compatibility table, and known issues. ### Run directly from the Hugging Face Hub ```bash llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0 ``` ```bash llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0 ``` ### Run an exact GGUF file from the Hub Use this when the tree API shows custom file naming or the exact HF snippet is missing. ```bash llama-server \ --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \ --hf-file Phi-3-mini-4k-instruct-q4.gguf \ -c 4096 ``` ### OpenAI-compatible server check ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ {"role": "user", "content": "Write a limerick about Python exceptions"} ] }' ``` ## Python bindings (llama-cpp-python) `pip install llama-cpp-python` (CUDA: `CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir`; Metal: `CMAKE_ARGS="-DGGML_METAL=on" ...`). ### Basic generation ```python from llama_cpp import Llama llm = Llama( model_path="./model-q4_k_m.gguf", n_ctx=4096, n_gpu_layers=35, # 0 for CPU, 99 to offload everything n_threads=8, ) out = llm("What is machine learning?", max_tokens=256, temperature=0.7) print(out["choices"][0]["text"]) ``` ### Chat + streaming ```python llm = Llama( model_path="./model-q4_k_m.gguf", n_ctx=4096, n_gpu_layers=35, chat_format="llama-3", # or "chatml", "mistral", etc. ) resp = llm.create_chat_completion( messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is Python?"}, ], max_tokens=256, ) print(resp["choices"][0]["message"]["content"]) # Streaming for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True): print(chunk["choices"][0]["text"], end="", flush=True) ``` ### Embeddings ```python llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35) vec = llm.embed("This is a test sentence.") print(f"Embedding dimension: {len(vec)}") ``` You can also load a GGUF straight from the Hub: ```python llm = Llama.from_pretrained( repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF", filename="*Q4_K_M.gguf", n_gpu_layers=35, ) ``` ## Choosing a quant Use the Hub page first, generic heuristics second. - Prefer the exact quant that HF marks as compatible for the user's hardware profile. - For general chat, start with `Q4_K_M`. - For code or technical work, prefer `Q5_K_M` or `Q6_K` if memory allows. - For very tight RAM budgets, consider `Q3_K_M`, `IQ` variants, or `Q2` variants only if the user explicitly prioritizes fit over quality. - For multimodal repos, mention `mmproj-*.gguf` separately. The projector is not the main model file. - Do not normalize repo-native labels. If the page says `UD-Q4_K_M`, report `UD-Q4_K_M`. ## Extracting available GGUFs from a repo When the user asks what GGUFs exist, return: - filename - file size - quant label - whether it is a main model or an auxiliary projector Ignore unless requested: - README - BF16 shard files - imatrix blobs or calibration artifacts Use the tree API for this step: - `https://huggingface.co/api/models//tree/main?recursive=true` For a repo like `unsloth/Qwen3.6-35B-A3B-GGUF`, the local-app page can show quant chips such as `UD-Q4_K_M`, `UD-Q5_K_M`, `UD-Q6_K`, and `Q8_0`, while the tree API exposes exact file paths such as `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` and `Qwen3.6-35B-A3B-Q8_0.gguf` with byte sizes. Use the tree API to turn a quant label into an exact filename. ## Search patterns Use these URL shapes directly: ```text https://huggingface.co/models?apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending https://huggingface.co/?local-app=llama.cpp https://huggingface.co/api/models//tree/main?recursive=true https://huggingface.co//tree/main ``` ## Output format When answering discovery requests, prefer a compact structured result like: ```text Repo: Recommended quant from HF: