# Local GGUF Requantization

## When to Use

- You already have a larger quant (e.g. Q4_K_XL, Q5_K_M, Q8_0) and want a smaller one without downloading from HF
- The smaller quant isn't available on HF for your specific model (e.g. uncensored fine-tunes)
- You have a CPU-only or low-VRAM machine and want to offload the work

## Requirements

- `llama-quantize` built with **CPU support only** (no GPU needed — quantization is pure math)
- Build targets: use `cmake -DGGML_CUDA=OFF -DGGML_BLAS=ON` or just plain `-DGGML_CUDA=OFF`
- Recommended: a dedicated CPU-only build directory, e.g. `build-quantize-cpu`

## Build

```bash
# In your llama.cpp checkout
cmake -B build-quantize-cpu -DGGML_CUDA=OFF -DGGML_NATIVE=OFF
cmake --build build-quantize-cpu --config Release --target llama-quantize -j$(nproc)
```

## Usage

```bash
nice -n 19 \
  /path/to/build-quantize-cpu/bin/llama-quantize \
  --allow-requantize \
  input_model.gguf \
  output_model.gguf \
  Q4_K_M 8
```

Parameters:
- Positional arg 3: quantization type (Q4_K_M, Q5_K_M, Q8_0, etc.)
- Positional arg 4: thread count (8 is safe, up to `nproc` for speed)
- `--allow-requantize`: required when input and output quant types differ
- `nice -n 19`: keep the system responsive during this CPU-intensive operation

## Tradeoffs

| Aspect | Download from HF | Local requantize |
|--------|-----------------|------------------|
| Time | Network-dependent (minutes) | CPU-dependent (minutes to hours) |
| Quality | As uploaded by author | Minor cumulative rounding (requantize from Q4_K_XL→Q4_K_M loses an extra ~0.01 bpw) |
| Availability | Only if someone uploaded it | Any model you have on disk |
| Bandwidth | High (15-20 GB per model) | None |

## Notes

- `--allow-requantize` produces "WARNING: N of M tensor(s) required fallback quantization" — this is normal and safe. It means some tensors were already at or below the target precision and couldn't be further quantized.
- The `BPW` (bits per weight) in the output shows both original and quantized sizes. Smaller BPW = smaller file but more quality loss.
- Running on a CPU-only build avoids GPU VRAM conflicts. The process uses as much RAM as the model file size (~16 GB for 26B) plus working space.