# Fork-Required Model Compatibility Assessment

When a model requires a custom llama.cpp fork (e.g., PrismML Bonsai-8B with Q1_0), assess compatibility systematically rather than guessing or attempting a blind build.

## Assessment Steps

### 1. Identify the quantization format

- Check the model card / HuggingFace config for `ggml_type` or quantization labels
- Look for non-standard type names: `Q1_0`, `Q2_0`, or proprietary format identifiers
- If it's a standard GGUF type (Q4_K_M, Q5_K_M, etc.), it works with any upstream llama.cpp — no fork needed

### 2. Check your llama.cpp version

```bash
llama-server --version 2>&1 | head -1
# Returns version + commit hash
```

The `ggml_type` enum in `ggml/src/ggml-common.h` is the ground truth. A tool doesn't need to be bleeding-edge for standard types, but fork-only types won't exist in upstream at all.

### 3. If a fork is required, assess the divergence

- Find the fork repo and compare with upstream master
- Key metrics: commits ahead, commits behind, files changed
- Example: PrismML/llama.cpp was 10 commits ahead but **696 behind** upstream — the fork is stale and diverging fast
- Check whether the fork's backend (CUDA 12.4, 12.8, Vulkan, Metal) matches your hardware

### 4. Determine migration path

| Path | When to choose |
|------|---------------|
| **Download pre-built binary** | Fork provides pre-compiled binaries matching your platform + CUDA version. Fastest option — just swap the binary and restart llama-swap |
| **Build from fork source** | No pre-built binary for your platform, or you need custom flags. Trade-off: stale fork may lack upstream bug fixes |
| **Wait for upstream merge** | Fork is very new (< 1 month), patches are small, and the maintainers are known contributors. Unlikely for proprietary formats |
| **Skip** | Fork is abandoned (>6 months stale), critical security patches missing, or the format has no performance benefit for your use case |

### 5. Verify binary compatibility

- **CUDA version:** `nvcc --version` — must match the fork's build CUDA toolkit version (SMs are backward-compatible within major CUDA versions)
- **Compute capability:** `nvidia-smi --query-gpu=compute_cap --format=csv,noheader` — binary must include your SM version or an older compatible one
- **Shared library path:** If using system-installed llama.cpp, verify `LD_LIBRARY_PATH` or the built binary's `RPATH` resolves to the correct `libllama.so`

### 6. Swap strategy (binary replacement)

```bash
# Backup current binary
cp /usr/local/bin/llama-server /usr/local/bin/llama-server.bak

# Copy fork binary into place
cp <downloaded-fork-binary> /usr/local/bin/llama-server
chmod +x /usr/local/bin/llama-server

# Restart inference stack
sudo systemctl restart llama-swap
sleep 3
curl -s http://127.0.0.1:9292/v1/models | jq '.data[].id'

# Test generation
curl -s -X POST http://127.0.0.1:9292/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"bonsai-8b","messages":[{"role":"user","content":"test"}]}'
```

### 7. Rollback if needed

```bash
cp /usr/local/bin/llama-server.bak /usr/local/bin/llama-server
sudo systemctl restart llama-swap
```

## Pitfalls

- **Stale fork = missing security fixes.** A fork 600+ commits behind upstream is missing months of CVEs and bug fixes. Only use for evaluation, not production
- **Proprietary quantization binds you to the fork.** Q1_0 GGUF files from PrismML cannot be loaded by upstream llama.cpp. You're locked into maintaining that fork
- **Pre-built binaries may not include your SM.** Check the release notes — PrismML built CUDA 12.4 for SM 50–90, but not all fork vendors test across the full GPU range
- **llama-swap doesn't care about the binary swap** as long as the port stays `:9292` and the API stays OpenAI-compatible. The swap is transparent to the proxy