# Bonsai-8B / PrismML Q1_0 Fork — Setup & Investigation

## Model: Bonsai-8B (Q1_0)

- **Author:** PrismML (prism-ml on HF)
- **License:** Apache 2.0
- **HuggingFace:** `prism-ml/Bonsai-8B-gguf` (58k downloads, 708 likes)
- **GGUF file:** `Bonsai-8B-Q1_0.gguf` (1.1 GB, Q1_0 quantization)

### Architecture (from model metadata)

- **Base architecture:** Qwen3
- **Parameters:** 8.19 B total
- **Layers:** 36
- **Embedding dim:** 4096
- **Heads:** 32 (query), 8 (KV)
- **Key/Value dim:** 128 each
- **FFN:** 12288
- **Vocab:** 151,669 tokens (BPE, Qwen2 tokenizer)
- **Native context:** 65,536 tokens (`n_ctx_train` in GGUF metadata). Max tested on RTX 2080 Ti 11GB. 131K exceeds VRAM.
- **Rope scaling:** YaRN, factor 4.0, original context 16384
- **Quantization type:** Q1_0 (1-bit, file_type=40 in GGUF)
- **Tensor count:** 399 total (145 f32, 254 q1_0)

### Benchmarks (RTX 2080 Ti, 11GB VRAM, SM75, CUDA 12.6)

**Generation speed** (full GPU offload, -ngl 99):

| Context | VRAM Used | Free | Prompt t/s | Gen t/s |
|---------|-----------|------|------------|---------|
| 2K | 1,608 MiB | 9,394 MiB | 124 | 150 |
| 32K | 6,106 MiB | 4,896 MiB | — | — |
| 49K | 8,490 MiB | 2,512 MiB | — | — |
| 61K | 10,218 MiB | 784 MiB | — | — |
| 65K | 10,714 MiB | 288 MiB | 97 | 147 |

**Max context:** 65,536 tokens — model's native `n_ctx_train`. KV cache at 65K = ~9.2 GiB, total GPU usage = 10.7 GiB (288 MiB headroom). 131K would need ~18 GiB KV cache, exceeding 11GB VRAM.

**Model VRAM breakdown at 2K ctx:**
- Model weights (Q1_0): 1,015 MiB
- KV cache (f16): 288 MiB
- Compute buffers: 304 MiB
- Overhead: 177 MiB
- **Total: 1,608 MiB**

**KV cache scaling:** ~0.14 MiB per token (36 layers × 8 KV heads × 128 dims × 2 bytes × 2 for K+V).

### PrismML Published Benchmark Comparison

| Benchmark | Bonsai 8B | Qwen3 8B (16GB) | Llama 3.1 8B (16GB) |
|-----------|-----------|------------------|---------------------|
| MMLU Redux | 65.7 | 83.0 | 72.9 |
| MuSR | 50.0 | 55.0 | 51.3 |
| GSM8K | 88.0 | 93.0 | 87.0 |
| HumanEval+ | 73.8 | 82.3 | 63.4 |
| IFEval | 79.8 | 81.5 | 76.4 |
| BFCLv3 | 65.7 | 81.0 | 51.5 |
| **Average** | **70.5** | **79.3** | **67.1** |

## PrismML llama.cpp Fork

- **Repo:** https://github.com/PrismML-Eng/llama.cpp
- **Branch:** prism (10 commits ahead of upstream, 696 behind)
- **Stars:** 191, Forks: 42
- **Build version:** b8846-d104cf1b6
- **Compiler:** GCC 11.4.0 (pre-built CUDA 12.4 binary)
- **CUDA arch targets:** sm_50, sm_61, sm_70, sm_75, sm_80
- **Open issues:** 2 (Vulkan performance, Windows CUDA release missing)
- **Latest PR:** #37 AVX-512-VNNI dot for Q1_0/Q2_0 (merged 2 days ago)

### Pre-built Binary Release

Linux CUDA options (https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b8846-d104cf1):
- CUDA 12.4: `llama-prism-b8846-d104cf1-bin-linux-cuda-12.4-x64.tar.gz`
- CUDA 12.8: `llama-prism-b8846-d104cf1-bin-linux-cuda-12.8-x64.tar.gz`
- CPU-only and Vulkan variants also available.

## Segfault Investigation

### Symptom

When spawned by llama-swap (Go process): `ExitError >> signal: segmentation fault, exit code: -1`
When run from shell (bash): works fine.

### Known Parallel

llama-swap issue [#169](https://github.com/mostlygeek/llama-swap/issues/169): exact same error, Jun 2025. Fixed by upgrading past v128 (shlex parsing change in v126). Our binary is newer but still hits this with the Prism fork.

### Root Causes

1. **Missing LD_LIBRARY_PATH** (PRIMARY) — Prism's `libggml-cuda.so.0.9.11` vs system's 0.0.9341. Go's os/exec doesn't pass shell env vars, so `env LD_LIBRARY_PATH=...` in YAML cmd is treated as the program name. Fix: systemd drop-in (`/etc/systemd/system/llama-swap.service.d/env.conf`), not `env` prefix in YAML. Once LD_LIBRARY_PATH was set globally, the binary loaded through llama-swap successfully.

2. **Compiler mismatch** — Pre-built binary (GCC 11.4.0) vs server (GCC 13.3.0). Different ABI/stack alignment triggers SIGSEGV when spawning with Go's os/exec with aggressive threading flags (`--threads 16 --parallel 1`).

3. **Thread scaling** — Minimal flags (`-t 1 -c 2048`) work through llama-swap. Full flags (threads=16, batch=2048/512, 131K ctx) segfault even with LD_LIBRARY_PATH set. Threading edge case in older CUDA backend.

### Workarounds

1. **Minimal flags through llama-swap** — still 150 t/s, Q1_0 is so lightweight threading doesn't matter.
2. **Build Prism fork from source** — compile with server's GCC 13.3 + CUDA 12.6. Eliminates ABI mismatch.
3. **Try CUDA 12.8 pre-built binary** — closer to our CUDA 12.6 than 12.4.
4. **65K native context is the practical max** — model's `n_ctx_train`. Attempting 131K requires ~18 GiB KV cache which exceeds 11 GB VRAM. No point pushing past 65K on a 2080 Ti.

## Files Installed

| Path | Description |
|------|-------------|
| `/usr/local/bin/llama-server-prism` | Prism fork llama-server (v8846) |
| `/usr/local/bin/llama-cli-prism` | Prism fork llama-cli |
| `/usr/local/lib/prism/` | Prism fork shared libraries |
| `/etc/systemd/system/llama-swap.service.d/env.conf` | Sets LD_LIBRARY_PATH for llama-swap children |
| `/models/downloads/Bonsai-8B-Q1_0.gguf` | The 1.1 GB Q1_0 model |