# Fleet Audit: Devstral-24B & Moonlight-16B (2026-06-19)

## New Models

### Devstral-Small-2-24B-Instruct-2512-UD-Q5_K_XL

| Property | Value |
|----------|-------|
| Source | `unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF` |
| Quant | UD-Q5_K_XL (~5.3 bits/param) |
| File | 15.6 GB, 16.7B bytes |
| Arch | `mistral3` |
| Params | 23,572,403,200 |
| Layers | 40 |
| n_embd | 5120 |
| n_head | 32 |
| n_kv_head | 8 (GQA) |
| Native ctx | 393,216 (384K) via YaRN |
| Rope freq_base | 10,000,000.0 (custom in GGUF) |
| Rope dim_count | 128 |
| Vocab | 131,072 |
| Scaling | `yarn`, original_ctx=8192, factor=48.0 |
| Chat template | `[INST]` / `[/INST]` (Llama-style) |

**Raw GGUF metadata offsets:**
- arch @ 32: `mistral3` (9 chars)
- block_count @ 966: 0x28 = 40
- context_length @ 1002: 0x00060000 = 393216
- embedding_length @ 1041: 0x00001400 = 5120
- head_count @ 1136: 0x20 = 32
- head_count_kv @ 1181: 0x08 = 8
- rope.freq_base @ 1219: float32
- rope.dimension_count @ 1407: 0x80 = 128
- rope.scaling.type @ 1452: "yarn"
- rope.scaling.factor @ 1502: float48 (48.0)
- rope.scaling.original_context_length @ 1707: 0x2000 = 8192
- feed_forward_length @ next key after embedding_length

**Benchmarks (auto-fit, upstream binary):**
| Context | Speed | VRAM | Cold Start |
|---------|-------|------|------------|
| 32K | 6.6 t/s | 10.2 GB | 13s |
| 65K | 5.5 t/s | 9.8 GB | 13s |
| 131K | 4.6 t/s | 9.4 GB | 13s |

- `--flash-attn on` required
- `--cache-type-k q8_0 --cache-type-v q8_0`
- `--mlock --prio 2` work fine (dense model, no tensor_buft_overrides)
- `--no-host -fitt 512` helps VRAM
- Explicit `-ngl 20` gave 6.02 t/s vs auto-fit 6.55 t/s at 32K
- Dense 24B is too large for fast inference on 11GB VRAM

---

### Moonlight-16B-A3B-Instruct-bruno Q5_K_M

| Property | Value |
|----------|-------|
| Source | `mradermacher/Moonlight-16B-A3B-Instruct-bruno-GGUF` |
| Quant | Q5_K_M |
| File | 11.2 GB, 12 GB on disk |
| Arch | `deepseek2` (deepseek_v3) |
| Layers | 27 |
| n_embd | 2048 |
| n_head | 16 |
| n_kv_head | 1 (MLA) |
| Native ctx | 8192 (8K) |
| Rope dim_count | 64 |
| Chat template | `[INST]` / `[/INST]` (Llama-style) |

**No YaRN scaling** — no `rope.scaling.*` fields in GGUF.

**Raw GGUF metadata offsets:**
- arch @ 32: `deepseek2` (10 chars)
- block_count @ 876: 0x1b = 27
- context_length @ 913: 0x00002000 = 8192
- embedding_length @ 953: 0x00000800 = 2048
- head_count @ 1050: 0x10 = 16
- head_count_kv @ 1096: 0x01 = 1

**Benchmarks:**
| Context | Speed | VRAM | Notes |
|---------|-------|------|-------|
| 8K (native) | 52.4 t/s | 5.8 GB | Clean, working |
| 32K (forced) | 85.5 t/s | 9.8 GB | Output broken — repeated "example" |

**Forced extension failed:** `--override-kv deepseek2.context_length=int:32768 --rope-scaling linear --rope-freq-scale 4.0` loaded and ran at 32K (85.5 t/s) but quality was unusable.

---

## Fleet Benchmarks (2026-06-19 update)

| # | Model | t/s | VRAM | Ctx | Arch | Type |
|---|-------|-----|------|-----|------|------|
| 1 | gemma-12b | 117.8 | 8.8 GB | 131K | gemma3 | Dense |
| 2 | gemma-26b-200k | 63.5 | 10.1 GB | 200K | gemma3 | MoE |
| 3 | nemotron-term-14b | 61.8 | 10.6 GB | 32K | Qwen2 | Dense |
| 4 | moonlight-16b-bruno | 52.4 | 5.8 GB | 8K | deepseek2 | MoE-MLA |
| 5 | qwen36-35b-mtp | 45.8 | 7.3 GB | 250K | qwen2.5 | MoE |
| 6 | nemotron-3-nano | 36.4 | 3.4 GB | 128K | qwen2 | MoE |
| 7 | deepseek-v2-lite | 33.9 | 10.2 GB | 32K | deepseek2 | MoE-MLA |
| 8 | devstral-24b | 6.6 | 10.2 GB | 32K | mistral3 | Dense |

## Redundancy

- **nemotron-term-14b**: Redundant with gemma-12b (faster, better) and qwen36-35b (bigger context, better code)
- **devstral-24b**: Only for 384K native context need. 8-18× slower than alternatives.

## GGUF v3 Metadata Parsing Reference

When `python3 -c "from gguf import GGUFReader; ..."` times out on large files (>5 GB):

```python
import struct

def parse_gguf_v3_metadata(path, field_name):
    """Extract a specific field value from GGUF v3 metadata."""
    with open(path, 'rb') as f:
        buf = f.read(4*1024*1024)  # 4MB covers all metadata
    
    key_bytes = field_name.encode()
    idx = buf.find(key_bytes)
    if idx < 0:
        return None
    
    # Position after the key string + null terminator
    after_key = idx + len(key_bytes)
    # Next: uint32 value_type
    val_type = struct.unpack_from('<I', buf, after_key)[0]
    
    if val_type == 4:  # int32
        return struct.unpack_from('<i', buf, after_key + 4)[0]
    elif val_type == 6:  # float32
        return struct.unpack_from('<f', buf, after_key + 4)[0]
    elif val_type == 8:  # string
        str_len = struct.unpack_from('<Q', buf, after_key + 4)[0]
        return buf[after_key + 4 + 8:after_key + 4 + 8 + str_len].decode('utf-8', errors='replace')
    
    return f"type_{val_type}"
```