# llama-swap — Auto-Swapping Proxy for Multiple Local Models

[llama-swap](https://github.com/mostlygeek/llama-swap) (4k+ ⭐) is a Go proxy that sits in front of llama-server instances and **automatically stops the current model and starts the requested one** when it receives an API request with a different model name.

## Why Use It

- **Replaces individual systemd services** per model — no more `sudo systemctl start/stop llama-server-X`
- **Replaces swap scripts** — no more `switch-model` scripts or `remote-switch.py`
- **Automatic VRAM management** — models unload after a configurable TTL (timeout)
- **Single URL for all models** — frontends (LibreChat, OpenCode, Open WebUI) point to one port
- **Built-in Web UI, metrics, logs** — `http://host:9292/ui`, `/metrics` (Prometheus), `/logs`
- **Works with any OpenAI-compatible server** — llama.cpp, vllm, tabbyAPI, stable-diffusion.cpp, etc.

## Installation

```bash
# Homebrew (macOS/Linux)
brew tap mostlygeek/llama-swap
brew install llama-swap

# Pre-built binary from releases
curl -sL https://github.com/mostlygeek/llama-swap/releases/latest/download/llama-swap-linux-amd64.tar.gz \
  | tar xz -C /usr/local/bin llama-swap

# Docker
docker pull ghcr.io/mostlygeek/llama-swap:unified-cuda
```

## Config Structure

```yaml
# ~/.config/llama-swap/config.yaml
listen: :9292

models:
  model-short-name:
    cmd: /usr/local/bin/llama-server -m /models/llm/path/to/model.gguf --port ${PORT} --host 127.0.0.1 [other flags]
    aliases: ["alt-name"]
    ttl: 1800        # seconds before auto-unload after last request
```

Key points:
- `${PORT}` — llama-swap assigns a free port automatically
- `--host 127.0.0.1` — bind to localhost since llama-swap forwards traffic
- Model ID (key under `models`) is used by clients, e.g. `model: qwen3.6-35b`
- `ttl` is in **seconds**, not a duration string (1800 = 30 min)
- `aliases` lets multiple model names route to the same server

## Example Config (8-model fleet)

```yaml
listen: :9292

models:
  qwen3.6-35b:
    cmd: /usr/local/bin/llama-server -m /models/llm/daily/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf --port ${PORT} --host 127.0.0.1 -ngl 99 --no-mmap --mlock --jinja -c 128000 --ctx-size 128000 --batch-size 4096 --ubatch-size 1024 --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 --metrics
    ttl: 1800

  qwen3.5-9b:
    cmd: /usr/local/bin/llama-server -m /models/llm/uncensored/qwen3.5-9b-uncensored-Q4_K_M.gguf --port ${PORT} --host 127.0.0.1 -ngl 99 --no-mmap --mlock --jinja -c 128000 --ctx-size 128000 --batch-size 1024 --ubatch-size 256 --temp 0.7 --top-p 0.8 --min-p 0 --top-k 20 --metrics
    ttl: 1800
```

## Systemd Service

```ini
[Unit]
Description=llama-swap - auto-swapping local LLM proxy
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/llama-swap --config /home/user/.config/llama-swap/config.yaml --listen :9292
Restart=on-failure
RestartSec=5
User=user
Group=user

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now llama-swap.service
```

## Migrating from Systemd Services

1. **Stop and disable** all individual `llama-server-*.service` units:
   ```bash
   sudo systemctl stop llama-server.service llama-server-9b.service ...
   sudo systemctl disable llama-server.service llama-server-9b.service ...
   ```

2. **Create llama-swap config** with the same `-m` paths and flags from each unit's `ExecStart=` line.

3. **Replace `${PORT}` with `${PORT}`** (llama-swap assigns dynamically). Change `--host 0.0.0.0` to `--host 127.0.0.1`.

4. **Start llama-swap** and verify with:
   ```bash
   curl http://localhost:9292/v1/models
   # Should list all configured models
   curl http://localhost:9292/v1/chat/completions -d '{"model":"model-name","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
   ```

## Verifying

```bash
# List available models
curl http://host:9292/v1/models

# Test a specific model (auto-swaps if needed, ~30-90s for first load)
curl http://host:9292/v1/chat/completions -d '{"model":"model-name","messages":[{"role":"user","content":"Say hi"}],"max_tokens":5}'

# Web UI
open http://host:9292/ui

# Check running models
curl http://host:9292/running

# Metrics (Prometheus)
curl http://host:9292/metrics
```

## Debugging Process Failures

When llama-swap reports `upstream command exited prematurely but successfully` or `signal: segmentation fault`, the model binary itself is crashing. Debug methodically:

### 1. Run the binary directly to confirm it works outside llama-swap

```bash
timeout 30 /usr/local/bin/llama-server -m /models/downloads/model.gguf --port 15999 --host 127.0.0.1 -ngl 99 --no-mmap -c 4096 --ctx-size 4096 2>&1 | head -5
```

If it works directly but fails through llama-swap, the issue is in the spawning environment.

### 2. Check LD_LIBRARY_PATH contamination

The most common cause is a global `LD_LIBRARY_PATH` making the child load wrong shared libraries:

```bash
# Check if llama-swap inherits LD_LIBRARY_PATH
sudo cat /proc/$(pgrep -x llama-swap)/environ | tr '\0' '\n' | grep LIBRARY

# Check for systemd drop-in files overriding the service env
ls /etc/systemd/system/llama-swap.service.d/
cat /etc/systemd/system/llama-swap.service.d/*.conf
sudo systemctl show llama-swap -p Environment
```

If `LD_LIBRARY_PATH` is set globally (e.g. to `/usr/local/lib/prism`), it overrides each binary's RUNPATH. A binary like `llama-server-sm75` that links against build-specific shared objects will load the wrong versions and SIGSEGV. **Remove the global LD_LIBRARY_PATH** — per-model wrappers (like `llama-server-prism-wrapper`) handle their own library path.

### 3. Check for orphaned llama-server processes holding VRAM

After killing llama-swap, zombie child processes may still hold CUDA memory:

```bash
sudo lsof /dev/nvidia* | grep llama
sudo kill -9 <PID>
```

If `nvidia-smi` shows memory used but no running processes, leaked CUDA contexts persist:

```bash
sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
```

### 4. Isolate with a minimal config

Create a single-model config on a test port:

```yaml
listen: :9293
models:
  test:
    cmd: /usr/local/bin/llama-server -m /models/downloads/test.gguf --port ${PORT} --host 127.0.0.1 -ngl 99 --no-mmap -c 4096 --ctx-size 4096 --metrics
    ttl: 1800
    concurrency: 1
```

```bash
llama-swap --config /tmp/test.yaml --listen :9293
curl http://127.0.0.1:9293/v1/chat/completions -d '{"model":"test","messages":[{"role":"user","content":"hi"}]}'
```

If minimal works but full config doesn't, compare every `cmd:` field for path/file/flag errors.

## Pitfalls

- **First request to a cold model is slow** (30-90s) — the model must load into VRAM from disk. Subsequent requests are instant.
- **Model IDs MUST match between client and config** — LibreChat sends `model: qwen3.6-35b`, llama-swap's config key must be `qwen3.6-35b`.
- **Port conflicts** — llama-swap requires free ports in the ephemeral range. If another service occupies ports, it will fail to start a model.
- **Duplicated model flags** --copy the full `ExecStart=` flags from your systemd units; don't guess. Pay attention to MoE-only flags (`--n-cpu-moe`) and model-specific context sizes.
- **No auth by default** — llama-swap has no API key enforcement unless configured. Put it behind a firewall or Tailscale, or use `apiKey` in the llama-swap config.
- **TTL vs VRAM pressure:** A 30m TTL means the model stays hot. If you have many large models and limited VRAM, reduce TTL or add an unload script (`curl -X POST http://host:9292/api/models/unload`).

## Frontend Integration

### LibreChat

Replace multiple custom endpoints with one:

```yaml
endpoints:
  custom:
    - name: "Local Models"
      apiKey: "not-needed"
      baseURL: "http://host.docker.internal:9292/v1"
      models:
        default:
          - "qwen3.6-35b"
          - "qwen3.5-9b"
          # ... all models
        fetch: false
      titleConvo: true
      titleModel: "qwen3.6-35b"
      modelDisplayLabel: "Local (llama-swap)"
```

### OpenCode

Replace multiple provider entries with one:

```json
{
  "provider": {
    "local-llamaswap": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local Models (llama-swap)",
      "options": {
        "baseURL": "http://192.168.1.50:9292/v1"
      },
      "models": {
        "qwen3.6-35b": { "name": "Qwen 3.6-35B", "limit": { "context": 128000, "output": 65536 } },
        "qwen3.5-9b": { "name": "Qwen 3.5-9B", "limit": { "context": 128000, "output": 65536 } }
      }
    }
  }
}
```