# Frontend Integration with llama.cpp Servers

When llama.cpp `llama-server` processes are running on a machine, various frontends can consume them as OpenAI-compatible endpoints. This reference covers the common patterns.

> **Prefer llama-swap for multi-model setups.**  
> If you have 2+ models and want seamless auto-switching from the frontend UI (just pick a model, it works),
> use [llama-swap](references/llama-swap.md) instead of managing per-port systemd services.
> The multi-endpoint pattern below still works but requires manual model switching outside the frontend.

## Port Architecture

Each `llama-server` process serves exactly **one model on one port**. A fleet of models means a fleet of ports:

| Port | Service | Model |
|------|---------|-------|
| 8081 | llama-server | Daily driver (Qwen 35B etc.) |
| 8082 | llama-server-9b | Smaller uncensored model |
| 8090 | llama-server-gemma4 | Gemma 4 |
| 9021-9025 | llama-server-{coder,mtp,...} | Specialized models |

## LibreChat Configuration

LibreChat uses `librechat.yaml` (typically at the project root, e.g. `~/projects/LibreChat/librechat.yaml`). 

### With llama-swap (Recommended)

Single endpoint with all models listed — swapping happens automatically:

```yaml
endpoints:
  custom:
    - name: "Local Models"
      apiKey: "not-needed"
      baseURL: "http://host.docker.internal:9292/v1"
      models:
        default:
          - "qwen3.6-35b"
          - "qwen3.5-9b"
          - "qwen3-coder-30b"
          # ... all models
        fetch: false
      titleConvo: true
      titleModel: "qwen3.6-35b"
      modelDisplayLabel: "Local (llama-swap)"
```

### Without llama-swap (Legacy multi-endpoint)

Each llama-server port gets its own **custom endpoint** entry.

### Docker → Host Networking

When LibreChat runs in Docker and llama-server runs on the host, use `host.docker.internal` as the hostname. This requires a `--add-host` flag in the Docker compose or run command:

```yaml
# In docker-compose.yml for the LibreChat service:
extra_hosts:
  - "host.docker.internal:host-gateway"
```

### Multi-Model YAML Pattern

```yaml
endpoints:
  custom:
    # ── One entry per port ─────────────────────────
    - name: "Model Name (visible in UI)"
      apiKey: "not-needed"        # llama.cpp doesn't require auth
      baseURL: "http://host.docker.internal:8081/v1"
      models:
        default:
          - "short-model-id"      # Friendly slug for LibreChat
        fetch: false              # Don't query /v1/models
      titleConvo: true
      titleModel: "short-model-id"
      modelDisplayLabel: "Friendly Display Label"

    - name: "Second Model"
      apiKey: "not-needed"
      baseURL: "http://host.docker.internal:8082/v1"
      models:
        default:
          - "second-model-id"
        fetch: false
      titleConvo: true
      titleModel: "second-model-id"
      modelDisplayLabel: "Second Model Display"
```

Key points:
- **`apiKey: "not-needed"`** — llama.cpp accepts any API key (or none)
- **`fetch: false`** — LibreChat won't try to list models from the server
- **One entry per port** — each llama-server is a separate endpoint
- **Model ID in requests** — llama.cpp accepts *any* model name in the request body, so the slug doesn't need to match the GGUF filename

### Restart After Config Change

```bash
cd ~/projects/LibreChat
docker compose restart
# Check logs for endpoint loading:
docker logs LibreChat 2>&1 | grep -E 'name|modelDisplayLabel'
```

### Verifying Reachability

From inside the LibreChat container:

```bash
docker exec LibreChat curl -s http://host.docker.internal:8081/v1/models
```

If `curl` isn't inside the container, test from the host:

```bash
curl -s http://localhost:8081/v1/models
```

## OpenCode Configuration

OpenCode reads `~/.config/opencode/opencode.json` (global) or `./opencode.json` (per-project).

### With llama-swap (Recommended)

Single provider, all models under it:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-llamaswap": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local Models (llama-swap)",
      "options": {
        "baseURL": "http://192.168.1.50:9292/v1"
      },
      "models": {
        "qwen3.6-35b": { "name": "Qwen 3.6-35B", "limit": { "context": 128000, "output": 65536 } },
        "qwen3.5-9b": { "name": "Qwen 3.5-9B", "limit": { "context": 128000, "output": 65536 } }
      }
    }
  }
}
```

### Without llama-swap (Legacy multi-provider)

Each llama-server port gets its own provider entry.

See the `opencode` skill's `templates/local-models.json` for a full multi-model template.

### Structure

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-model-a": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Model A (local)",
      "options": {
        "baseURL": "http://192.168.1.50:8081/v1"
      },
      "models": {
        "model-a.gguf": {
          "name": "Model A Display",
          "limit": { "context": 128000, "output": 65536 }
        }
      }
    }
  }
}
```

### Remote llama.cpp servers

When OpenCode runs on a **different machine** than the llama.cpp servers:

- **Use Tailscale IP** (`http://100.x.x.x:PORT/v1`) if both machines are on Tailscale — no firewall config needed
- **Use LAN IP** (`http://192.168.x.x:PORT/v1`) for same-subnet access — needs UFW rules (see below)

### Firewall Rules for Remote Access

On the server hosting llama.cpp, open each port to the client subnet:

```bash
# Restrict to a specific client IP
for port in 8081 8082 8090 9021 9022 9023 9024 9025; do
  sudo ufw allow from 192.168.1.111 to any port $port proto tcp
done

# Or open to whole subnet
for port in ...; do
  sudo ufw allow from 192.168.1.0/24 to any port $port proto tcp
done
```

**UFW does not affect Tailscale traffic** — Tailscale has its own `ts-input` iptables chain that accepts everything from `tailscale0`.

### Model ID Discovery

The model ID in OpenCode config must match what the server's `/v1/models` endpoint returns. For llama.cpp, this is the raw GGUF filename:

```bash
curl http://host:port/v1/models | jq '.data[].id'
# Example output: "Qwen3.6-35B-A3B-UD-Q4_K_S.gguf"
```

However, **llama.cpp accepts any model ID in requests** — so OpenCode models config entries can use friendlier names as long as the config ID matches what the server returns. If using `fetch: false` in OpenCode, the config must still contain the exact model ID from the server.

## Verifying End-to-End

1. Start the llama-server: `sudo systemctl start llama-server`
2. Wait for health: `curl http://localhost:8081/health` → `{"status":"ok"}`
3. Check model ID: `curl http://localhost:8081/v1/models`
4. Test completion: `curl http://localhost:8081/v1/chat/completions -d '{"model":"<model-id>","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'`