![Thumbnail (1920x1080)](https://i.ytimg.com/vi/VytSYCDhWQ0/maxresdefault.jpg)
# [I Asked Claude Fable 5 to Improve llama.cpp.. and It Did](https://www.youtube.com/watch?v=VytSYCDhWQ0)

**Visibility**: Public
**Uploaded by**: [Codacus](https://www.youtube.com/@Codacus)
**Uploaded at**: 2026-07-05
**Published at**: 
**Length**: 16:41
**Views**: 911
**Likes**: 142
**Category**: Science & Technology

## Description

```
I pointed Fable Anthropic's new model at llama.cpp, the engine that runs local LLMs on consumer hardware, and asked one thing: make it faster. It made prefill ~64% faster on the exact same GPU. No new hardware, same model, token-identical output.

Running a 35B mixture-of-experts model (Qwen3.6 35B-A3B) on a 12GB RTX 3060 means the experts live in system RAM and stream across PCIe every prefill pass — and that bus, not the GPU, is the real bottleneck. Fable found it, wrote the patches, benchmarked every change itself, and even caught a bug in its own code.

Four optimizations. Two worked, two didn't — and I show you exactly why, including the one that ran 14× slower and lost to my hardware, not the AI. The 64.5% is the honest CODE gain (1143 → 1880 tokens/sec), kept separate from config tuning. All the patches are on my public GitHub fork so you can test them on your own rig.

This is what taking back control of local AI actually looks like: the tools we build AI with, starting to improve themselves — on a desk, not in a frontier lab.

⏱️ CHAPTERS
0:00 I Used Fable to Improve llama.cpp
0:52 The Free Hand
2:50 Where the Math Happens
5:54 The Dead Switch (Win #1, +21%)
8:18 The Overlap (the headline — 41.7% → 2.8% idle)
11:10 The Second Front (two honest failures)
13:55 The Verdict
15:04 The Flywheel

🔗 GitHub fork (all the patches): github.com/thecodacus/llama.cpp
🛠️ Rig: RTX 3060 12GB · Ryzen 5 5600X · 32GB DDR4 · Ubuntu 24.04 · Qwen3.6 35B-A3B, --n-cpu-moe 26

💬 What would YOU point Fable at? Drop it in the comments. Local AI you actually own — subscribe.

#llamacpp  #claudefable5  #fable  #localai  #localllm #ai #llm
```

## Transcript

I used Fable to improve llama.cpp.
And it did. When Fable came out, people
built a lot of eye-catching stuff, cool
marvels, but almost nobody tried the one
thing Fable is actually best at,
improving the technology we already
have.
>> [snorts]
>> Taking what already exists and making it
better, more robust. So, I thought, why
not point Fable at local AI itself?
I let Fable loose on the llama.cpp code
base, freehand, do any analysis, find
any optimization that makes local AI
faster.
And it found real ones. It made
llama.cpp almost 65% faster, four
optimizations in total. Some worked,
some didn't. We are going to dissect all
four, what Fable tried, what worked, and
what it actually did.
Now that Fable is safe to use, I
thought, instead of making flashy
experiments and flashy demos, why not
use Fable to improve the existing tools
we already have for local AI?
And the most popular and crucial one,
the tool that runs on most
consumer-grade hardware, is none other
than llama.cpp.
And the changes here compound. If I can
get even a 1% improvement, it reflects
on every model llama.cpp can run. And
all the future optimizations other
people ship will stack right on top of
it.
But there's a catch.
The llama.cpp repo doesn't accept pull
requests that are fully or predominantly
AI generated. Their words, "Fully AI
generated PRs provide no value.
Maintainers have AI tools, too."
And it's a fair call. Someone has to
understand and maintain every line.
But it does set the bar high. One of the
most tuned pieces of software on Earth
is basically saying an AI has nothing to
add here.
It's a decision I respect. And they say
it right there, private forks are
exempt. So, that's what I did. I forked
it to see if that's actually true.
If any change Fable makes can really
improve the engine.
I didn't want to give Fable any
direction, any instruction. I didn't
want my own biases influencing its
decisions or its thought process.
So, I just asked it to pull the
llama.cpp repo, read everything
thoroughly, and find the places where we
could improve or optimize for mid-range
local LLMs.
For the setup, I'm running a 12 GB 3060
and 32 GB of DDR4.
And yeah, that's the big brother of my
old 6 GB 1060 setup.
So, I let Fable loose on the llama.cpp
codebase. And here's the interesting
part. It asked if it could SSH into my
rig.
I gave it permission, and it SSH'd into
my AI rig, found all the model folders,
the Turbo Quant setup I was running, and
my llama.cpp preset config.
It saw I leaned heavily on an MoE model.
So, it went looking for a way to
optimize the MoE side of the engine.
The model's a 35 billion parameter MoE,
only about 3 billion active per token.
It doesn't fit in 12 GB. So, the experts
for most of the layers live in system
RAM. The exact big model small GPU setup
a lot of you run.
And one thing about how it worked start
to finish.
For every change, it benchmarked before
and after, compared the output with and
without, and confirmed it worked with no
regression. All on its own.
Now, while it was exploring and giving
me suggestions, some of its explanations
gave me doubts. So, I asked a bunch of
clarifying questions. And I found out I
had something wrong.
I used to think that when llama.cpp
offloads the MoE layers, the experts
sitting in system RAM get executed by
the CPU and the ones in VRAM get
executed by the GPU. So, you never move
weights back and forth across PCIe.
But surprisingly, that's not the case.
That's only half the story. It's true in
decode when you're generating one token
at a time, that's exactly how llama.cpp
does it. The CPU runs the experts right
there in RAM.
But prefill is different. In prefill,
you're processing thousands of tokens at
once.
Every token pulls eight experts from a
layer. Thousands of tokens, only 128 or
256 experts. So, almost every expert
gets selected.
The CPU would have to run the entire
layer, every single expert, which is
brutally compute heavy for a CPU.
So, what llama.cpp does is past a
certain batch threshold, instead of
running it on the CPU from RAM, it pulls
those weights back into VRAM and lets
the GPU execute the experts in parallel.
And that's where the bottleneck is.
The weird part, the GPU runs the whole
attention and the router first. Then the
system takes the expert IDs it needs to
pull from RAM to VRAM. And while that
copy happens, the GPU just sits idle.
Everything streams from RAM to VRAM over
PCIe and only then does the GPU execute
the experts.
When you actually profile a prefill
pass, the GPU spends more time receiving
weights than computing with them. About
4.4 seconds copying versus
seconds of real math.
The villain isn't the GPU, it's the bus,
it's PCIe.
Before we go any further, let me set the
baseline. On that 35B MoE at UBatch
2048, I'm getting about 1143 tokens per
second on prefill prompt processing.
That's the starting line for everything
Fable does next.
So, after Fable explored the code base,
it found a dead section of code, a piece
that lets you pin the paged memory. And
nothing was calling it. Someone had
already written the feature. There's an
environment variable to turn it on and
off, but the place where it's supposed
to be called just doesn't call it. Maybe
that was intentional, but if it was, I
think they should have removed the dead
code. To explain what that code does, I
need to give you some background. In a
lot of my previous videos, I've
mentioned the no mmap flag. Normally,
llama.cpp uses mmap, paged memory that
the OS can evict. No mmap instead loads
the model into a private memory section
that won't get evicted. But there's a
drawback to that. With mmap, model
loading is basically instant. It feels
like zero time, almost negligible to
load the model. With no mmap, it has to
pull all those weights into that private
section. So, you wait a lot longer for a
model load or a model swap. Now, the
dead code Fable found lets you pin the
existing mmapped paged memory. So, you
get the same transfer performance as no
mmap, but you keep the perks of mmap
like instant model loading. And it only
pins the expert weights, not the entire
model, so it doesn't take that much
extra space. Our baseline was 1143
tokens per second on prefill. With this
change, we are getting about 1385,
around 21% faster. Now, you'll get
roughly the same number if you just use
no mmap. So, I won't pretend it's a huge
gain. The point is we got there without
no mmap's drawback. You keep the instant
model loading and swapping. Real quick,
the next optimization is the one that
actually gets us to the headline number.
And it's where Fable did something I
didn't think it could do. If you want
local AI to keep getting faster,
subscribing helps. All right, back to
it.
So, in prefill, remember, llama.cpp
pulls the experts back to the GPU and
lets the GPU do the compute. Because
with hundreds of tokens at once, almost
every expert gets selected and it's way
too much work for the CPU.
But, here's the plumbing problem. First,
the GPU computes the attention. Then, it
runs the router, which gives you the IDs
of the experts that need to execute.
Then, the scheduler schedules the job to
pull those experts from system RAM back
to VRAM. And while that copy is
happening, the GPU just sits idle
waiting for the experts to arrive before
it can compute anything. And this
happens around 40 times per batch. And
Fable profiled all of this. What you're
seeing on screen is the exact dump from
that profiling. This is not an estimated
diagram, it's the real data.
The first stretch is the attention
computing. Then, it drains the weights
from the previous layer, gets the new
expert IDs, and pulls those experts from
system RAM to the GPU. And during that
copy, the compute is idle.
Only once the copy is done, does the GPU
calculate.
Add it all up and the GPU is sitting
idle about 41.7%
of the time. Effort and time we could
have spent computing the next batch.
The fix is actually simple and it's a
pattern used all over the place. Two
lanes running in parallel. One lane
computes the attention, the other pulls
the expert layers from system RAM up to
the GPU.
But here's the catch. Until the router
runs, we don't know which experts to
pull.
Except at prefill, when you're
processing 500 to 2,000 tokens in one
batch, each pulling eight experts, it
turns out almost all of the 128 or 256
experts get selected anyway.
So, why wait for the router to hand us
the IDs? While the attention is still
computing, we just pull the whole layer
back to VRAM. So, by the time the router
is done, the weights are already sitting
on the GPU ready. No waiting for the
copy, it all happens in parallel. And
that's what you're seeing below. Again,
the actual profiling of the patched
code, real data across those 35 to 45
milliseconds. The compute isn't idle
anymore. The GPU is always fed, always
working. And that takes the GPU idle
time from 41.7%
down to 2.8%.
The best part? We lose zero quality. The
output is exactly what it would be
without the optimization. This is purely
a fix on the pipeline.
And it did one more thing that genuinely
impressed me. It found the bug in its
own patch and fixed it before I even saw
it.
So far, everything Fable did sped up
prefill, prompt processing. But that's
only half the story. The other half is
decode, actually generating the tokens.
And this is the harder half. If you
watched my Dflash video, this is that
same territory. When Fable profiled
decode, it killed one of my assumptions.
I always thought decode on an offloaded
MoE was CPU bound. The experts run on
the CPU, so the CPU has to be the
bottleneck. It's not. It's a near 50/50
split, the GPU and the CPU trading off
back and forth the whole time. And
there's a trap hiding in here.
Speculative decoding is where you draft
a few tokens ahead and verify them all
in one pass. And normally, longer drafts
mean more speed up. But on an offloaded
MoE, that flips. The cost of verifying
isn't the experts for one token, it's
the union of the experts for every token
you drafted. Draft more tokens and that
combined set balloons toward all the
experts, and the CPU has to read every
one of them. So past a point, longer
drafts actually get slower. Drafting
eight tokens ahead was slower than not
speculating at all. And that's the thing
that finally explains the Dflash
collapse on my old 1060 that I could
never figure out. So Fable had an idea
for this. Build an adaptive controller,
something that watches the workload and
picks the right draft length on the fly,
instead of a fixed number. That's the
clever move. It should win. Fable built
it, then rebuilt it five times over.
Each version it measured, found exactly
why it was losing, fixed that, measured
again. And every single time, it lost to
a dumb fixed cap. The decision space was
just too flat. The learning cost more
than it saved. This is the rare honest
one. The smart thing lost to the dumb
thing. But it wasn't a total loss. It
handed me a real tip. On an offloaded
MoE, just cap your speculative draft at
two or three tokens, and you're about
23% faster with none of the slowdown.
Then came Fable's boldest idea. What if
the CPU and the GPU worked on the same
layer at the same time? Split the expert
work between them. On paper, that's free
performance. In practice, it ran 14
times slower. And here's the part I want
to be straight about. The mechanism
actually worked. Every piece of it
measured correctly. The reason it lost
wasn't Fable. It was my hardware. PCIe
Gen 4 is just too slow next to reading
straight from system RAM. So, shipping
that work across the bus costs more than
it saves. On a machine with PCIe 5 or
NVLink, it would pay off. The wall was
physics.
So, four optimizations. Did they work?
Two of them did. Pinning the memory and
overlapping the uploads. Together,
that's the 64.5% on prefill, same GPU,
same settings. The other two, the
adaptive controller and the CPU GPU
split, didn't. They lost and I showed
you exactly why. And I want to be
precise about that number because it's
easy to inflate. The 64.5%
is the code gain. What Fable's patches
did. 1143 to 1880. Nothing else touched.
That's separate from any config tuning.
And like I said at the start, every
single change Fable benchmarked before
and after and diffed the output itself.
The result is token identical to
mainline. Same answers, just faster. And
one more thing, even on the ideas that
failed, Fable surfaced a couple of
genuine latent bugs in mainline
llama.cpp. The kind that have just been
sitting there that nobody has hit yet.
And I've got all the changes up on my
GitHub fork. So, if you want to test any
of this yourself, it's all there.
So, step back for a second. An AI read a
code base, profiled it, found real
bottlenecks, wrote the patches,
benchmarked them, and even caught a bug
in its own code. And it made one of the
most tuned pieces of software on Earth
measurably faster on a desk, not in a
frontier lab. That's the loop. The thing
we build AI with can now improve the
thing we build AI with. And here's the
part I didn't expect. The loop didn't
just make the code faster, it made me
better. That whole picture of where the
expert math actually runs, CPU for one
token and uploaded to the GPU for a big
batch, I didn't know that until Fable's
profiling walked me through it. So, the
loop improves the code and it improves
the person running it. Now, I'll be
honest about one thing. Fable isn't a
local model. This was a cloud model
improving your local stack. But, think
about where this is heading. The moment
a model you can run locally on llama.cpp
gets good enough to improve llama.cpp
itself, that's when the loop closes all
the way. A model on your own hardware
making the engine it runs on faster,
which makes that same model run better.
No cloud, no frontier lab, just you and
your machine improving each other.
That's the version I actually want to
see and that's when we'll truly bring
the power back to us. So, that's the
experiment. I'd genuinely love to know
what you'd point Fable at, drop it in
the comments. And if you want to see
where this goes next, you know what to
do. I'll see you in the next one.