FLE Link

FLE Link / Models

Run Katy 27B locally on a 12 to 32 GB GPU

Katy is a quantization of Swift-1.5-Qwen3.8-27B, a dense 27B model, as IQ4_XS with the DFlash2 draft model. From 20 GB of GPU memory all 66 layers are on the card; below that the rest runs on the CPU, which is slower.

flelink run katy-27b-dflash

What you need

One NVIDIA card from 12 GB and 24 GB of RAM. A 12 GB card is the minimum the recipe was measured with.

Measured speed

RecipeDecodeMeasured on
Katy 27B + DFlash2dense 27B with a draft model104 tok/sfrom 20 GB up all 66 layers are on the card; one run, engine direct, context 32768, in a machine with 128 GB RAM
Katy 27B + DFlash2dense 27B with a draft model19.4 tok/s16 GB card, 51 of 66 layers on it; one run, Ryzen 9 9950X3D with 128 GB RAM; a slower CPU will be slower
Katy 27B + DFlash2dense 27B with a draft model11.2 tok/s12 GB, 33 of 66 layers; one run, memory limited on a larger card to simulate the size

Does Katy 27B + DFlash2 fit my card?

Graphics32 GB RAM64 GB RAM128 GB RAM
12 GB NVIDIA cardfitsfitsfits
16 GB NVIDIA cardfitsfitsfits
24 GB NVIDIA cardfitsfitsfits
32 GB NVIDIA cardfitsfitsfits
Apple Silicon Macno engineno engineno engine
No GPUno engineno engineno engine

"no engine": this recipe needs the FLE engine, which has an NVIDIA CUDA build only. File size 16 GiB including the draft model.

The verdicts are the rule FLE Link applies before it downloads anything (the rules in full). A speed is shown only for the hardware it was measured on. A size nobody measured yet? flelink bench --task pagoda --share posts yours to the benchmarks board.

Good to know

Sources

Install FLE Link

curl -fsSL https://flelink.run/install.sh | sh

Windows: irm https://flelink.run/install.ps1 | iex. Open source, MIT. No account, no telemetry. What FLE Link does →