Run Katy 27B locally on a 12 to 32 GB GPU
Katy is a quantization of Swift-1.5-Qwen3.8-27B, a dense 27B model, as IQ4_XS with the DFlash2 draft model. From 20 GB of GPU memory all 66 layers are on the card; below that the rest runs on the CPU, which is slower.
flelink run katy-27b-dflashWhat you need
One NVIDIA card from 12 GB and 24 GB of RAM. A 12 GB card is the minimum the recipe was measured with.
Measured speed
| Recipe | Decode | Measured on |
|---|---|---|
| Katy 27B + DFlash2dense 27B with a draft model | 104 tok/s | from 20 GB up all 66 layers are on the card; one run, engine direct, context 32768, in a machine with 128 GB RAM |
| Katy 27B + DFlash2dense 27B with a draft model | 19.4 tok/s | 16 GB card, 51 of 66 layers on it; one run, Ryzen 9 9950X3D with 128 GB RAM; a slower CPU will be slower |
| Katy 27B + DFlash2dense 27B with a draft model | 11.2 tok/s | 12 GB, 33 of 66 layers; one run, memory limited on a larger card to simulate the size |
Does Katy 27B + DFlash2 fit my card?
| Graphics | 32 GB RAM | 64 GB RAM | 128 GB RAM |
|---|---|---|---|
| 12 GB NVIDIA card | fits | fits | fits |
| 16 GB NVIDIA card | fits | fits | fits |
| 24 GB NVIDIA card | fits | fits | fits |
| 32 GB NVIDIA card | fits | fits | fits |
| Apple Silicon Mac | no engine | no engine | no engine |
| No GPU | no engine | no engine | no engine |
"no engine": this recipe needs the FLE engine, which has an NVIDIA CUDA build only. File size 16 GiB including the draft model.
The verdicts are the rule FLE Link applies before it downloads anything (the rules in full). A speed is shown only for the hardware it was measured on. A size nobody measured yet? flelink bench --task pagoda --share posts yours to the benchmarks board.
Good to know
- Below 20 GB the speed depends on the CPU and the RAM: the numbers were measured on a Ryzen 9 9950X3D with 128 GB RAM, and a slower machine will be slower.
- Licence of the model: Swift Open License v1.0, see the model page.
Sources
Install FLE Link
curl -fsSL https://flelink.run/install.sh | shWindows: irm https://flelink.run/install.ps1 | iex. Open source, MIT. No account, no telemetry. What FLE Link does →