Not my settings! All credits to “Stainless-Bacon”. I did however replicate the setup just now and I think it’s worthy of spreading.

The trick is in a targeted offloading of only a specific kind of layers to CPU making it possible to run a higher quant that otherwise would be too slow to be usable.

Remember to compile llama.cpp with GGML_CUDA_FA_ALL_QUANTS=ON

Original settings below. Since I’m using the iGPU for the system I’m slightly tweaking them to use even more VRAM. On my 5060Ti I’m getting prefill 500-600tps (ub at 2048) and tg at ~10tps.

export GGML_CUDA_DISABLE_GRAPHS=1
llama-server \
  --model Qwen3.6-27B-Q4_K_M_MTP.gguf \
  --chat-template-file froggeric_fix.jinja \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --fit off \
  --n-gpu-layers 99 \
  --override-tensor 'blk\.(2[0-9]|3[0-9]|4[0-3])\.ffn_.*=CPU' \
  --ctx-size 96000 \
  --batch-size 512 \
  --ubatch-size 512 \
  --cache-type-k q5_0 \
  --cache-type-v q4_1 \
  --parallel 1 \
  --temp 0.60 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --flash-attn on \
  --no-mmap \
  --host 0.0.0.0