I finally sat down and set this up last night. The Qwen 3.6 35b is very impressive and following this vid will get you running at workable speeds. I haven’t bumped it up to the 256k context window yet, but even with the 128 I’m able to have it work on projects I’d rather not be putting up on the cloud.
I’ve been rocking Qwen3.6-35B-A3B for months now but from a company macbook. It gets like +70tk/s which is nuts. Definitely going to try to this on my desktop and see how must better it can do.
17tk/s is not good though. iirc my laptops could churn like 11tk/s just running on CPU.
17tk/s is not good though. iirc my laptops could churn like 11tk/s just running on CPU.
It’s relative. I’ve previously tried running a 35b model on my machine in ollama and it crashed more often than not after taking faaaar too long to get going. I’m only getting around 27tk/s at the moment but the context window is huge and I’m not reading much faster than that anyway. Tbh I’m giving it such big jobs that I basically tell it what I want and then just walk away for a while to let it think.
Which model are you getting 27tk/s on? Is that with the steps listed in the video?
I’m getting 10tk/s with just
llama bench -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q8_0and that jumps to 17 tk/s with-ngl 999 -ncmoe 35 -lm none(-lm noneis the updated version of--no-mmap). The turbo cache optimizations seem to come from a fork of llama-cpp with that feature added. Hopefully that gets pulled into the main repo later.Regardless, this was neat but I’m probably not going to stop using the macbook for this. I think the CPU on the macbook is just faster than my 2019 GPU.
Which model are you getting 27tk/s on? Is that with the steps listed in the video?
Yeah, slightly updated from his docker commands in the comments, some of it was deprecated, I guess. 23.4 tk/s now, I did something to piss it off. Lol.
–gpus all
–cap-add=IPC_LOCK
ghcr.io/ggml-org/llama.cpp:server-cuda Qwen3.6-35B-A3B-Q4_K_M.gguf -ngl 999
–n-cpu-moe 36
–load-mode mlock
–ctx-size 128000Note that this on a 1080 (not ti), i7-6xxx, with 24 gigs of ram running PopOS. I’ve not messed with the turbo cache optimization yet.
I’ll need to check what I get with the Q4_K_M variant instead of the Q8_0. Then I’ll get an idea how much CUDA is better than Vulkan.
Word, be curious to hear how that turns out for ya.
I have a eight gigabyte GPU and I wanted to run a thirty billion model or thirty-two or thirty-four or whatever. And being able to run that big models with actually decent speed is super cool.
I found a YouTube link in your post. Here are links to the same video on alternative frontends that protect your privacy:
