(Oops I got my Gemma and Qwen speeds mixed up, edited the post to fix it.)

But now with the new commits they added, with the same number of hot experts, Qwen is up to about 34. If I increase hot experts to 48 then I get around 37.

Gemma is still around 23 with just 10 hot experts. With 16 hot experts I get about 26 TPS. If I overprovision my VRAM (thanks to GGML_CUDA_ENABLE_UNIFIED_MEMORY=1) then 24 hot experts can give me 29 TPS, and 32 hot experts 34 TPS.

In a few minutes a significant performance improvement incoming

👀

this has been a crazy few weeks! lol

true, it’s not perfectly clear

also I just saw this

have you tried Qwen 3.6 35b a3b? check my guide, it’s still relevant to you just with different numbers because you have 12GB

https://lemmus.org/post/24235317

Gemma is probably good for that, as long as it’s consistently succeeding at the tool calls.

make sure that holds up with large context, you might need to step down to Q3 (which I’ve heard is still good for this model, many people are even using IQ2)

You’re looking for “2160p remux” torrents

this appears to be the untouched model in GGUF format, 180 GB, MXFP4

https://huggingface.co/bartowski/DeepSeek-V4-Flash-0731-GGUF

the original model has a lot of parts that were natively trained in 4 bit, so those layers can’t go higher

--n-cpu-moe 36 --spec-type draft-mtp --spec-draft-n-max 3 does seem to speed up token generation for me

Can’t use llama-bench for MTP. In a basic tests it seems to improve from about 26 to 30 tokens per second output. But it seems to hurt my input speed from about 1300 pp down to 800.