Surprised to hear about the big difference between Q8 and Q6. Some GGUF makers (like bartowski) even recommend Q6 (but I also use Unsloth quants, they even recently released 3.1).

Fair!

I tried to hit play on your screenshot 😅

For a 16Gb card I use UD-IQ4_XS with 94208 of context and Q4 for KV cache. But I don’t have desktop on this GPU, so you’ll need to either reduce the context size or use a 3 bit quant.

It’s is a dense model, so RAM is irrelevant. It will be too slow if the model leaks into RAM. You’ll need to put all layers on the GPU.

But what is your GPU? It’s a dense model, running it from RAM will be extremely slow.

If you want to utilize your RAM, you need a MoE model.

What are your specs?

Same thing, I bought mine last month!

I offload mmproj to the CPU to keep as much model weights in VRAM as possible 🙂

27B and Qwen flash next can run with somewhat realistic specs. I run 27B on my 7900 XTX.

That would be such a good title! Why didn’t I think of that? 😅

The woman’s modelling career.

WDYM? The joke about Chinese models (like Qwen, DeepSeek, etc.) won’t work without this.

That’s insane. I bought mine last month for ~900$.

Oh yeah! But I think something like Qwen 3.8 27B should be able to handle it 🤔

We’re in !ocalllama@sh.itjust.works. So if this goes to the service you own (runs on your PC, for example) - I think it’s fine. The author confirmed the local models are supported.

If the model is local and since the APP is FOSS, it’s probably fine.

deleted by creator

I didn’t know it has ads. I used it to avoid sending a direct link to Twitter.

I assume it’s to simplify the review process. Or to generate more hype. Or both.

They announced that it would be released next week, so I doubt they’ve only just started training it.

There will be Flash, 27B and 2 other models called Plus and Max.