GPU Model Lab

What runs on the cheap-VRAM cards, how fast, and how to make it faster. Compiled 2026-08-23. MEASURED = our own benchmark runs COMMUNITY = published community results ESTIMATE = derived, unverified

NVIDIA CMP 170HX — 64GB HBM2e (unlocked), ~$1,450

A100 silicon, 1.5TB/s bandwidth, compute-nerfed, PCIe x4. CUDA — mainline vLLM works. Requires cmpunlocker on pinned driver 610.43.03.

One card (64GB)

Model	Quant / size	Stack	Decode tok/s	Prefill tok/s	Source
GLM-4.5-Air 106B-A12B	AWQ q4 · 59GB	vLLM 0.27	20.3 single · 150 @ 8 users	3,060 (8k in 2.6s)	MEASURED ours, 2026-08-21
Qwen3.6-35B MoE	q4	vLLM	113	1,700	COMMUNITY wiki — the sweet spot
Qwen3.6-27B dense	BF16	vLLM	19.2	2,231	COMMUNITY
Qwen3-32B dense	Q4 GGUF	llama.cpp	33.1	361	COMMUNITY 40GB-unlock card
Mistral-7B	Q4_K_M	llama.cpp	33	400	COMMUNITY stock driver, no unlock
gemma-4-31b (DonorGraph screening)	AWQ q4	vLLM, full pipeline	110–147 screens/hr, flat vs concurrency		MEASURED ours, 2026-08-08

What fits on one 64GB card (q4): GLM-4.5-Air (barely — 8k ctx max), Qwen3-32B / Qwen3.8-27B / gemma-4-31b with big KV headroom, any ≤70B dense, 30–35B MoE with room to spare. Does NOT fit: DeepSeek-V4-Flash (101GB), GLM-4.7 (192GB).

Four cards (256GB, pipeline-parallel)

Model	Quant / size	Stack	Decode tok/s	Prefill tok/s	Source
DeepSeek-V4-Flash 284B-A13B	q4 · 101GB	vLLM PP=4	98	5,300	COMMUNITY
GLM-4.7 358B-A32B	AWQ q4 · 192GB	vLLM PP=4	~25–40	~2,000–3,000	ESTIMATE fits; no published run yet
Qwen3-235B-A22B	q4 · ~125GB	vLLM PP=4	~40–70	~3,000–5,000	ESTIMATE
GLM-4.5-Air	AWQ q4	vLLM PP=2–4	~20–25/stream, long ctx + many users	~3,000+	ESTIMATE from 1-card measurement
Reference: GLM-5.2 744B on EIGHT cards	q4 · ~376GB	vLLM PP=8	30.2 @ 131k ctx	2,675	COMMUNITY zero hard faults

⚠️ Multi-card caveats: use the official cmpunlocker pinned driver (610.43.03) on owned hardware — Vast's rental fleet runs an unofficial driver fork that bricked GPU#2 on 2 of 2 rigs we tested during concurrent weight loading. Big PP configs scale poorly with users (community: 16 users → only 2.25× aggregate). Model loads over x4 links are slow (GLM-5.2 class: 7–10 min).

Speed modifications that work (and ones that don't)

✅ cmpunlocker — 8GB→64GB VRAM + PCIe Gen1→Gen2. THE unlock. 8GB SKU only (10GB caps at 40GB; 80GB mod tested-and-rejected as unstable).
✅ Pipeline parallel, never tensor parallel — TP measured 2.3–2.8× SLOWER prefill for +23% decode. PP is mandatory at x4 links.
✅ Power-limit to 200W (nvidia-smi -pl 200) — decode is bandwidth-bound; ~free power/heat savings.
✅ llama.cpp: compile with --fmad=false — the mining nerf throttles FMA but not FMUL/FADD; this flag routes around it.
✅ Batching — small-MoE aggregate scales near-linearly (measured: 1→8 users = 20→150 tok/s). Fan one big job out into parallel section-writers ("skeleton-of-thought") for ~3× effective speed on multi-file builds.
✅ PCIe x16 capacitor solder mod — fixes slow model loads only (24 caps, hand soldering; some eBay cards ship pre-modded ~+$400). No inference speedup.
⚠️ MTP / speculative decode — llama.cpp +21%, but vLLM −23% on these cards. A/B per stack; expect DSpark-style tricks to underwhelm (no spare compute to spend).
❌ int8 KV cache — measured SLOWER (147→108 screens/hr). KV bandwidth was never the bottleneck. fp8 KV impossible (compute cap 8.0).
❌ CUDA graphs at TP8 — crashes (vLLM #48285); eager-mode workaround costs ~4× prefill.

AMD Instinct MI50 — 32GB HBM2, ~$650

Vega 20, 1.0TB/s, real PCIe 4.0 x16, full-strength compute for its age. gfx906 dropped from official ROCm — runs on community forks (vllm-gfx906, gfx906 llama.cpp builds). Passive card, needs shroud.

One card (32GB) — community-measured (kyuz0 toolboxes)

Model	Quant	Stack	Decode / throughput tok/s	Prefill tok/s	Source
GPT-OSS-20B	MXFP4	llama.cpp	~123	1,141	COMMUNITY
Nemotron-3-Nano-30B MoE	Q4_K_XL	llama.cpp	fast (MoE)	737	COMMUNITY
Qwen3.6-35B MoE	Q4_K_XL	llama.cpp	—	816	COMMUNITY
gemma-4-26B	Q4_K_XL	llama.cpp	—	916	COMMUNITY
Qwen3.6-27B dense	Q4_K_XL	llama.cpp	~20 class	211	COMMUNITY
Llama-3.1-8B	AWQ	vllm-gfx906	451 (batch)	—	COMMUNITY
Qwen3.6-27B	AWQ-INT4	vllm-gfx906	154 (batch)	—	COMMUNITY
gemma-4-31B	AWQ-4bit	vllm-gfx906	66 (batch)	—	COMMUNITY

Single-stream text generation ranges ~20 tok/s (27B dense) to ~123 tok/s (20B MoE). "Batch" = aggregate throughput under load via the vllm-gfx906 fork.

Four cards (128GB)

Model	Fits?	Expected	Source
GLM-4.5-Air (59GB q4)	✅ comfortably	~20–30/stream — people run GLM-4.6-class MoE on these rigs (dedicated Docker builds exist)	ESTIMATE runs confirmed, tok/s unpublished
Llama-70B / Qwen-72B q4	✅	~10–15/stream (dense = slow)	ESTIMATE
DeepSeek-V4-Flash (101GB)	⚠️ barely, no KV room	impractical	ESTIMATE
GLM-4.7 (192GB)	❌ does not fit	—	—

Speed modifications

✅ gfx906-optimized llama.cpp fork — up to 2× prompt processing; Vega-specific tiles, custom q8 KV kernels.
✅ vllm-gfx906 fork (nlzy + spin-offs) — real continuous batching on a card AMD abandoned; AWQ/GPTQ quants. ~vLLM 0.11-era features.
✅ Real x16 links make multi-card splits far less painful than the 170HX's x4.
⚠️ Everything lives on community forks a year+ behind mainline. No fp8, no modern kernels, and the gap widens every month.

AMD Radeon Pro V620 — 32GB GDDR6, ~$450

RDNA2 cloud-gaming card (Azure fleet castoff). 512GB/s, NO matrix/tensor units, headless SR-IOV firmware, gfx1030 second-class in ROCm.

Model	Quant	Stack	Decode tok/s	Prefill tok/s	Source
Llama-3.2-1B	Q4_K_M	llama.cpp	102	4,608	COMMUNITY LocalScore — the ONLY published result
7–13B class	q4	llama.cpp Vulkan	~25–45	weak (no matrix cores)	ESTIMATE
30B+ class	q4	—	fits in VRAM, painful in practice		ESTIMATE

Verdict: one published benchmark on a 1B model is the whole public record — that absence is the data. No inference community, no fork ecosystem, no matrix hardware for prefill, cloud-card driver quirks. The cheapest 32GB you can buy and the most expensive tokens you'll ever get. AVOID

Bottom line

	170HX ($1,450)	MI50 ($650)	V620 ($450)
Usable VRAM	64GB	32GB	32GB
Best single-card result	113 tok/s (Qwen3.6-35B MoE)	123 tok/s (GPT-OSS-20B)	102 tok/s (1B model…)
Software	mainline CUDA vLLM (pinned unlock driver)	community forks, ~1yr behind	effectively none
4-card headline	DeepSeek-V4-Flash @ 98 tok/s	GLM-4.5-Air (unbenchmarked)	—
Scales to	8 cards → GLM-5.2 744B @ 30 tok/s	128GB ceiling for sanity	—
Verdict	the capacity play — chosen	honorable budget rig	avoid

Sources: our benchmark runs (2026-08-08 / 08-21), the cmp170hx community wiki (Operations-LLM-Inference, Procedures), kyuz0/mi50-gfx906-toolboxes, Level1Techs gfx906 fork thread, kostakoff (HF), LocalScore. Estimates are ours and labeled.