Local LLM Serving and GPU Speed Measurements
Measured tokens/s for Qwen 27B and 35B models on RTX 5090, R9700 and 7900 XTX with vLLM and llama.cpp: concurrency, MTP, quantization, and heterogeneous-GPU serving.
Measurements from running the AI analysis pipeline of a Korean-stock trading bot on local GPUs. I ran a 27B dense model with vLLM on a single RTX 5090 and a 35B MoE model with llama.cpp on an AMD R9700, and compared tokens/s, concurrent slots, MTP (speculative decoding) and quantization.
The same card can give opposite conclusions depending on the model. On the R9700, Vulkan beat HIP, four concurrent slots was the knee, and MTP only helped a single request. On the 5090 the KV cache was the bottleneck. Read the posts below in order: purchase decisions, speed comparison, tuning, then the heterogeneous-GPU serving postmortem.
- Local LLM Speed by GPU Generation, and Our Production's Daily Token VolumeModel names, serving configs, and full prefill/decode measurements included (reorganized as of 2026-09-26)
- Qwen3.8-27B on a Single RTX 5090 with vLLM: NVFP4 + MTP Tuning Results and the KV Cache BottleneckMeasured tokens/s while going from 3 to 14 concurrent slots to MTP + 8 slots: how I cut a 100-stock nightly batch from six-plus hours to under three, and why you shouldn't fully trust that number
- Qwen3.6-35B-A3B on an AMD R9700 with llama.cpp: Vulkan vs HIP, Concurrent Slots, and MTP Measured in tokens/sVulkan beat HIP, four concurrent slots was the knee, and MTP only helped with a single request
- Qwen3.8-Flash-Next Review - Running It on an RTX 5090 + R9700 PairThree quantization swaps and sixteen rounds of advice later, the story of a local model swap that shipped to production
- Comparing GPU Performance After a Hardware Swap - Why the Verdict Flipped TwiceMy first direction call was wrong, and my later purchase call was wrong too, for two completely different reasons — the root cause both times was how I was reading benchmarks
- Gigabyte RTX 5090 AORUS MASTER ICE Purchase Review - Consolidating Two GPUs Into OneWhat changed after replacing two split GPUs with a single card
- I Finally Bought a Card I'd Rejected Several Times - AMD Radeon AI PRO R9700The story of buying a card I'd turned down as a main GPU more than once, for a completely different purpose