Quant Trading Bot Devlog

한국어로 보기

Local LLM Serving and GPU Speed Measurements

Measured tokens/s for Qwen 27B and 35B models on RTX 5090, R9700 and 7900 XTX with vLLM and llama.cpp: concurrency, MTP, quantization, and heterogeneous-GPU serving.

Measurements from running the AI analysis pipeline of a Korean-stock trading bot on local GPUs. I ran a 27B dense model with vLLM on a single RTX 5090 and a 35B MoE model with llama.cpp on an AMD R9700, and compared tokens/s, concurrent slots, MTP (speculative decoding) and quantization.

The same card can give opposite conclusions depending on the model. On the R9700, Vulkan beat HIP, four concurrent slots was the knee, and MTP only helped a single request. On the 5090 the KV cache was the bottleneck. Read the posts below in order: purchase decisions, speed comparison, tuning, then the heterogeneous-GPU serving postmortem.