Chasing a 100-Stocks-in-10-Hours Goal for a Local LLM
Tracked down why a setting that won on benchmark lost in production, and closed out a statistical alert I'd been re-litigating for days
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Trying to run a local LLM through 100 stocks in 10 hours
I spent most of today running an open-source large language model locally, on the stock-analysis pipeline, using my main GPU together with a spare card I picked up recently.
This model uses a mixture-of-experts structure — a large total parameter count, but a much smaller set of parameters actually active during inference. The goal was to process 100 stocks in under 10 hours.
Running requests one at a time already came in over 40% past that target. I expected batching multiple requests together to help, but instead it either slowed things down or made the GPU stop responding entirely.
At first I wrote those failures off as transient network errors. Asking the AI to dig through the system logs showed otherwise: the GPU itself was actually hanging, a hardware-level fault. From then on, I kept concurrency experiments within a range where that fault didn't reproduce.
A separate benchmark confirmed the batching engine itself wasn't at fault. The real bottlenecks were two structural things: the initial prompt-processing stage hogging its turn and stalling other requests behind it, and memory-reference range growing continuously as conversations got longer.
The most promising fix looked like moving part of the computation (the expert layers) entirely onto the secondary GPU. In single-request benchmarks this was clearly faster, and the gain grew the longer the request got.
But re-testing it under real concurrent load flipped the result completely. The setting that won on benchmark was actually slower in production.
I asked the AI to diagnose it again. Raw decoding speed was still faster with this setup, but the initial prompt-processing stage lost out in exactly the opposite way — switching back and forth between the two GPUs happened far more often, and an optimization that overlaps multiple requests' pipelines was disabled entirely under this configuration.
So I dropped that approach and went back to the simpler one — splitting computation by layer, spread evenly across both cards — which turned out to be the best option after all.
Instead, I found gains on a different axis. I applied a patch that reorders prompts so multiple stocks can share a common prefix, and another that nudges the model away from unnecessarily verbose answers, both of which cut processing time.
I couldn't yet tell whether these patches affect the actual stock ratings themselves or just fall within normal run-to-run variance, so I held off on a final verdict there.
I also reviewed the model's compression (quantization) level separately. The level I'm currently using turned out to be fairly aggressive. Going up a level for better quality would exceed GPU memory and slow things down significantly, so I decided against that — instead I'll pull a middle-ground level and run a small side-by-side comparison.
By the end of the day: I'd reached the best result server tuning alone could give, getting close to the 10-hour target but still short of it. Closing the remaining gap needs further code changes, which I'll pursue separately after getting approval.
(Added 2026-09-26) The speed tuning that followed this goal is written up in the post tuning the 27B dense on the 5090(new tab) and the post tuning the 35B MoE on the R9700(new tab).
Closing out a statistical alert after days of re-litigation
I finally closed out a statistical anomaly alert today that I'd been re-examining for days.
Switching to a different model recently increased the number of nightly comparison samples. But the alert's threshold was still calibrated on the old absolute-count basis.
That meant the alert could trip easily even when nothing was actually wrong. Recomputing it as a ratio instead showed the recent activity wasn't statistically distinguishable from the normal baseline.
I closed this case with no action taken, and I'm redefining the alert threshold to use a ratio going forward instead of an absolute count.
While investigating, I also found and fixed two bugs in an observation-only tracking path (used for measuring predictive signal quality). Neither touched live trading logic, so they had no bearing on this verdict.
Also wrapped up today
A few days ago, a motherboard swap and reboot briefly set the system clock to the past; the moment the time server corrected it, the system mistook a schedule that hadn't actually passed yet for one that had, and triggered a scheduled retraining job on a day it wasn't supposed to run.
Today I added two safeguards to the retraining pipeline as follow-up: one that automatically demotes an out-of-schedule run to a dry run instead of a real deployment, and one that holds off deployment if it comes too soon after the last one.
I also retuned the power limit on the spare GPU downward to deal with fan noise, and finished a head-to-head comparison of two driver backends on that card — the existing one won, so that's confirmed as the pick going forward.
With the recent full hardware swap now reflected in the docs, a few old hardware verdicts had their underlying assumptions disappear entirely, so I marked them invalid.