"[Sep 7] A VRAM Cleanup Patch Came Back as an Evening Production Outage"

A GPU memory cleanup applied earlier in the day triggered three straight production failures that evening, and I also fixed a brokerage API pagination bug

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

The VRAM cleanup patch that caused an evening outage

Earlier in the day I cleaned up an issue where one of the signal models kept holding onto a small amount of graphics card memory.

This model never really needed the GPU in the first place — its compute load is small — but it wasn't releasing all of its memory even after finishing. I pinned it to run on CPU only at the code level, and to clear the last remaining bit, added a setting to its service unit that made that graphics card invisible to it entirely.

The trouble started that evening. That setting didn't just apply to the signal model I was targeting — it also got inherited by a separate large language model server process running under the same service unit.

From that server's point of view, the graphics card it needed had suddenly vanished, so it failed to start. It fell back to a backup path, but the backup model wasn't installed either, so it failed three times in a row, quickly.

Fortunately, a safety mechanism built for exactly this kind of situation kicked in. When failures pile up within a short window, it safely stops that day's run with zero tickers processed — so no bad data got written, it just quietly stopped.

Once I traced the cause, I narrowed the fix so the setting only applies inside the signal model's own code, not the whole service unit. After restarting, the language model server found its graphics card again normally, and that day's ticker analysis picked back up as expected.

A brokerage API pagination bug

A separate issue: one held position was silently missing from a paper-trading account's holdings query.

Both the internal ledger and the actual trade fill records showed the position correctly, but the brokerage API's balance query came back without it. The trading logic's ledger-reconciliation safety check(new tab) flagged the mismatch and safely halted trading on that ticker.

The root cause was pagination. The brokerage API splits large balances across multiple pages and signals whether more pages remain via a response header.

The existing code only ever read the first page. The missing position happened to be sitting on the second page, so it dropped out silently with no error.

I fixed it to check for and follow the next-page flag until the response was exhausted, and confirmed the position was recognized correctly again on the next round. Other API calls using the same pagination pattern might have the same gap — flagged for a separate check later if needed.

Also today — an intraday shadow server healthcheck bug

A separate observation-only language model server that runs during market hours always seemed to take the maximum wait time just to report itself as ready.

It turned out the server was actually finishing startup in under 90 seconds — the healthcheck code was just assuming a response format it never sent, so it misread every healthy response as a failure. As a result, a perfectly healthy server was waiting out its full timeout every single time before the code moved on.

After aligning the check with the same logic used by the other servers, startup confirmation now passes at normal speed.