"[Sep 2] A Live-Market Observation Pipeline Goes Live, and a Candidate-Pool Rebalance"

Three bugs surfaced back to back the moment a new pipeline first fired, and the same day I rebalanced which stocks even get considered

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

Three bugs hit back to back the moment the observation pipeline first fired

I'd spent the last few days building an "intraday MoE shadow" run — a pipeline that runs a secondary model on its own separate track during market hours, purely to observe its performance, with zero influence on actual trading decisions.

The wiring itself was already finished by yesterday: a new exception window that allows GPU use during the day, a frozen pre-registration document, and full data isolation with a trigger scheduled to keep it from ever touching the production path.

But the moment it first fired this morning, three problems hit in quick succession.

The first was GPU memory. I didn't realize a leftover server from a previous run was still holding onto resources, and the new instance crashed trying to start up.

I added a pre-check for available memory, and changed the logic so that if the production server is already serving the same model, the pipeline just reuses it read-only instead of spinning up a new one.

The second was a minor logging bug — when server setup failed, it was calling the nightly-batch skip logger instead of the intraday one. An easy fix.

The third was the serious one. The logic that decides when to stop intraday trading didn't know about the new exception window I'd added today, so shortly after firing it terminated early having processed almost none of the stocks.

I found it right away, added a check for the exception window into the stop logic, wrote a test for it, and fixed it. After all three bugs were resolved, I re-ran it manually — this time every stock made it through to completion.

Today became the first valid observation day for this pipeline. Whether to actually fold this model into live trading decisions is still a question for once enough data has accumulated — for now, this is pure observation.

Rebalancing how many KOSPI vs. KOSDAQ stocks make the candidate pool

Separately, there'd been a lingering question of whether too many KOSDAQ stocks were making the candidate pool while KOSPI names were underrepresented.

I asked an AI model for its take, and the first pass came back with what looked like a contradiction between signal ranking and market-cap ranking, so I asked it to double-check.

The re-check confirmed the two were essentially uncorrelated, and the picture came together: KOSDAQ wasn't actually oversupplied, while a handful of large-cap KOSPI names sitting just outside the candidate pool were consistently ranking near the top on signal strength.

The initial recommendation was to build a zero-cost observation logger and wait about a month for data to accumulate before deciding anything.

But when I asked whether we could just look at the past 30 days instead, it turned out the full-universe signal scores get overwritten every night — there was no historical record to look back at.

So I wrote a one-off script that reruns the current model against the past month of price history, reconstructing what the signal rankings would have looked like in retrospect.

Working from that reconstruction and another round of review, I made the final call to shift the pool toward more KOSPI candidates and fewer KOSDAQ ones. I synced the eleven places in the codebase that consumed this setting to a single shared value, ran the full regression suite, and restarted the resident processes to apply it immediately.

The rebalance pushes one currently-held position toward the bottom of the KOSDAQ ranking, so I'll be watching tonight's overnight batch to see whether it actually triggers a forced sale.