"[Sep 4] Closing Out Yesterday's Workaround and Two Misfiring Bugs"

Fixed the hardcoding that blocked manual trading rounds at the root, and tracked down why a ledger-contamination bug recurred and why an overnight recovery guard kept misfiring

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

Going back to fix yesterday's workaround properly

While recovering from yesterday's overnight batch failure, I found that manually triggering a trading round always failed with zero orders placed.

At the time I patched around it by running the underlying script directly instead of going through the normal service layer, and left the real fix for later.

Today I actually dug into the cause. A flag marking "this is a scheduled automatic round" was hardcoded directly into the service unit files, so even a manual run got mistaken for a scheduled one every time.

I asked an AI for a design recommendation, and we settled on using a separate environment variable that systemd only injects when a unit is triggered by its timer. No new files or markers needed — just fixing how the existing check reads that signal.

The work went in three stages. First I added the new check function to the code and covered it with tests, then ran a rehearsal with disposable timers before touching the real units, to confirm the check held up even across retries.

Finally I removed the hardcoded flag from the three real service units and synced the repository's copies of those unit files to match.

Along the way I found a blind spot in how those copies were maintained: one recently-added unit had never been registered in the repository copy at all, so sync checks had been missing it silently. I added it this time.

The final confirmation is still pending — I want to see next week's first scheduled round log the marker correctly before calling this done.

Tracking the ledger contamination bug to its root, two days in

The phantom trade record problem I'd left unexplained yesterday showed up again today. This time I traced the recurrence back far enough to actually find the root cause.

The culprit was, again, a regression test. Same family of bug as yesterday's but hitting a different function — this test also left the real code path of an internal fill-recording function unmocked.

So every time the full test suite ran, it wrote a few fake fill records straight into the live production database. I backed up the contaminated records, removed exactly the fake ones, and reconciled the ledger against actual broker holdings again.

I fixed the test to properly isolate that function. And to keep this class of bug from recurring a third time, I formally registered that database file in the test-isolation list — something that would have caught both days' incidents immediately if it had been there from the start.

Separately, during the same investigation I confirmed another, unrelated fact: one paper-trading account failed to place any trades because a price-lookup call was rejected outright. That wasn't a code bug — it turned out the broker simply doesn't offer that lookup endpoint for paper accounts. The round ended safely with no trades and no real loss.

An overnight recovery guard was misjudging normal runs as stuck

Separately that day, I looked into a safety mechanism that watches a nightly model-swap job and force-reverts it if the job looks stuck.

It turned out this guard had misfired on both of the last two nights the job actually completed successfully — not a coincidence, but a structural problem that fired every single time.

The cause was a timing mismatch between the check interval and how long the job actually needs. The check interval was shorter than the job's normal completion window, so the guard would declare "stuck" while the job was still legitimately running.

I asked the AI to dig further back, and it found the actual trigger for when this misjudgment first appeared. On that earlier night, someone had run the same job manually via a separate script; when the regular process restarted right afterward, the two processes' markers got crossed, and the guard misread the first process's leftover marker as a stuck state.

So I changed the fix's basis from "how long has this been running" to directly checking whether the process actually doing the job is still alive. Since that check doesn't care which process started the job, it also resolves the cross-process case found today.

Deployment avoided a restart during market hours — it's scheduled to restart quietly once, in a fixed off-hours window. Whether it actually stops misfiring only gets confirmed by tonight's run.

All three things that landed on the same day followed a similar shape: one problem patched around yesterday, one recurrence of the same bug family in a different function, and one safety mechanism that had been misjudging a normal situation as an incident. In each case, I didn't just stop the symptom — I traced it back to the actual root cause before calling it closed.