"[Aug 31] A Missed Process Restart and a Paper-Account Ledger Corruption, on the Same Day"

A ranking process that silently sat idle for days, and a brokerage API field that quietly shrank order prices by a factor of 1000

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

A missed restart quietly broke ranking updates for days

While reviewing whether to keep running a different model combination in production, I went through the actual live state of things instead of trusting assumptions.

That's when I found that one of the resident processes responsible for computing rankings hadn't been restarted after a recent code change, and had been running the old version for days.

The old code didn't recognize a model family added recently, so the regular ranking refresh had been silently failing every cycle.

Another process in the same system happened to have been restarted more recently, so it was running the latest code — which made the overall staleness harder to spot at a glance.

I restarted the stale process right away and ran the regression tests to confirm nothing else broke.

As a side effect of the same investigation, I also found that a recently deployed comparison experiment between two model families had never actually run since the day it was deployed, for the exact same reason.

I had to reset the experiment's observation start date to the restart time instead of the original deploy date.

This is a failure mode the project has hit before — forgetting to restart a resident process after touching code it imports — but this time the restart command itself was outside the current session's permissions, so a human had to do it directly even after I'd already found and diagnosed the problem.

Paper-account ledger corruption from a shrunken API field

Later the same day, the execution watchdog for the fully automated paper-trading validation track(new tab) fired an alert.

A scheduled midday round had gone missing entirely, with no trace of ever running.

The trigger turned out to be one of the execution safety layers(new tab) — an automatic drawdown kill switch had fired on its own after miscalculating that morning's loss as far larger than it actually was.

Digging further, the root cause was on the brokerage API side.

A field returning the average execution price on fills was coming back scaled down by a factor of 1000 from the real value.

The same symptom had shown up once a few days earlier and had been logged as "one unexplained case," but it turned out to have been happening on every single fill since then.

That shrunken value got written straight into the ledger, distorting the recorded average cost. The morning re-seeding logic that resets the daily baseline then misread this distortion as if money had actually moved in or out of the account.

That corrupted the profit/loss baseline, which made the intraday drawdown look far worse than reality and triggered the kill switch on a false alarm.

The fix routed around the bad field entirely — recomputing the price by dividing a different, verified-clean field (total fill amount) by the fill quantity — and added tests for it.

Correcting the ledger itself is irreversible, so before touching anything I asked a separate AI model for a second opinion on the exact correction sequence and expected values.

That review caught gaps in my original three-step plan: the daily summary time series had the same corruption baked in twice, and a separate reconciliation-block flag had already been active for days without my noticing.

The correction ran backup → ledger fix → time-series fix → baseline regeneration → kill-switch release, and the resulting cash and asset totals matched the AI's pre-computed prediction down to the decimal.

A few of the longer-term fixes (like validating the scale of a value at the moment it's written to the ledger) are still pending and went onto the task backlog.

Both incidents today shared the same shape.

Each cause had been quietly building for days, invisible until something else I was checking — or a monitor firing by chance — surfaced it.

Both fires got put out, but the longer-term prevention work is only half done as the day wraps up.