"[Sep 13] Kill-Switch Postmortem - the Root Cause Was Still There"

Yesterday I patched the symptom of a false kill-switch trip in a hurry. Today's postmortem found the root cause was still live and could have blocked Monday's real trading again.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

The symptom was gone, but the root was still there

Yesterday I scrambled to fix a false trip of the live account's kill switch(new tab).

I fixed the immediate cause — a missing account tag — right away. But when I ran a full postmortem on the whole incident today, I found that only the symptom had been cleared. One root cause was still sitting there.

Here's what happened. Right before yesterday's false trip, a partial withdrawal had gone through on the account. That withdrawal got recorded in the ledger, but nothing re-anchored the loss-tracking reference point (the peak balance) to match.

With that peak left stale, the next scheduled round would compare today's balance against that old, higher peak.

That would make it look like a huge loss had happened when none actually did — which meant the same kill switch could fire again on Monday morning's regular round.

As soon as I found this, I re-anchored the peak to the current balance. I also found and fixed two more spots where the same missing-account-tag problem was still lurking.

I also revisited a memory-throttling safeguard I'd put in place. Looking at the actual measurements again, that setting could actually trigger memory pressure across the whole system earlier, not later.

So I removed it and kept only a much higher ceiling that stays out of the way under normal conditions and only steps in during a genuine runaway.

A false positive hiding in the reconciliation alert, too

The daily reconciliation process compares account balance against the ledger, and escalates an alert if the discrepancy exceeds a threshold two days in a row.

But the counter behind that streak had a flaw. On a day with multiple reconciliation runs, even if the day's last run came back clean, the counter still marked the whole day as "over threshold" if even one earlier run that day had been over.

So a day that had actually been fully corrected by end of day kept showing up in the "consecutive days over threshold" count — turning one real day of unresolved discrepancy into a false two-day streak. I fixed the counter to only look at each day's final verdict and added regression tests for it.

Bugs a code review caught while rolling out the auto-tuning system

I finished implementing the execution engine for a system I designed a few days ago — one that automatically re-tunes performance settings whenever the GPU or the model changes.

After finishing the implementation, I ran an AI code review on it, and several review angles cross-confirmed real bugs. The worst one: the logic that decides which trial is the quality baseline was inferring it from a naming convention in the trial ID, instead of an explicit field.

Left as-is, the baseline would silently shift to a new candidate every round, making any "did this actually get faster" comparison meaningless. I replaced that with an explicit role field set during the design step instead.

Beyond that, there were bugs that only show up once you actually run the thing — a port number not getting refreshed, an environment variable being overwritten after it was already locked in — and I fixed those too.

Then I tried the first real run on actual GPU hardware, and hit a different kind of problem. Partway through, I found that some of the core execution steps were still unimplemented stubs.

The system's safety design mandates that the GPU can only be touched through this one controlled path, so patching in a workaround on the spot to get past the stub would have violated that principle. I stopped the run instead of working around it, logged the reason, and left the rest of the implementation for the next session.

Verifying a safety net end to end on the spare GPU

There's a safeguard that, in a port conflict, is supposed to clean up only the production process and leave any experimental process alone. Until now it had only ever been checked with unit tests, since the two processes had never actually been running at the same time in practice.

Today I used a spare GPU I recently picked up to reproduce a real conflict — launching two actual OS processes under the same name as production, without loading any real model.

I then called production's actual cleanup logic against them, and confirmed that only the production process got killed while the experimental one stayed alive. That was the first time this safeguard was verified end to end under a real conflict, not just in a unit test.