"[Jul 27-29] A Silent Crash, a Missing Record, and 27 Hours of Signal Silence"
Three days spent chasing incidents where "nothing happened" rather than "money went out wrong" — a silent Monday-morning crash, a performance-record gap found via a reader tip, and over a day of signal silence during a circuit-breaker event
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Spent several days doing nothing but incident response, so there was no time to write. Bundling the 27th, 28th, and 29th into one post.
Looking back, this was a stretch unusually full of "silently failing" incidents — not orders going out wrong, but things just stopping with nothing happening at all.
Monday pre-dawn: trading stopped with no alert
Right after Monday's market open, two different criteria for judging cache freshness disagreed with each other, and the execution path stopped dead with no alert. Fortunately no bad orders went out and no money moved.
It just quietly stopped with nothing happening — which is exactly why it almost went unnoticed for a while. Found and fixed the root cause the same day and shipped it.
A missing record found through a reader's comment
Someone reading the blog left a comment saying the performance stats looked stuck at an old value. Checking it out, the recommendation performance record had in fact been silently missing for a few weeks, caused by the same kind of cache-judgment bug as above.
After fixing it, I also went back and double-checked whether the recently strong-looking performance numbers were real. The sample was small and happened to overlap with one stock's sharp swing, so the conclusion was that it's not yet a solid basis for action.
So I left it alone as a metric to keep watching rather than acting on. Doubting good-looking numbers and re-verifying them is one of the recurring routines in this project.
Tuesday: 27 hours of signal silence
On Tuesday, the real-time monitoring process stopped during an intraday circuit-breaker event. Signals went silent for nearly 27 hours with no alert, and it wasn't discovered and restarted until the next day.
This was the heaviest incident of the three days. It reconfirmed that a failure where "nobody knows nothing is happening" is much harder to notice than a failure where money goes out wrong.
Rebuilt the alerting setup
This chain of events prompted a rework of the alerting setup. Split trading alerts and system-health alerts into separate channels, and added heartbeat and stall-detection to the monitoring process so the same kind of silent failure gets caught immediately if it recurs.
Also changed how brokerage rate-limit hits are handled — instead of treating every failure the same, it now distinguishes retryable cases from non-retryable ones and retries accordingly. Layering safeguards this way is in the same spirit as the earlier writeup on live-trading safeguards(new tab).
During this work, narrowing the alert scope accidentally also turned off the daily morning report shared with family. Caught it quickly and reverted it the same day.
The story of splitting processes and monitoring their health continues in the post on the operational structure(new tab).
Also this stretch
- Confirmed that an earlier read of "not trading a certain group of stocks" was a misread snapshot — trades were actually going through and being closed out normally all along.
- Hit the pre-set "5 consecutive incident-free days" milestone.
- Cleaned out an old folder with no commit history, and reconfirmed a physical copy of the backup passphrase was on hand.
Across these three days, more time went into "making sure failures don't slip by quietly" than into new features. Not flashy, but this is exactly the kind of work needed before expanding live-trading scope.