From an Exchange Lockout to a GPU Teardown — a Week of Chasing Root Causes All the Way
I downgraded an exchange-lockout crash into a quiet failure and finished a one-share live-trading test by hand. A recurring token-invalidation bug turned out to be caused by restart-order mismatch, and a secondary GPU cleanup went all the way through instead of stopping halfway.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
An exchange lockout, downgraded from a crash to a quiet failure
Early in the week, the exchange blocked access and one trading round crashed outright.
The cause wasn't the trading logic itself — it was the external library that fetches exchange data, which tries to log in the moment it's imported. When access is blocked, it gets back an abnormal response, tries to parse it, throws, and drags down everything that imported it.
The fix: treat any exception from that login attempt as a normal "login failed" state, no matter what triggered it. That way a block just leaves that day's data quietly empty instead of crashing the whole round.
The same afternoon, in an unrelated track, I took a new research hypothesis using foreign-flow data from pre-registration through an independent AI code review to merge, all in one day. Since I'd already looked at this data once before in a different analysis, I locked in the hypothesis and the minimum sample size up front to keep the result statistically honest.
Loosened the exit criteria, and merged paper-trading fixes alongside it
Research that had been running for a few days reached a conclusion. Instead of liquidating a holding as soon as its rank slipped a bit, I loosened the threshold to let positions run longer, and retired a separate rule that force-liquidated purely based on how long something had been held.
The conclusion went through two checks: a backtest to confirm the direction first, then a re-check against actual fill data. Real buy-side friction turned out to be worse than in simulation, so I added a monitor for that friction before finalizing the change.
The same day, paper-trading fixes merged alongside it — handling market holidays correctly, closing a race where reconnecting after a dropped exchange connection could duplicate an order ID, and handling stock splits without corrupting holdings. This code went through two rounds of independent AI review, and only merged after the second review's additional finding was addressed.
After restarting, the full test suite turned up a handful of failures and a dozen-plus errors. Before assuming the merge broke something, I re-ran the same suite in a separate workspace rolled back to before the merge — the same failures reproduced there too, meaning they predated today's change. Not reverting on the first sign of failure, and checking causation first, is what let the merge stand.
A one-share live test, closed out by hand
There was one thing the live account had never actually done: place an order and confirm all the way through that it lands correctly in the books. So I planned a small test — buy one share, then sell it.
Two paths were on the table: reviving a retired automated sell-check path, or just placing the order by hand in the app and having the system only record and reconcile the result.
Reviving the retired path needed extra steps like moving funds around, too much for one day, so I went with the manual order. This is purely a order-recording and reconciliation(new tab) matter — it doesn't touch the trading engine's decision logic at all.
The test filled cleanly the next trading day. Confirming the exact fee and tax amounts needs a ledger reconciliation after the next deposit lands, so I bundled it with another small test that had been sitting unreconciled from the week before.
The same week, day one of the hold-longer experiment showed early secondary metrics — unfilled-order rate, idle cash after a sell — trending better than the prior baseline. Full judgment is deferred another 20 trading days.
A recurring token bug, traced to restart order
For a few days, restarting the resident processes kept re-triggering a bug where the market-data API's auth tokens would invalidate each other.
Matching this recurrence against the last one, both coincided with restarting only one of the two resident processes while leaving the other on an older generation. Each process issues its own token on restart, so one process moving to a new generation while the other stays behind means each keeps invalidating the token the other just issued.
Restarting the lagging process to catch it up to the same generation stopped the invalidation logs immediately. This token only feeds price lookups, not the live order path, so it never actually hit trading — but it confirmed clearly that the two residents can't be restarted independently.
A new-exchange entry experiment failed one criterion at a mid-point check — a liquidity-depth threshold that had no way to recover even with the remaining time. I asked another AI whether to cut the experiment short or wait for the planned final judgment date, and it said wait: this experiment is also gathering answers to other questions that only resolve at the final date. I took that advice and kept the original schedule.
The secondary GPU, cleaned up fully instead of halfway
A secondary GPU I'd been observing for a week, with an eye toward selling it, got physically pulled over the weekend while the system was powered off.
Right after reboot, two safety units — the boot checker and the power-cap service — flipped to red immediately. Both were built to never silently pass when the registered hardware list doesn't match what's actually installed, so seeing them fire exactly as designed confirmed the mechanism worked.
I'd been assuming the main trading path and the intraday auxiliary-analysis path were already running entirely on the primary GPU, and had another AI verify that assumption against the logs — it confirmed it. It also flagged that a few currently-dormant, standby-only features were written to refuse to start at all without the secondary card present.
I started with the smallest possible change — removing just one line for the secondary card from the hardware registry. That broke far more tests (10) than expected (2), so I reverted it for the moment. Then, that same evening, the decision flipped: stop straddling the fence and finish the cleanup properly instead.
I removed the secondary card from the registry entirely and fixed the broken tests with a dedicated fixture that simulates the old hardware only inside those specific tests. I restarted the four resident processes tied to this change and confirmed they came back up clean, and the full related test suite — over 400 cases — passed.
The thread through the week
What ran through this week was not settling for a half-fix.
The exchange lockout didn't just get left as a crash — it got downgraded to a quiet failure. The token bug wasn't just suppressed — I traced it to the root cause of two residents being on mismatched generations.
The GPU cleanup followed the same pattern. When the first attempt broke more tests than expected, I rolled it back, but didn't stop there — the same day, I committed to finishing it properly instead.
Even a small decision like the one-share live test followed the same logic: instead of reviving an automated path, I went with the manual route that could actually be finished, start to end, in a single day.