Postmortems
All Postmortems posts (22)
- A Scheduled Job Failed Every Action but Still Consumed Its One-Time Slot - A Run That Did Nothing Ate the ResourceA calendar library returned the wrong value for a once-a-year special day, and on top of that a "running = slot consumed" assumption poisoned the backup path
- A Test Run Sent Real Push Notifications - Because Mocking Side Effects Was Left to Each TestThe habit of letting each test intercept notification-sending leaked into a real send through one newly added alert path — a postmortem on why blocking side effects has to be the test harness's global default
- A Nightly Batch Job Hung for 26 Hours - One Unbounded Network CallMy first fix targeted the exact line that hung, but missed a sibling fallback call in the same function — a postmortem on why the recurrence-prevention fix was only half right
- A Research Backfill's Bulk Calls Crashed Live Trading - Root Cause Was an Import-Time Network CallA research backfill triggered an external API block, and that block killed live trading processes during library import
- How a Hardcoded sys.path Silently Broke Git Worktree IsolationEvery worktree's test suite passed — but that didn't mean it had actually tested that worktree's code
- A Manual Run Misclassified as a Scheduled Run - systemd Activation Metadata Outlived My AssumptionsA bug I thought I'd fixed a month earlier came back with a completely different root cause
- How Error-Recovery Guidance Caused a Double-Execution BugThe happy path passed hundreds of tests — the real critical bug was hiding in the "what to do when it fails" instructions
- An Order-Dependent Test Failure - Traced to a Monkeypatch Nobody RestoredA test passed in isolation and only failed inside the full suite. The culprit was a monkeypatch in another file
- The Market-Cap Column Came Back All NaN - Why No Error FiredWhen a sort key vanishes entirely, a sort function quietly hands back the original order instead
- The Dropped Temperature Parameter - Why the A/B Test Was Invalid From the StartPassing a value to a parent function and that value actually reaching the server turned out to be two different statements
- I Added a Second Broker API, and Every Incident Since Has Come From ItA class of bugs that barely showed up on KIS (Korea Investment & Securities) for weeks started firing almost weekly the moment NH Securities' Open API joined the stack
- When a Test Cleanup Step Killed a Production ProcessA pattern-matched kill command that collided with production naming, and an error classifier that only looked at a substring, together produced a completely wrong diagnosis
- Qwen3.8-Flash-Next Review - Running It on an RTX 5090 + R9700 PairThree quantization swaps and sixteen rounds of advice later, the story of a local model swap that shipped to production
- Why a Resident Process Could Never Recognize a Newly Registered ConditionThe code was deployed, but the process meant to run it had already been running since before that
- A Test's Missing Mock Corrupted a Production DatabaseIt took two straight days of the same incident to find the real cause - local imports and Python's default-argument binding
- The except Clause That Failed Silently - A New Module That Did Nothing for 8 DaysA single except line meant to quietly skip cache corruption also swallowed the case where the dependency library wasn't even installed
- 3 Cases Where Fully Automated Trading Still Needed a HumanMaking something run automatically and having it actually get handled to completion turned out to be two different sentences
- 3 Reasons Automated Trading Verification Tools Miss ErrorsEvery new verification tool I added this week failed in a different way — a bug in the tool itself, a blind spot nobody had wired a check for, and a gauge that lied about the moment it was reading
- Why the Devlog Automation Sat Broken for Nearly a MonthWith no logs at all, there was no way to tell "running" apart from "dead"
- A Trading Safety Guard Misfired - The Time Scale of the Threshold Was WrongA guard meant to catch cumulative losses reacted to a brief intraday dip instead — the fix came down to whether two numbers were measured on the same time scale
- Comparing GPU Performance After a Hardware Swap - Why the Verdict Flipped TwiceMy first direction call was wrong, and my later purchase call was wrong too, for two completely different reasons — the root cause both times was how I was reading benchmarks
- Building a Numeric Verifier for AI Report OutputA self-consistency check can't catch a number that's internally coherent but simply wrong — so I designed a different kind of verifier