A Nightly Batch Job Hung for 26 Hours - One Unbounded Network Call
My first fix targeted the exact line that hung, but missed a sibling fallback call in the same function — a postmortem on why the recurrence-prevention fix was only half right
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
There's a core batch job that runs automatically every night.
To prepare for the next trading day, it pulls the full list of tickers from an external data provider and builds the input for a ranking calculation. It starts at night and should be done by morning.
What happened
Checking in the morning, the previous night's output was still sitting there, untouched. The job hadn't run even once overnight.
The scheduler log made things look stranger. The tick responsible for the job had fired hundreds of times, and every single time it just logged "already running, skipping."
The log said it was running, but the output was still from the day before. The job hadn't "failed." It was stuck forever at the exact spot where it started.
What I thought at first
The first suspect was the scheduler config. I assumed a concurrency limit was clashing with some other job and causing it to get skipped.
But when I counted how long the "skipping" log had been piling up, it came out to nearly 26 hours. A simple collision should skip a handful of times and move on — not repeat the same line for over a full night.
This log was only ever written at WARNING level, and nothing was automatically reading it and alerting anyone. So the job had been stuck for 26 hours and nobody knew.
The real cause
Dumping the live thread state of the running process (using a tool that inspects Python thread stacks) showed the stuck thread was waiting on a response while establishing a TLS connection to the external data provider.
The third-party library used to fetch the ticker list had an internal HTTP call with no timeout set. Normally the response came back fast, so it never mattered. But that one time, the remote connection just sat there — no response, no disconnect either. With no timeout, the call waited forever.
The scheduler had a "only one instance at a time" limit on this job to prevent it from running twice concurrently. That safeguard, usually a good thing, made this incident worse. One stuck thread occupied the job's slot permanently, so every tick afterward saw "one is already running" and just skipped, over and over.
How I fixed it
The natural fix was to run the call on a separate thread and give up with an exception if it didn't finish within a set time. This pattern already existed elsewhere in the same codebase — a helper built after an earlier, similar incident. I reused it as-is instead of inventing something new.
When the call timed out, the existing fallback chain (retry from another source → fall back to the last successful result cached locally) kicked in as designed. The fix was a few lines, and the test was easy to write: mock a call that never responds and assert it ends in a timeout.
A few hours after deploying the fix, I asked for a second opinion (an outside advisory review) on whether it was sufficient. That's where something more important turned up.
Inside the same function, there was another fallback path — meant to kick in if the first call failed — that hit the same host the same unbounded way. The line I'd fixed was only "the exact line that hung that particular day." A second line with the structurally identical risk was still sitting right next to it, inside the same function.
I wrapped that second call the same way. Only then could I actually say "this function as a whole no longer waits indefinitely on an external response."
The generalizable lesson
One unbounded external call can hang an entire system. Combine it with a safeguard like "no concurrent instances," and the hang doesn't surface as an error — it surfaces as a permanent occupation of that job's slot. That's exactly why searching the error logs turns up nothing: no error ever occurred in the first place.
Fixing only the line that hung is closer to treating a symptom than preventing a recurrence. You have to go looking for structurally identical patterns elsewhere in the same function or module — especially in branches that rarely get exercised, like exception-handling or fallback paths. You have to look one level wider than what you first saw, to close off the whole class of bug instead of just the one line.
A WARNING-level log might just mean it's piling up with nobody reading it. You need a separate mechanism to escalate a log line to a real alert once it repeats abnormally often in a short window. Writing something to a log and someone actually reading it are two different sentences.
"No errors" and "making progress" are different claims. For a job that's supposed to run on a schedule, error monitoring alone can't catch this class of hang. You need separate, deadline-based monitoring that checks: has the time it should have started come and gone, with no trace of output or progress to show for it?
Whether a timeout is explicitly set on every third-party library call is worth a permanent line item on a code review checklist — especially when that library has multiple layers of fallback and retry, and it's easy to check only one of them and move on.