Quant Trading Bot Devlog

한국어로 보기

A Scheduled Job Failed Every Action but Still Consumed Its One-Time Slot - A Run That Did Nothing Ate the Resource

A calendar library returned the wrong value for a once-a-year special day, and on top of that a "running = slot consumed" assumption poisoned the backup path

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

My automated trading system runs trading rounds on a fixed schedule during the day. Each round has a one-time slot attached to it that records "this round already ran today."

It's there to stop the same round from executing twice in one day. This slot mechanism isn't part of the stock-picking logic — it belongs to the order execution/safety layer, whose full design I've written up separately in the live-account post(new tab).

And if the morning's first round can't run for some reason, there's a separate backup round that tries once more a little later.

What happened

The problem showed up on a special day. It was a day with a national exam, when the stock exchange opens an hour later than usual.

But the first morning round still fired at its normal time. That was a time when the market hadn't opened yet.

Naturally, every order was rejected by the exchange with a "market not open" code. Not a single one filled — effectively, nothing happened in that run.

The real trouble came next. When the market actually opened and the backup round ran, it ran only in a degraded fallback mode instead of the normal path.

Looking into why, the first round that had been rejected wholesale that morning had already consumed the one-time slot. From the backup round's point of view, the slot for today was "already used."

A run that filled nothing — that failed entirely for environmental reasons — had eaten up the day's resource.

What I thought at first

At first I suspected the timing. Seeing the round fire at an odd hour, I assumed the scheduler was misconfigured and started digging through the schedule definitions.

The schedule was fine. The round had fired at its normal morning time, exactly as intended.

The problem wasn't the time the round fired — it was the assumption that the market was open at that time. That assumption was right on every normal day, and wrong only on this one.

The real cause — two layers

The cause had two layers.

Layer one: the calendar library returned the wrong value for the special day.

The system read "what time does the market open today" from an external exchange-calendar library. On normal trading days, this value was correct.

But for this special day (the one where the open is pushed back an hour), the library returned the usual opening time. It's a day that happens at most once a year, so the library hadn't accounted for the exception either.

So the code believed "we're past the open" and placed orders as usual.

Layer two — and this is the real lesson: the slot was consumed at "the moment the run started."

The slot-consuming logic ate the slot based on the fact that the round had "started," not on whether the round had actually accomplished anything.

So a run where 100% of the orders were rejected and nothing filled was still recorded, as far as the code was concerned, as "today's round ran."

Success and failure of a run weren't separated from resource consumption. Even without layer one (the calendar bug), the same problem would recur on any day a round got rejected wholesale for some other environmental reason.

How I fixed it

I can't fix the calendar library, so I built a separate, trustworthy opening-time decision.

On normal days it still uses the library's value, but for special days I know about in advance (delayed-open days) it first consults a manually maintained table. If that manual table is empty for any reason, I made it fall back to a safe default.

Then, when the morning round fires, I put a gate that first checks "are we past the open?" using this decision. If we're before the open, it holds the round and lets the backup round take over once the market has actually opened.

One design choice here is worth calling out. I thought, "couldn't I just make a fully rejected round release its slot automatically?"

I did consider that option. But I decided not to add automatic release.

There were two reasons. One is that this situation itself is rare — on the order of once every few years — so an auto-release path wouldn't be worth having switched on all the time.

The other, more important one, is that automatic release creates a new double-execution risk. The path of "it failed, so return the slot → execute again" opens a window to place the same round twice when the decision is ambiguous.

So instead of auto-release, I chose to raise an alert when a wholesale rejection is detected and let a human handle it manually. I judged explicit alert + manual intervention to be safer than rare, risky automatic recovery.

I also kept the ordering where the slot is claimed before orders are placed. If you defer the slot claim until after placing orders, you open the door to double buys under contention.

Generalizing

Don't consume a resource because you "started" — tie it to "what you actually did." If you consume a one-time slot, token, or idempotency key at "the moment the run starts," a run that failed without doing anything eats that resource.

"It ran" and "it did the work" are different statements. A run that failed entirely for environmental reasons (a dependency wasn't up, a precondition wasn't met) often should be treated as not having happened at all, rather than as having consumed the resource.

The once-a-year edge case is exactly where a library's default is wrong. Frequently used paths get walked by many people, so bugs surface early. Rare special days, leap seconds, daylight-saving transitions — these have fewer tests and fewer users, so an external library quietly returns a wrong value more easily.

For those rare paths, it's cheaper to not blindly trust the library and instead lay a thin, verified manual override on top.

If you have a degraded fallback path, check that the main path's failure doesn't poison the fallback. The backup round existed for "when the main round couldn't run," yet the very way the main round broke (consuming the slot and failing) is what kept the backup from running in normal mode.

When you design a failure-contingency path, you have to separately verify that the state it assumes (here, "the slot is still free") survives a main-path failure.

Choosing not to add automatic recovery is also a design decision. Logic that automatically undoes a failure looks elegant, but the recovery path itself can create a new risk (here, double execution).

If a failure is rare and auto-recovery introduces a worse new failure, alert + manual intervention can be the better choice. Automating everything isn't always the right call.

Since this incident, whenever I write new code that consumes a one-time resource, I've picked up the habit of checking once more: "is this consumption tied to the start of the run, or to the actual outcome?"

#software quality #debugging #automation #postmortem