"[260914] The Day Market-Cap Data Came Back Empty - A Universe Contamination Incident"

When market-cap data came back entirely empty, stock rankings silently reverted to alphabetical order — so I added three layers of defense to catch it next time.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

This morning, while the system was refreshing its list of tradable stocks (the "universe"), the external market-data provider had an outage and returned empty market-cap values for every single stock.

The problem was that the system let those empty values pass through. Once market-cap sorting became meaningless, the raw list's original order — alphabetical — got mistaken for "top N by market cap."

As a result, nearly the entire KOSDAQ slice and close to half of the KOSPI slice got swapped out for the wrong stocks in a single day.

The way I found out was a bit embarrassing. I noticed the real account had bought an unfamiliar stock and asked, "why did it buy this one?"

Following that question back led straight to the root cause: a trade that had gone out with no real signal behind it, chosen essentially at random. I consulted an AI twice to pin down the cause and scope precisely.

The actual damage turned out to be limited. The KOSPI side was safe, protected by a cache that gets locked in once each morning — the contamination only hit the KOSDAQ slice.

Only a handful of trades went out on the bad signal, and the loss wasn't directional — just transaction-cost-level churn from buying and selling without any real signal. One stray position from the bad buy was still sitting in the account, so I liquidated it as soon as things were back to normal.

Adding three layers of defense

The first move was halting trading on every affected account. While trading was paused, I worked on restoring the universe to its last known-good state from the day before.

One trap surfaced mid-recovery: an old version of the logic, not yet restarted with the fix, re-contaminated the data once more during the restore. I moved the restart earlier in the sequence to close that gap.

For the root fix, I stacked several layers of validation on the market-cap data. If the fraction of missing values from a data source is too high, it's rejected outright now. If the day-over-day membership change is abnormally large, the system raises an alert and keeps the previous known-good membership instead of applying the change.

Finally, there's one more revalidation step right before the universe is finalized. All three layers catch the same class of contamination but at different points — the data source, the update logic, and the final commit.

A GPU resource conflict on the same morning

Separately, the same morning, a manual performance benchmark that was running ate up most of the graphics card's memory, causing the scheduled report-generation job to fail three times in a row.

The failure alerts actually fired every time, but I missed them while focused on the incident above. Stopping the benchmark resolved it immediately, but there's still no process in place to check for resource conflicts between manual runs and scheduled jobs ahead of time.

What's next

Today's incident also exposed a reliability problem with the data source itself — I'm considering reporting the missing-data issue to the provider.

I'm also reconsidering whether the system needs to re-query market data live during trading hours at all. Since the universe is already locked in once each morning, an intraday re-query adds risk like today's without any real benefit.