Quant Trading Bot Devlog

한국어로 보기

Perishable Data - a Side Project That Grew From One Stream to Three

This is the English version of a post originally written in Korean. See the Korean original and other posts(new tab).

Not long ago I split a small project off from the main quant bot project. It's now grown into three. It started with just archiving old stock news.

Why I built it separately

I set a rule for deciding whether to spin up a new project: perishability. Is this data I'll lose forever if I don't capture it now, or is it something I can always recreate later?

So I split news out into its own dedicated project first. An interesting conclusion followed from that: if I archive news thoroughly enough, I can reconstruct a full snapshot of any past moment from it alone.

Price and macro data can always be recreated on demand, so once I have the news, I can backfill the rest whenever I need it. That makes this project less of a storage system and more of a materials warehouse the main project pulls from when needed.

Keeping the format 100% identical to the original

The thing I cared about most when re-collecting old news was that it had to match the exact same format as the news being collected in real time. Even a small format mismatch would make it impossible later to tell whether a difference in the results came from the data being different, or from an actual real-world difference.

So instead of writing new filtering/processing logic from scratch, I reuse the exact code the main project already uses. I could have made this a fully independent project, but that risked the two codebases slowly drifting apart over time. Instead I deliberately made this depend on the main project, removing that risk of format drift entirely.

Finding a collection path

Figuring out where and how to get the old news took some trial and error. The first approach I tried had too shallow a reach, and the next one produced too much data to afford. I eventually settled on scanning entire portal news sections and picking out only the articles that mention a given ticker — and confirmed this approach can reach any past date I need (in practice, back to 2015).

Right now there's a background job running continuously, working backward from the most recent date to 2015. It's built to quietly step out of the way of the main project's nightly analysis jobs so they don't compete for resources, and collecting everything should take a couple of months.

What it's for, and what it's not for

I deliberately narrowed this data's use case up front. I decided not to use news reconstructed from the past to validate anything tied to actual future returns — since it was collected after the fact, there's no guarantee it exactly matches "which articles actually existed at that point in time," and mixing it into live validation risks distorting the results. Instead, I limited it to a use case that isn't directly tied to returns: checking how consistent models are across the same input.

From news to three streams

What started as a news project has now grown into three branches.

Investor sentiment from stock discussion boards and intraday minute-level price data got added under the same criterion (perishability). Minute-level prices are especially urgent — portal news at least gives you a few days of leeway, but minute-level prices can already be unrecoverable within a few hours, sometimes as soon as the same afternoon.

All three store data the same way: I keep the first time something was observed and, separately, the last time I re-checked whether the value had changed.

Board post view counts and reactions keep changing, and the still-forming last candle of a minute bar keeps updating. Without recording both timestamps, I couldn't reconstruct "what the value actually was at that point in time" later.

The board data is still just being collected for now. Feeding that signal into another comparison experiment I'm currently running could contaminate it, so I've deliberately held off on using it.

What the three streams share

As the project grew to three, some natural overlap emerged.

Ticker-name recognition logic, portal request-rate throttling, and detecting and tolerating network failures are needed by all three projects in almost identical form. I originally implemented each one separately, but after seeing the same class of bug repeat across all three, I pulled them out into a shared module.

Now these three pieces don't belong to any single stream — they sit as shared components directly under the umbrella project. Fix one place, and all three streams get fixed at once.

As the main project keeps growing, I keep having to decide "should this live together, or split off." The original criterion I set (perishability) has now been applied three times over, consistently. If similar data comes up again, it'll probably fold naturally into this umbrella project.